Data Engineer

Please note this role requires you to pass Security Check clearance. For further information, please see 'Selection process details'.

Are you an experienced Data Engineer with strong Analytical skills?

Do you want to utilise best in class technologies to deliver exceptional cloud-based data products for Data Analysts and Scientists?

DWP’s Digital Group is looking for experienced Data Engineers to fill post in Management Information (MI) team.

The MI team provides strategic capabilities for MI reporting and data visualisations across DWP. Working with modern cloud environments alongside legacy on-premises technologies, you’ll play a key role in developing the DWP Data Engineering Strategy for MI products and services.

Successful candidates will join a fast-paced environment with a wide portfolio of products for analytical use and a challenging delivery roadmap. You’ll focus on a variety of tasks, supporting platform live service and delivering high quality, data transformation products.

• Building and maintain scalable data engineering solutions that power analytical reporting, working in line with DWP’s engineering standards.
• Collaborating with stakeholders across DWP to gather and clarify requirements where needed and translate them into scalable back-end data processes that support MI reporting needs.
• Maintaining comprehensive technical documentation and collaborative workspaces in Confluence to improve transparency, knowledge sharing, and onboarding.
• Working closely with other junior data engineers to provide guidance, share technical expertise, and actively support ongoing knowledge sharing.
• Reviewing code developed by fellow engineers to ensure quality, maintainability, and adherence to DWP coding standards.

This job is broadly open to the following groups:

• UK nationals
• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in a new window)

This vacancy is using Success Profiles (opens in a new window), and will assess your Experience and Technical skills.

Stage 1: Application

Your application will consist of four parts:

1. A Personal Details application form.

2. Employment history - this should contain your work experience and any skills, qualifications and accomplishments relevant to the jobs you have completed.

3. Personal statement - up to 750 words. This statement should be used to provide examples of how you meet the essential criteria below:

• Practitioner: Ability to orchestrate PySpark jobs on EMR, managing dependencies, workflow execution, and operational efficiency.
• Working: Knowledge of designing event-driven architectures, leveraging AWS Lambda functions to trigger PySpark workflows based on real-time data ingestion and processing needs.
• Practitioner: Experienced in provisioning cloud infrastructure using Amazon EC2, and automating deployment, configuration, and data workflows with Bash scripting across scalable environments.
• Working: Ability to apply Agile best practices in developing CI/CD pipelines, using continuous integration, testing, and deployment within a collaborative development framework.
• Practitioner: Understanding of optimising queries, storage formats, and data structures to enhance efficiency, reduce processing time, and ensure high data quality standards in complex cloud architectures.

4. Technical statement (up to 250 words). The following statement is aligned to the required technical skill of Technical Breadth. This statement should be used to provide example(s) of how you meet the criterion below:

• Practitioner: Good understanding of designing and automating ETL pipelines using PySpark and Spark SQL, ensuring seamless extraction, transformation, and loading of structured and unstructured data across cloud environments.

Ensure that all examples provided in your statements are taken directly from your own experience and that you describe the examples in your own words. If you choose to use gen-AI to support your statements, you must follow the guidelines outlined in the Artificial intelligence and recruitment guide.

The sift panel will use the information in your employment history, personal statement, and technical statement to assess your experience, skills and knowledge.

An initial sift will be conducted using the technical statement. Candidates who pass the initial sift will be progressed to a full sift.

If you do progress to a full sift, you will be provided with one combined overall assessment score for both your employment history and Personal Statement.

For Hints and Tips on completing your application visit Applying for jobs at DWP Digital.

Important Information

• You will be asked to complete your employment history. Any information that you would customarily share on a CV should therefore be entered onto the application form.
• Personal details that could be used to identify you including your name, contact details and address must be removed for your application to be considered.
• If your employment history, personal statement or technical statement contain any personal details your application will be withdrawn.

Stage 2: Interview 1

If you’re successful at sift stage, you will be invited to a 15 minute video interview via Microsoft Teams.

You will be asked questions to further assess the experience and evidence you provided in your sift application.

Interviews will take place from late September 2026. Sift and Interview dates to be confirmed.

Stage 3: Interview 2

If you’re successful at the stage 2, you’ll be invited to a video interview via Microsoft Teams. During the interview, you’ll be assessed against the experience criteria listed under both the essential criteria and technical skill.

Interviews will take place from early October 2026. Sift and Interview dates to be confirmed.

Further Information:

If high application volumes are received, the benchmark for candidates to proceed to the next stage may be raised. In line with our commitment to the Disability Confident Scheme (DCS), we aim to advance all candidates applying under the DCS who meet the minimum standard. However, we may only progress those candidates who best meet the required standards.

All successful candidates and those placed on reserve will be posted in merit list order by location.

For these vacancies, we strongly recommend that applicants consult with an immigration specialist or qualified advisor to assess their eligibility for Visa Sponsorship before deciding to apply. Please note that while we consider sponsorship requests in accordance with current DWP guidance and Home Office policy, sponsorship cannot be guaranteed.

Security Clearance Requirement

You must meet the security requirements before you can be appointed. The level of security needed is security check.

You must have resided continuously in the UK for at least 3 of the last 5 years, 2 of which must have been the immediately preceding years from the point of applying for this job.

For further information on National Security Vetting please visit the Demystifying Vetting website.

Further Information​

At the Department for Work and Pensions (DWP) we value diversity and inclusion and actively encourage and welcome applications from everyone, including those that are underrepresented in our workforce.

Find out more about working for DWP on the DWP Careers Hub.

Reserve List

A reserve list may be held for a period of 6 months from which further appointments can be made.

If you are placed on a reserve list but we cannot immediately offer you a post, please note:

• If you are later offered the role you applied for, in a location you have expressed a preference for, and you decline the offer or are unable to take up the post within a reasonable timeframe you will be withdrawn from the campaign and removed from the reserve list, other than in exceptional circumstances (e.g. a contractual Part Time Working Pattern cannot be accommodated in the initial role offered or in cases of serious ill health).
• If DWP makes an offer of an alternative role or location to that which you originally applied for, and you decline that offer, you will be able to remain on the reserve list.

Disability Confident Scheme

If high application volumes are received, the benchmark for candidates to proceed to the next stage may be raised. In line with our commitment to the Disability Confident Scheme (DCS), we aim to advance all candidates applying under the DCS who meet the minimum standard. However, we may only progress those candidates who best meet the required standards.

Reasonable Adjustments

We consider visible and non-visible disabilities, neurodiversity or learning differences, chronic medical conditions, or mental ill health. Examples include dyslexia, epilepsy, autism, chronic fatigue, or schizophrenia.

If you need a change to be made so that you can make your application, you should contact the Government Recruitment Service via DWPRecruitment.grs@cabinetoffice.gov.uk as soon as possible before the closing date to discuss your needs.

Complete the “Reasonable Adjustments” section in the “Additional requirements” page of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional.

If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the 'Contact point for applicants' section.

Integrity, plagiarism and Civil Service Principles

The Civil Service values honesty and integrity and expects all candidates to abide by these principles.

Please ensure that all examples provided in your application are taken directly from your own experience and that you describe the examples in your own words.

Examples of plagiarism can include:

• Presenting the work, ideas and experience of others as your own
• Copying content or answers from an online or published source that is not your own

Disclosure and Barring Service and Internal Fraud Database Checks

To process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service on your behalf. However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading.

Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment. A candidate is not eligible to apply for a role within the Civil Service if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government.

If successful and transferring from another Government Department, a criminal record check will be carried out.

Important

New entrants are expected to join on the minimum of the pay band.

Before applying for this vacancy, current employees of DWP should check whether a successful application would result in changes to their terms & conditions of employment, e.g. mobility, pay, allowances. If you are a current employee and are successful you must be able to be released from your current post within four weeks.

Those on protected TUPE/ COSoP terms and conditions applying on promotion or voluntary permanent level move will adopt DWP’s Terms and Conditions and this may have a different impact on pay and allowances. Please review this prior to acceptance of a role.

Civil Servants that would transfer into DWP from other government organisations, following successful application, will assume DWP's terms & conditions of employment current on the day they are posted, unless DWP has stated otherwise in writing. Accepting a post will be taken to mean acceptance of revised terms & conditions.

Civil Servants applying on promotion will usually be appointed to the salary minimum or within 10% of existing salary.

Any move to DWP from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may, however, be eligible for other government schemes, including Tax Free Childcare. Determine your eligibility at the Childcare Choices Website.

For further information on reasonable adjustments, terms and conditions and how we recruit visit the How we recruit page.

Feedback will only be provided if you attend an interview or assessment.