Senior Data Engineer
Key Responsibilities
- Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
- Develop and maintain data pipelines following the Medallion Architecture (Bronze, Silver, Gold) to deliver scalable and trusted data products.
- Implement ingestion frameworks for structured and semi-structured data from APIs, JSON files, databases, and other enterprise sources.
- Manage and optimize data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and associated Azure services.
- Build, schedule, and support workloads using Databricks Workflows/Jobs, with experience in Delta Live Tables (DLT) or Lakeflow considered a strong advantage.
- Implement schema management practices, including schema enforcement and schema evolution.
- Develop and maintain data quality controls, validation frameworks, reconciliation processes, and automated testing.
- Design and implement data transformation, cleansing, and standardization processes to support business reporting and analytics.
- Establish metadata management, data lineage, and governance capabilities using Unity Catalog and related technologies.
- Develop API-based ingestion and export integrations for upstream and downstream systems.
- Support solutions involving document metadata management and references to PDF and eLabel assets.
- Drive performance tuning, partitioning strategies, and platform optimization to improve scalability and efficiency.
- Implement CI/CD practices for notebooks, code deployments, and automated testing throughout the development lifecycle.
- Develop monitoring, alerting, error-handling, and reprocessing capabilities to ensure operational stability and reliability.
- Support regulated data environments with a strong focus on traceability, auditability, reproducibility, and controlled release management.
FAQs
Congratulations, we understand that taking the time to apply is a big step. When you apply, your details go directly to the consultant who is sourcing talent. Due to demand, we may not get back to all applicants that have applied. However, we always keep your resume and details on file so when we see similar roles or see skillsets that drive growth in organizations, we will always reach out to discuss opportunities.
Yes. Even if this role isn’t a perfect match, applying allows us to understand your expertise and ambitions, ensuring you're on our radar for the right opportunity when it arises.
We also work in several ways, firstly we advertise our roles available on our site, however, often due to confidentiality we may not post all. We also work with clients who are more focused on skills and understanding what is required to future-proof their business.
That's why we recommend registering your resume so you can be considered for roles that have yet to be created.
Yes, we help with CV and interview preparation. From customized support on how to optimize your CV to interview preparation and compensation negotiations, we advocate for you throughout your next career move.