Senior Data Engineer

Key Responsibilities

• Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Develop and maintain data pipelines following the Medallion Architecture (Bronze, Silver, Gold) to deliver scalable and trusted data products.
• Implement ingestion frameworks for structured and semi-structured data from APIs, JSON files, databases, and other enterprise sources.
• Manage and optimize data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and associated Azure services.
• Build, schedule, and support workloads using Databricks Workflows/Jobs, with experience in Delta Live Tables (DLT) or Lakeflow considered a strong advantage.
• Implement schema management practices, including schema enforcement and schema evolution.
• Develop and maintain data quality controls, validation frameworks, reconciliation processes, and automated testing.
• Design and implement data transformation, cleansing, and standardization processes to support business reporting and analytics.
• Establish metadata management, data lineage, and governance capabilities using Unity Catalog and related technologies.
• Develop API-based ingestion and export integrations for upstream and downstream systems.
• Support solutions involving document metadata management and references to PDF and eLabel assets.
• Drive performance tuning, partitioning strategies, and platform optimization to improve scalability and efficiency.
• Implement CI/CD practices for notebooks, code deployments, and automated testing throughout the development lifecycle.
• Develop monitoring, alerting, error-handling, and reprocessing capabilities to ensure operational stability and reliability.
• Support regulated data environments with a strong focus on traceability, auditability, reproducibility, and controlled release management.