Lead Data Engineer

Data Principal Engineer

Data Engineer – Lead

Responsibilities:

• Lead the design and implementation of scalable data pipelines and data processing frameworks in Microsoft Fabric.
• Define data engineering standards, development practices, naming conventions, coding guidelines, and reusable technical patterns.
• Lead implementation of Bronze, Silver, and Gold layers in the Medallion Architecture.
• Oversee ingestion, transformation, orchestration, validation, and publication of data from multiple enterprise, clinical, operational, and cloud-based sources.
• Guide development of Fabric Pipelines, Dataflows Gen2, Notebooks, Lakehouse tables, Warehouse objects, and curated datasets.
• Ensure scalability, performance, reliability, maintainability, security, monitoring, and optimization of data solutions.
• Define standards for data quality, reconciliation, logging, error handling, auditability, and lineage.
• Conduct technical design reviews, code reviews, performance reviews, and deployment readiness reviews.
• Mentor and guide data engineering teams across onshore/offshore locations.
• Collaborate with architects, platform engineers, BI teams, QA teams, AI/ML teams, functional consultants, and stakeholders.
• Support deployment planning, release activities, production support, and operational handover.

Skills:

• Strong hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, OneLake, Pipelines, Dataflows Gen2, Notebooks, and Power BI integration.
• Expertise in ETL/ELT, data pipelines, distributed data processing, and cloud-scale data engineering.
• Strong SQL, Python, PySpark, and data modeling skills.
• Experience with Lakehouse, Warehouse, and Medallion Architecture.
• Understanding of Delta tables, dimensional modeling, star schema, facts, dimensions, and curated analytical datasets.
• Experience integrating structured, semi-structured, file-based, API-based, enterprise application, and cloud data sources.
• Experience with data quality, reconciliation, logging, monitoring, and error-handling frameworks.
• Experience leading technical teams and coordinating onshore/offshore delivery.
• Experience with Git, CI/CD, Azure DevOps, branching, code reviews, and release management.
• Good to Have: Experience with Azure Data Factory, Synapse, Databricks, ADLS Gen2, Azure SQL, Microsoft Purview, or related Azure services.
• Good to Have: Experience with CRO, clinical research, pharma, healthcare and clinical systems.

Experience Level

Senior Level