Technical Lead - Data Engineering

Technical Lead-Data Engg

Area(s) of responsibility

Job Description: Azure Data Engineer

Key Responsibilities

1. Data Solution Design & Architecture

• Design end‑to‑end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
• Architect and implement scalable and secure Modern Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.

2. Databricks & PySpark Engineering

• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
• Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

 

Experience Level

Senior Level