EY - GDS Consulting - AI And DATA -Azure Databricks-Senior

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

EY GDS – AI & Data

Databricks Senior Data Engineer

AI-enabled data platform engineering

The opportunity

Our AI & Data practice helps clients build trusted, modern data platforms that power analytics, operational decisions, and responsible AI. As a Databricks Senior Data Engineer, you will own end-to-end data products and technical workstreams, from design through production operation. You will combine strong engineering fundamentals with practical use of native AI capabilities on Databricks, always prioritising business value, security, and reliable production operation.

Your key responsibilities

Data engineering and platform delivery

• Build lakehouse solutions using Delta Lake, Lakeflow, Databricks SQL, Unity Catalog, and medallion or domain-oriented data-product patterns.
• Engineer secure batch and streaming pipelines using Spark, Delta Live Tables / Lakeflow Spark Declarative Pipelines, Auto Loader, workflows, and declarative quality expectations.
• Design and own scalable batch and streaming data products, selecting appropriate data models, processing patterns, quality controls, and service-level expectations.
• Troubleshoot complex performance, reliability, and cost issues; establish reusable engineering patterns and guide code reviews and delivery standards.
• Translate business requirements into reusable, tested data products with clear ownership, contracts, documentation, data-quality checks, and service-level expectations.

AI-ready data and native AI capabilities

• Use Mosaic AI, AI/BI Genie, Agent Bricks, vector search, model serving, MLflow, and Databricks Assistant / Genie Code capabilities where appropriate to accelerate governed AI delivery.
• Design AI-ready data products and implement production patterns for retrieval, semantic search, evaluation, safety, and reliable operation.
• Assess when a governed RAG or agent workflow is justified and implement the supporting ingestion, metadata, access-control, evaluation, and deployment foundations.
• Develop and test scoped system prompts, context-assembly patterns, agent skills, and approved MCP integrations with clear tool contracts, least-privilege access, input/output validation, and traceability.
• Partner with data scientists, analytics teams, security, and business stakeholders to select the right pattern: deterministic analytics, semantic layer, retrieval-augmented generation (RAG), agent workflow, or model-based solution.
• Ensure AI solutions have documented data sources, access controls, quality thresholds, evaluations, human oversight where needed, and clear release controls.

Governance, security, and engineering excellence

• Apply Unity Catalog governance for data, AI assets, lineage, access control, discovery, auditability, and cost-aware compute design.
• Implement automated testing, source control, code reviews, CI/CD, release controls, monitoring, alerting, incident learning, and clear runbooks.
• Work in Agile teams and communicate progress, dependencies, risks, and design decisions clearly to technical and non-technical stakeholders.
• Lead technical delivery for a workstream, mentor engineers, and communicate design choices and delivery risks to client stakeholders.

Skills and attributes for success

• Hands-on Python, PySpark, SQL, Delta Lake, Spark performance tuning, data modelling, and Git-based engineering practices.
• Databricks workspace administration concepts, jobs/workflows, compute policies, CI/CD, infrastructure as code, and Azure, AWS, or GCP integration.
• Practical experience implementing or supporting RAG, semantic search, agent workflows, context engineering, system prompts, agent skills, MCP integrations, or LLM evaluation would be an added advantage.
• Strong analytical problem solving, written and verbal communication, and a consulting mindset grounded in measurable client outcomes.
• Experience with API-based integrations, data contracts, security-by-design, and cloud-native identity, networking, and secrets-management concepts.
• Ability to explain trade-offs between batch, streaming, SQL, Python, PySpark, platform-native AI, and external AI services.

Experience and qualifications

• 5–8 years of relevant data engineering, analytics engineering, platform engineering, or comparable consulting experience.
• Bachelor’s or master’s degree in computer science, engineering, information systems, data science, or a related discipline; equivalent practical experience will be considered.
• Demonstrable delivery of production data solutions using Python, PySpark, and SQL. Experience with modern data modelling, distributed processing, and cloud data security is essential.
• Databricks certification (Data Engineer, Data Analyst, Machine Learning Professional, or equivalent) is preferred.

Ideally, you will also have

• Experience in financial services, consumer, healthcare, public sector, or another regulated industry.
• Experience with data mesh, domain ownership, event-driven architecture, or FinOps practices.
• Evidence of building accessible, inclusive teams and improving engineering practices through reusable accelerators, standards, or coaching.

EY | Building a better working world

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.

Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.

Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.