Data Engineering Lead - Python, Pyspark, Azure Databricks

Optum, part of UnitedHealth Group, is a global organization delivering technology-enabled care to help millions of people live healthier lives. As a Data Engineering Lead, you will design, build, and maintain scalable batch and streaming data pipelines, driving data modernization and mentoring junior engineers.

• Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using Medallion Architecture
• Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, and deduplication
• Implement data governance, lineage and security using Unity Catalog, including row/column-level security
• Build data quality, validation and reconciliation frameworks to keep data accurate and reliable
• Drive CI/CD and DataOps practices using Azure DevOps or GitHub with automated testing
• Troubleshoot production issues, perform root cause analysis, and mentor junior engineers

• B.E./B.Tech./MCA or equivalent degree in Computer Science, IT or Engineering
• 6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks
• Hands-on experience with Delta Lake, Lakehouse architecture and Unity Catalog
• Solid proficiency in Python, PySpark and SQL, with a strong understanding of Spark internals and performance tuning
• Experience with Azure services including ADF, ADLS Gen2, Key Vault, and Azure SQL
• Solid knowledge of data modeling, ETL/ELT design, Git, CI/CD and Agile delivery