Data Analyst, data engineers, data specialist

Senior Azure Databricks Data Engineer responsible for designing and building enterprise lakehouse architecture on Azure Databricks, implementing Medallion layers, ensuring data quality, governance, performance, and cost optimization, while leading a team and collaborating with stakeholders.

• Design and build Medallion lakehouse from scratch on Azure Databricks and Delta Lake
• Define layer contracts (Bronze, Silver, Gold) and build batch/streaming ingestion pipelines using Auto Loader, Structured Streaming, Delta Live Tables, Azure Data Factory
• Implement incremental processing, CDC, SCD, MERGE patterns, schema evolution, and late‑arriving data handling
• Set data quality rules, quarantine, and alerting for bad records
• Architect Azure Databricks environment: workspaces, VNet injection, Private Link, ADLS Gen2 layout
• Define cluster policies, compute types, and orchestration with Databricks Workflows
• Build reusable frameworks: metadata‑driven ingestion, common libraries, templates, and logging
• Tune Spark and Delta workloads: partitioning, liquid clustering, Z‑ORDER, OPTIMIZE/VACUUM, file sizing, caching, AQE, Photon
• Diagnose and resolve performance issues using Spark UI and query profiles
• Right‑size clusters, use autoscaling, spot instances, and serverless to optimise cost
• Implement Unity Catalog with metastore, catalogs, role‑based access, row/column security, data masking
• Enable lineage, auditing, data classification, and integrate with Microsoft Purview
• Manage secrets with Azure Key Vault and identities with Entra ID service principals
• Build CI/CD pipelines with Azure DevOps or GitHub Actions and Databricks Asset Bundles
• Provision infrastructure as code with Terraform or Bicep
• Add unit and integration tests, monitoring, and alerting for pipelines
• Act as technical lead: mentor engineers, review code and designs, set best practices
• Collaborate with architects, analysts, data scientists, and stakeholders to turn requirements into data products

• 7+ years in data engineering, with 3+ years on Azure Databricks
• Proven track record building a Medallion architecture from scratch
• Expert in PySpark and Spark SQL; strong Python and SQL skills
• Deep knowledge of Delta Lake (ACID, time travel, MERGE, CDF, OPTIMIZE, liquid clustering)
• Hands‑on with Unity Catalog, access control, lineage, auditing, PII protection
• Strong Azure stack experience: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics
• Demonstrated results in Spark performance tuning and cloud cost optimisation
• Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns
• CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform
• Excellent communication skills to explain design trade‑offs to engineers and business stakeholders