Data Engineer - Azure Databricks/Python

Varite is seeking a skilled Data Engineer to design, develop, and maintain data infrastructure using Databricks, Python, and SQL to support business needs.

• Develop and implement efficient data pipelines and ETL processes to migrate and manage client, investment, and accounting data in Databricks
• Work closely with the investment management team to understand data structures and business requirements, ensuring data accuracy and quality
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability of data systems
• Optimize database performance by designing scalable and cost-effective solutions

• 5+ years of experience in data engineering or a similar role
• Proficiency in Apache Spark, Databricks Data Cloud, including schema design, data partitioning, and query optimization
• Exposure to Azure
• Exposure to Streaming technologies (e.g. Autoloader, DLT Streaming)
• Advanced SQL, data modeling skills, and data warehousing concepts
• Experience with ETL/ELT tools like snap logic and programming languages (Python/Scala/R programming)
• Familiarity with workload automation and job scheduling tools such as Control M
• Familiar with data governance frameworks and security protocols
• Bachelor's degree in computer science, IT, or a related discipline