Specialist Software Engineer - Data Engineer Databricks pySpark

Société Générale is seeking a Specialist Software Engineer - Data Engineer to design, develop, and optimize robust, high-performance data pipelines using Databricks and pySpark within complex Big Data environments.

• Understand business and technical requirements and translate them into scalable, maintainable data engineering solutions
• Own the design and development of robust, high-performance data pipelines using Databricks and pySpark
• Build and optimize data flows within the Hadoop ecosystem
• Ensure performance tuning, resource optimization, and efficient cluster usage on the Databricks platform
• Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation
• Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams
• Support production processes, troubleshoot issues, update runbooks, and maintain system stability

• Bachelor of Engineering in Computer Science, Information Technology or equivalent streams
• Strong hands-on experience in pySpark for distributed processing and performance optimization
• Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst
• Good command of Python for supplemental processing and pipeline utilities
• Strong understanding of data modeling, partitioning strategies, and governance principles
• Proficiency in Git and modern SDLC practices
• Experience with CI/CD pipelines (GitHub Actions, Ansible, AWX)