Lead Data Engineer - Azure Databricks

Posting title : Lead Data Engineer (Databricks)

Experience : 7+ Years

Location : Pune / Ahmedabad

Work mode : Hybrid

Primary skills : Databricks, Data Modelling / Data Lakehouse, Gen AI, SQL, Python / Scala, Azure Data Factory (ADF) / AWS Glue, Docker, Team Lead

Qualification : Any Technical Degree

Roles & Responsibilities :

• Lead the design, development, and optimization of data solutions using Databricks, ensuring they are scalable, efficient, and secure.
• Collaborate with cross-functional teams to gather and analyse data requirements, translating them into robust data architectures and solutions.
• Develop and maintain ETL pipelines, leveraging Databricks and integrating with Azure Data Factory as needed.
• Implement machine learning models and advanced analytics solutions, incorporating Generative AI to drive innovation.
• Ensure data quality, governance, and security practices are adhered to, maintaining the integrity and reliability of data solutions.
• Provide technical leadership and mentorship to junior engineers, fostering an environment of learning and growth.
• Stay updated on the latest trends and advancements in data engineering, Databricks, Generative AI, and Azure Data Factory to continually enhance team Qualifications :
• Bachelors or masters degree in computer science, Information Technology, or a related field.
• 7+ to 10 years of experience in data engineering, with a focus on Databricks.
• Proven expertise in building and optimizing data solutions using Databricks and integrating with Azure Data Factory / AWS Glue.
• Proficiency in SQL and programming languages such as Python or Scala.
• Strong understanding of data modelling, ETL processes, and Data Warehousing / Data Lakehouse concepts.
• Familiarity with cloud platforms, particularly Azure, and containerization technologies such as Docker.
• Excellent analytical, problem-solving, and communication skills.
• Demonstrated leadership ability with experience mentoring and guiding junior team members.

Preferred Skills :

(ref : hirist.tech)