Azure Databricks / PySpark Developer - ETL

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and ETL / ELT processes using PySpark, SQL, and Python.
• Build and manage data workflows on Azure Data Lake, Azure Data Factory (ADF), and Databricks.
• Collaborate with data scientists, analysts, and other stakeholders to understand data needs and ensure data quality and availability.
• Implement data modeling, transformation, and integration solutions from various structured and unstructured data sources.
• Design and implement CI / CD pipelines for data engineering projects using industry best practices.
• Monitor and optimize performance of data systems, ensuring high availability and reliability.
• Document data workflows, schemas, and architectures to support knowledge sharing and maintenance.
• Actively participate in code reviews and contribute to a culture of continuous improvement.
• Communicate effectively with technical and non-technical stakeholders, translating business requirements into technical Qualifications :
• 6+ years of hands-on experience in Data Engineering.
• Proficiency in SQL, Python, and PySpark for data processing and transformation.
• Experience working with Azure Data Lake Storage, Azure Data Factory, and Databricks.
• Solid understanding of data warehousing concepts, data modeling, and data integration.
• Experience with CI / CD tools and deployment practices (e.g., Azure DevOps, Git, Jenkins).
• Strong analytical, problem-solving, and debugging skills.
• Excellent verbal and written communication skills.
• Strong team player with the ability to work independently when needed

(ref : hirist.tech)