Data Engineer

Description :

• Design, develop, and maintain scalable data pipelines using Azure Databricks, Apache Spark, and Delta Lake.
• Implement data ingestion, transformation, and integration from multiple structured and unstructured data sources.
• Optimize ETL / ELT processes for performance, cost-efficiency, and scalability in Azure environments.
• Build and manage data models, ensuring data quality, consistency, and lineage across systems.
• Collaborate with cross-functional teams (data scientists, analysts, business users) to understand requirements and deliver data solutions.
• Integrate Databricks with other Azure services such as Azure Data Lake Storage (ADLS), Synapse Analytics, Azure SQL Database, and Azure Data Factory.
• Develop and manage notebooks, jobs, and clusters within Azure Databricks for batch and streaming workloads.
• Implement CI / CD pipelines for data workflows using tools like Azure DevOps or GitHub Actions.
• Monitor, debug, and tune performance of Spark jobs and clusters.
• Ensure compliance, data security, and governance following organizational and regulatory standards.
• Document technical designs, processes, and best practices.

Required Skills and Qualifications :

Preferred Skills :

Key Attributes :

(ref : hirist.tech)