This job offer is not available in your country.

Data Engineer

ACL Digitalthane, maharashtra, in

30+ days ago

Job description

Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.

Design, develop, and optimize AWS pipeline as applicable.

Implement and manage GitHub asset bundles for version control, CI / CD workflows, and collaborative development.

Leverage Databricks capabilities including Delta Lake, MLflow, and notebooks to build robust data solutions.

Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-ready datasets.

Ensure data quality, integrity, and security across all stages of the pipeline.

Monitor and troubleshoot performance issues in Spark jobs and Databricks clusters.

Document technical designs, workflows, and best practices.

Required Qualifications

Minimum 5+ years of hands-on experience with Spark in a production environment.
Strong proficiency in Databricks platform and ecosystem.
Solid understanding of GitHub asset bundle management, including branching strategies, pull requests, and release workflows.
Proficient in Python, Scala, or SQL for data engineering tasks.
Experience with Delta Lake, Structured Streaming, and Databricks Workflows.
Familiarity with cloud platform (AWS) and AWS data services.
Excellent problem-solving and communication skills.

Data Engineer • thane, maharashtra, in