Talent.com
This job offer is not available in your country.
Skan.ai - Data Engineer - Azure Databricks

Skan.ai - Data Engineer - Azure Databricks

Skan aiBangalore
30+ days ago
Job description

Job Summary :

We are seeking a skilled Data Engineer with 3 to 5 years of experience in building scalable data pipelines and solutions, with strong hands-on expertise in Databricks.

The ideal candidate should be proficient in working with large-scale data processing frameworks and have a solid understanding of Delta Lake, PySpark, and cloud-based data platforms.

Key Responsibilities :

  • Design, build, and maintain robust ETL / ELT pipelines using Databricks (PySpark / SQL).
  • Develop and optimize data workflows and pipelines on Delta Lake and Databricks Lakehouse architecture.
  • Integrate data from multiple sources, ensuring data quality, reliability, and performance.
  • Collaborate with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
  • Monitor and troubleshoot production data pipelines; ensure performance and cost optimization.
  • Work with DevOps teams for CI / CD integration and automation of Databricks jobs and notebooks.
  • Maintain metadata, documentation, and versioning for data pipelines and assets.

Required Skills :

  • 3- 5 years of experience in data engineering or big data development.
  • Strong hands-on experience with Databricks (Notebook, Jobs, Workflows).
  • Proficiency in PySpark, Spark SQL, and Delta Lake.
  • Experience working with Azure or AWS (preferably Azure Data Lake, Blob Storage, Synapse, etc.
  • Strong SQL skills for data manipulation and analysis.
  • Familiarity with Git, CI / CD pipelines, and job orchestration tools (e.g, Airflow, Databricks Workflows).
  • Understanding of data modeling, data warehousing, and data governance best practices.

    • Preferred Qualifications :

  • Databricks certification (e.g, Databricks Certified Data Engineer Associate).
  • Experience with Power BI, Snowflake, or Synapse Analytics is a plus.
  • Exposure to streaming data pipelines (e.g, using Kafka, Structured Streaming).
  • Understanding of cost optimization and performance tuning in Databricks

    • (ref : hirist.tech)

    Create a job alert for this search

    Azure Data Engineer • Bangalore