Talent.com
No longer accepting applications
Lead Databricks Engineer

Lead Databricks Engineer

KoantekHyderabad, Republic Of India, IN
2 days ago
Job description

We are seeking an experienced Databricks Full Stack Data Engineer with 3-6 years of industry

experience. The ideal candidate will have a proven track record of working on live projects.

He / she will play a key role in developing and maintaining scalable data solutions using

Databricks and related technologies.

Key Responsibilities :

  • Develop, and deploy end-to-end data pipelines and solutions on Databricks,

integrating with various data sources and systems.

  • Collaborate with cross-functional teams to understand data, and deliver effective BI

    • solutions.

  • Implement data ingestion, transformation, and processing workflows using Spark

    • (PySpark / Scala), SQL, and Databricks notebooks.

  • Develop and maintain data models, ETL / ELT processes ensuring high performance,

    • reliability, scalability and data quality.

  • Build and maintain APIs and data services to support analytics, reporting, and

    • application integration.

  • Ensure data quality, integrity, and security across all stages of the data lifecycle.
  • Monitor, troubleshoot, and optimize pipeline performance in a cloud-based environment.
  • Integrate data from various sources, including APIs, relational and non-relational

    • databases, IoT devices, and external data providers.

  • Ensure adherence to data governance, security, and compliance policies.

    • Required Skills and Experience :

  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
  • 3-6 years of hands-on experience in data engineering, with a strong focus on

    • Databricks and Apache Spark.

  • Strong programming skills in Python / PySpark and / or Scala, with a deep understanding

    • of Apache Spark.

  • Experience with Azure Databricks.
  • Strong SQL skills for data manipulation, analysis, and performance tuning.
  • Strong understanding of data structures and algorithms, with the ability to apply them to

    • optimize code and implement efficient solutions.

  • Strong understanding of data architecture, data modeling, ETL / ELT processes, and

    • data warehousing concepts.

  • Experience building and maintaining ETL / ELT pipelines in production environments.
  • Familiarity with Delta Lake, Unity Catalog, or similar technologies.
  • Experience working with structured and unstructured data, including JSON, Parquet,

    • Avro, and time-series data.

  • Familiarity with CI / CD pipelines and tools like Azure DevOps, version control (Git), and

    • DevOps practices for data engineering.

  • Excellent problem-solving skills, attention to detail, and ability to work independently or

    • as part of a team.

  • Strong communication skills to interact with technical and non-technical stakeholders.

    • Preferred Qualifications :

  • Experience with Delta Lake and Databricks Workflows.
  • Exposure to real-time data processing and streaming technologies (Kafka, Spark

    • Streaming).

  • Exposure to data visualization tool Databricks Genie for data analysis and reporting.
  • Knowledge of data governance, security, and compliance best practices.

    • Must Have Skills : Spark, Migration Experience, ETL

    Create a job alert for this search

    Lead Engineer • Hyderabad, Republic Of India, IN