Talent.com
Data Engineer - Databricks Specialist

Data Engineer - Databricks Specialist

Tata TechnologiesRepublic Of India, IN
1 day ago
Job description

Purpose of the Role :

We are seeking a skilled Databricks with 4 to 6 years of experience in data engineering and cloud platforms. Designs and implements scalable data architectures and solutions. The jobholder has expertise in Databricks Lakehouse, data modelling, and cloud integration, ensuring high performance, security, and reliability.

Job Responsibilities :

  • Design and implement Databricks-based data architectures to meet business requirements.
  • Develop and optimize data pipelines using PySpark, Scala, or SQL.
  • Establish the Databricks Lakehouse architecture for batch and streaming data.
  • Collaborate with cross-functional teams to integrate Databricks with cloud platforms (e.G., AWS, Azure, GCP).
  • Ensure data security and compliance with best practices.
  • Monitor and troubleshoot Databricks environments for performance and reliability.
  • Stay updated on Databricks advancements and industry trends.

Desired Candidate Profile :

  • 4 to 6 years of experience in data engineering using Databricks or Apache Spark-based platforms.
  • Proven track record of building and optimizing ETL / ELT pipelines for batch and streaming data ingestion.
  • Good understanding of AWS cloud services
  • Proficiency in programming languages such as Python, Scala, or SQL for data processing and transformation.
  • Expertise in Spark (PySpark, Spark SQL, or Scala) and Databricks notebooks for large-scale data processing.
  • Familiarity with Delta Lake, Delta Live Tables, and medallion architecture for data lake house implementations.
  • Build and query delta lake storage solutions
  • Process streaming data with Azure Databricks structured streaming
  • Design Azure Databricks security and data protection solutions
  • Flatten nested structures and explode arrays with spark
  • Experience with orchestration tools like Databricks Jobs for scheduling and automation.
  • Knowledge of Git for source control and CI / CD integration for Databricks workflows, cost optimization, performance tuning.
  • Familiarity with Unity Catalog, RBAC, or enterprise-level Databricks setups.
  • Ability to create reusable components, templates, and documentation to standardize data engineering workflows.
  • Solutioning and presales - Architecting frameworks, defining roadmaps, and engaging with stakeholders.
  • Experience in defining data strategy, evaluating new tools / technologies, and driving adoption across the organization.
  • Must have experience of working with streaming data sources and Kafka.

    • Preferred Experience :

  • Bachelor’s degree in computer science, Information Technology, or related field
  • Proven experience as a Databricks Architect or similar role
  • Complete knowledge in Azure Databricks platform architecture
  • Databricks certification (e.G., Certified Data Engineer, Associate Developer)
  • Expertise in Python / Scala / SQL / R
  • Experience with cloud platforms like AWS, Azure, or GCP
  • Strong understanding of data modelling and cloud integration
    • Create a job alert for this search

    Data Engineer • Republic Of India, IN