Databricks EngineerTTC Group • Chennai, IN

Databricks Engineer

TTC Group • Chennai, IN

3 days ago

Job description

Role : Databricks Engineer

Position : Remote

Type : Contract

Duration : 3 - 6 months (extendable)

Role Overview

We are seeking a Mid-Level Databricks Engineer with strong data engineering fundamentals and hands-on experience building scalable data pipelines on the Databricks platform. The ideal candidate will be comfortable working with distributed data processing, designing Delta Lake architectures, implementing CI / CD for data workloads, and optimizing Spark jobs for performance and reliability.

Required Skills

Strong hands-on experience in building data pipeline using scripting language includes Python, PySpark and SQL
Strong working experience with Databricks (Unity Catalog is plus) and Azure Data Factory (ADF)
Hands-on working experience with Azure DevOps Repo and CI / CD
Strong understanding of modern Data Architecture, Data Warehousing concepts, Data Modelling and ETL / ELT processes
Understanding of Agile ways of working
Any experience working in DBT is a plus
Working with Retail Domain is a plus

Platform Engineering & Operations

Configure and optimize Databricks clusters, including autoscaling, cluster sizing, and runtime selection.

Manage databases, schemas, and Delta Lake tables, including vacuuming, Z-ordering, partitioning, and optimization.

Work with catalogs, Unity Catalog (UC), and access controls for secure data governance.

Monitor and troubleshoot performance issues across jobs, clusters, and workflows.

CI / CD & DevOps

Implement CI / CD pipelines for Databricks workflows using GitHub Actions / Azure DevOps / GitLab.

Work with Databricks Repos and Git workflows (branching, PRs, code reviews).

Version and promote data pipelines across environments.

Cloud & Integration

Integrate Databricks with cloud services such as :

Azure : ADLS Gen2, Key Vault, ADF, Event Hub, Synapse

AWS : S3, Glue, IAM, Lambda, Kinesis (depending on cloud environment)

Work with data ingestion frameworks, messaging systems, and event-driven architectures.

Quality, Testing & Documentation

Implement unit tests and integration tests using pytest, SQL tests, or dbx testing frameworks.

Maintain pipeline documentation, data dictionaries, and technical design artifacts.

Ensure high data quality, reliability, and observability.

Create a job alert for this search

Engineer • Chennai, IN

Related jobs

Data Engineer

Recro • Chennai, IN

Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement : Collaborate wi...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

EXL • Chennai, Tamil Nadu, India

Collaborate with project stakeholders (client) to identify product and technical requirements.Develop, implement, and tune large-scale distributed systems and pipelines that process large volume of...Show more

Last updated: 30+ days ago • Promoted

ETL & Tableau Data Engineer

Magma Consultancy • chennai, tamil nadu, in

ETL & Tableau Data Engineer (Remote | Full-Time | Join in 2 Weeks).We are a forward-thinking technology and data consulting firm dedicated to helping organizations unlock the full potential of thei...Show more

Last updated: 3 days ago • Promoted

Data Engineer

Mastek • Chennai, Chennai (district)

Work closely with our data science team to help build complex algorithms that provide unique insights into our data.Use agile software development processes to make iterative improvements to our ba...Show more

Last updated: 2 days ago • Promoted

Azure Databricks

Kumaran Systems • Chennai, IN

We are seeking a skilled Azure Databricks with strong hands-on experience in Microsoft Azure services such as Data Factory, Storage, Synapse, and Key Vault. The candidate should be proficient in Dat...Show more

Last updated: 5 days ago • Promoted

Senior Data Engineer

SGS & Co • Chennai, Tamil Nadu, India

Position Title : Senior Data Engineer.Experience Required : 8 to 12 Years.We are looking for a highly skilled and experienced Data Engineer with strong expertise in. The ideal candidate will play a ke...Show more

Last updated: 19 days ago • Promoted

Data Engineer

Raretech • Chennai, Tamil Nadu, India

RareTech is a leading IT Products and Services company catering to the needs of clients in Bahrain and the GCC region.The company focuses on providing tailored solutions, leveraging its expertise i...Show more

Last updated: 10 hours ago • Promoted • New!

Senior Data Engineer

Mindlance • Chennai, IN

We’re looking for a strong, hands-on Sr Data Engineer who can independently drive client conversations, collaborate with SMEs, and deliver high-quality data solutions in a cloud-native environment....Show more

Last updated: 3 days ago • Promoted

AWS Data Engineer

Tata Consultancy Services • chennai, tamil nadu, in

Aws data engineer having strong experience of Python.Technical / Behavioral Competency.Proficient in Python, with experience in deploying Python packages and OOP, Experience in ingesting data from di...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

Insight Global • Chennai, IN

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

MyData Insights Pvt Ltd • Chennai, IN

The ideal candidate has a strong background in building scalable data solutions on.CI / CD pipelines, infrastructure automation using Terraform, and cloud-native data engineering practices.Design, bu...Show more

Last updated: 3 days ago • Promoted

Azure Databricks+Pyspark

Tata Consultancy Services • Chennai, IN

TCS is conducting Virtual Drive for Azure Data Engineer + Pyspark.Years of Experience : 6 to 10yrs.Location : Bangalore & Hyderabad. Kindly apply if there is a skill match with 6+yrs of experience.Ove...Show more

Last updated: 19 days ago • Promoted

Azure Databricks Engineer

TechMantra Global • Chennai

Role : Azure Databricks.Location : Any Hyderabad, Chennai , Bangalore , Pune , Noida, Gurgaon are preferred.Skills : Azure, Notebooks, Spark-Scala, Bigdata, Cloudera P...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

loanDNA • Chennai, Tamil Nadu, India

We are seeking an experienced Snowflake Developer with a minimum of 5 years of hands-on experience in Snowflake development, integration, migration, and deployment. The ideal candidate will be respo...Show more

Last updated: 16 days ago • Promoted

Data Engineer

JIT Global Infosystems Pvt. Ltd. • Chennai, Tamil Nadu, India

Applications, Analytics, and Consulting services to clients worldwide.By combining technological expertise with deep domain knowledge, we ensure tangible improvements in business performance.Our ex...Show more

Last updated: 2 days ago • Promoted

Data Engineer

Tata Consultancy Services • Chennai, India

Role- Data Engineer – Databricks & Integration Pipelines.Design and implement data pipelines in Azure Databricks for ingesting and transforming data from upstream systems (SAP, PIPS, POS) into WF...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

IntraEdge • Chennai, India

Work Mode : Onsite / Hybrid (as applicable).We are looking for a highly skilled and motivated Data Engineer with 3 to 7 years of experience to join our growing team in Chennai.The ideal candidate wi...Show more

Last updated: 30+ days ago • Promoted

Data Engineer

JMAN Group • Chennai, India

JMAN Group is a fast-growing data engineering & data science consultancy.We work primarily with Private Equity Funds and their Portfolio Companies to create commercial value using Data & Artificial...Show more

Last updated: 30+ days ago • Promoted