Senior Data Engineer

Job Description

We are seeking an experienced Full Stack Data Engineer with 5–6 years of industry

experience. The ideal candidate will have a proven track record of working on live projects,

preferably within the manufacturing or energy sectors. He / she will play a key role in developing,

and maintaining scalable data solutions using Databricks and related technologies.

Key Responsibilities :

• Develop, and deploy end-to-end data pipelines and solutions on Databricks, integrating with
• various data sources and systems.
• Collaborate with cross-functional teams to understand data, and deliver effective BI
• solutions.
• Implement data ingestion, transformation, and processing workflows using Spark
• (PySpark / Scala), SQL, and Databricks notebooks.
• Develop and maintain data models, ETL / ELT processes ensuring high performance,
• reliability, scalability and data quality.
• Build and maintain APIs and data services to support analytics, reporting, and application
• integration.
• Ensure data quality, integrity, and security across all stages of the data lifecycle.
• Monitor, troubleshoot, and optimize pipeline performance in a cloud-based environment.
• Write clean, modular, and well-documented Python / Scala / SQL / PySpark code.
• Integrate data from various sources, including APIs, relational and non-relational databases,
• IoT devices, and external data providers.
• Ensure adherence to data governance, security, and compliance policies.

Required Skills and Experience :

Requirements

Requirements

Experience with Delta Lake and Databricks Workflows. Exposure to real-time data processing and streaming technologies (Kafka, Spark Streaming). Exposure to data visualization tool Databricks Genie for data analysis and reporting. Knowledge of data governance, security, and compliance best practices.