Job Title : Senior Data Engineer - Azure | ADF | Databricks | PySpark | AWS
Location : Bangalore, Hyderabad, Chennai (Hybrid Mode)
Experience Required : 5+ Years (Budget around 25 LPA)
Employment Type : Full-Time
Industry : IT Services & Consulting / Data & Analytics
Notice Period : Immediate to 30 days preferred
Job Description :
We are looking for a Senior Data Engineer who is passionate about designing and developing scalable data pipelines, optimizing data architecture, and working with advanced big data tools and cloud platforms. This is a great opportunity to be a key player in transforming data into meaningful insights by leveraging modern data engineering practices on Azure, AWS, and Databricks.
You will be working with cross-functional teams including data scientists, analysts, and software engineers to deliver robust data solutions. The ideal candidate will be technically strong in Azure Data Factory, PySpark, Databricks, and AWS services, and will have experience in building end-to-end ETL workflows and driving business impact through data.
Key Responsibilities :
- Design, build, and maintain scalable and reliable data pipelines and ETL workflows
- Implement data ingestion and transformation using Azure Data Factory (ADF) and Azure Databricks (PySpark)
- Work across multiple data platforms including Azure, AWS, Snowflake, and Redshift
- Collaborate with data scientists and business teams to understand data needs and deliver solutions
- Optimize data storage, processing, and retrieval for performance and cost-effectiveness
- Develop data quality checks and monitoring frameworks for pipeline health
- Ensure data governance, security, and compliance with industry standards
- Lead code reviews, set data engineering standards, and mentor junior team members
- Propose and evaluate new tools and technologies for continuous improvement
Must-Have Skills :
Strong programming skills in Python, SQL, or ScalaAzure Data Factory, Azure Databricks, Synapse AnalyticsHands-on with PySpark, Spark, Hadoop, HiveExperience with cloud platforms (Azure preferred; AWS / GCP acceptable)Data Warehousing : Snowflake, Redshift, BigQueryStrong ETL / ELT pipeline development experienceWorkflow orchestration tools such as Airflow, Prefect, or LuigiExcellent problem-solving, debugging, and communication skillsNice to Have :
Experience with real-time streaming tools (Kafka, Flink, Spark Streaming)Exposure to data governance tools and regulations (GDPR, HIPAA)Familiarity with ML model integration into data pipelinesContainerization and CI / CD exposure : Docker, Git, Kubernetes (basic)Experience with Vector databases and unstructured data handlingTechnical Environment :
Programming : Python, Scala, SQLBig Data Tools : Spark, Hadoop, HiveCloud Platforms : Azure (ADF, Databricks, Synapse), AWS (S3, Glue, Lambda), GCPData Warehousing : Snowflake, Redshift, BigQueryDatabases : PostgreSQL, MySQL, MongoDB, CassandraOrchestration : Apache Airflow, Prefect, LuigiTools : Git, Docker, Azure DevOps, CI / CD pipelinesSoft Skills :
Strong analytical thinking and problem-solving abilitiesExcellent verbal and written communicationCollaborative team player with leadership qualitiesSelf-motivated, organized, and able to manage multiple projectsEducation & Certifications
Bachelor's or Master's Degree in Computer Science, IT, Engineering, or equivalentCloud certifications (e.g., Microsoft Azure Data Engineer, AWS Big Data) are a plusKey Result Areas (KRAs) :
Timely delivery of high-performance data pipelinesQuality of data integration and governance complianceBusiness team satisfaction and data readinessProactive optimization of data processing workloadsKey Performance Indicators (KPIs) :
Pipeline uptime and performance metricsReduction in overall data latencyZero critical issues in production post-releaseStakeholder satisfaction scoreNumber of successful integrations and migrationsref : hirist.tech)