Data Engineer

Straivehyderabad, telangana, in

30+ days ago

Job description

Key Responsibilities :

Design, build and maintain scalable batch and (optionally) streaming data pipelines using Apache Spark 3.x, Scala, Python & SQL
Implement core ETL / ELT logic in Scala and Python; author efficient Spark DataFrame / Dataset jobs.
Write and optimize complex SQL for ingestion, transformation and consumption layers.
Tune Spark jobs for performance and cost : partitioning, join strategies, broadcast, memory / tuning, shuffle reduction.
Ensure code quality via unit tests, integration tests, CI / CD and code reviews.
Work with data modeling, schema evolution and data quality checks to ensure reliable outputs.
Collaborate with platform / DevOps teams to deploy and monitor pipelines (retries, logging, alerting).
Troubleshoot production issues and perform root-cause analysis.
Mentor and guide junior engineers, share best practices and drive improvements to the data platform.

Qualifications :

5–8 years industry experience as a Data Engineer.

Strong hands-on experience with Apache Spark 3.x (development, tuning, debugging).

Proficient in Scala programming for Spark ETL / ELT development (datasets, RDDs when required).

Experience with streaming frameworks like Spark Structured Streaming, Kafka .

Proficient in Python programming .

Advanced SQL skills (complex queries, window functions, CTEs, query optimization).

Experience building production-grade ETL / ELT workflows, testing and monitoring them.

Experience with orchestration tools like Apache Airflow

Strong software engineering fundamentals : version control (Git), modular code, code reviews, CI / CD pipelines.

Experience with cloud object storage and common data formats (Parquet / ORC / Avro).

Strong problem solving, communication and collaboration skills.

Experience with AWS Glue, EMR, Athena, Redshift, S3 will be a plus.

Data Engineer • hyderabad, telangana, in