Data EngineerStraive • Hyderabad, India
Data Engineer
Straive • Hyderabad, India
8 days ago
Job description5–8 years industry experience as a Data Engineer. Strong hands-on experience with Apache Spark 3.x (development, tuning, debugging). Proficient in Scala programming for Spark ETL / ELT development (datasets, RDDs when required). Experience with streaming frameworks like Spark Structured Streaming, Kafka. Proficient in Python programming. Advanced SQL skills (complex queries, window functions, CTEs, query optimization). Experience building production-grade ETL / ELT workflows, testing and monitoring them. Experience with orchestration tools like Apache Airflow Strong software engineering fundamentals : version control (Git), modular code, code reviews, CI / CD pipelines. Experience with cloud object storage and common data formats (Parquet / ORC / Avro). Strong problem solving, communication and collaboration skills. Experience with AWS Glue, EMR, Athena, Redshift, S3 will be a plus.
Key Responsibilities :
- Design, build and maintain scalable batch and (optionally) streaming data pipelines using Apache Spark 3.x, Scala, Python & SQL
- Implement core ETL / ELT logic in Scala and Python; author efficient Spark DataFrame / Dataset jobs.
- Write and optimize complex SQL for ingestion, transformation and consumption layers.
- Tune Spark jobs for performance and cost : partitioning, join strategies, broadcast, memory / tuning, shuffle reduction.
- Ensure code quality via unit tests, integration tests, CI / CD and code reviews.
- Work with data modeling, schema evolution and data quality checks to ensure reliable outputs.
- Collaborate with platform / DevOps teams to deploy and monitor pipelines (retries, logging, alerting).
- Troubleshoot production issues and perform root-cause analysis.
- Mentor and guide junior engineers, share best practices and drive improvements to the data platform.
Qualifications :
Create a job alert for this search
Data Engineer • Hyderabad, India
Related jobs
Customer Engineer (Data Engineer)
Konecta • Hyderabad, IN
In this role, you will be the technical point of contact for the customer – a trusted partner for their IT leadership – as the primary engineer on the project, working with multifunctional teams bo...Show more
Last updated: 3 days ago • Promoted
Data Engineer
Prudent Technologies and Consulting, Inc. • Hyderabad, Telangana, India
Seeking an experienced Senior Data Engineer with over 9 years of industry experience to architect, build, and manage advanced data pipelines and platforms in a cloud environment.The ideal candidate...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Digitalzone • Hyderabad, IN
As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics.
Develop and maintain scalable ETL / ELT pipelines using Py...Show more
Last updated: 22 days ago • Promoted
Data Engineer - Palantir Foundry
NP Group • secunderabad, India
Data Engineer - Palantir Foundry, Workshop, Pyspark & Typescript.Long Term (initially 6 months) full time contract.We have an immediate requirement for an experienced Data Engineer to join the glob...Show more
Last updated: 12 days ago • Promoted
Data Engineer
Azoca Technologies • Hyderabad, IN
We are seeking an experienced Data Engineer with strong hands-on expertise in Microsoft Fabric to design, build, and maintain scalable data solutions supporting global clients.The ideal candidate w...Show more
Last updated: 9 hours ago • Promoted • New!
Softobiz - Data Engineer - Synapse Analytics
Softobiz Technologies Pvt ltd • Hyderabad
Description : Job Title : Data Engineer.Career-defining role available with Tier 1 Blue Chip Australian Client.As a Data Engineer, you will be involved in all phases...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Atica Global • Hyderabad, IN
Atica is a leading, tech-first remote sales & revenue management company for Hotels and Hotel operators in the US, enabled by a unified tech-led solution.
We are a startup, funded by the top investo...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
HCLTech • Hyderabad, Telangana, India
Data Engineer - Databricks Job Description.The Data Engineer - Databricks will be responsible for building and optimizing our data pipelines, architectures, and data sets.He will work closely with ...Show more
Last updated: 19 days ago • Promoted
Data Engineer - Snowflake
Prudent Technologies and Consulting, Inc. • hyderabad, telangana, in
We are seeking a skilled Data Engineer with strong experience in Python, Snowflake, and AWS.The ideal candidate will be responsible for building and optimizing scalable data pipelines, integrating ...Show more
Last updated: 9 days ago • Promoted
Data Engineer
Recro • Secunderabad, Republic Of India, IN
Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake.
AI Data Enablement : Collaborate wi...Show more
Last updated: 22 days ago • Promoted
Data Engineer
Radiant Digital • Hyderabad, Telangana, India
As a Data Engineer, you will play a key role in designing, developing, and optimizing data pipelines and storage solutions in a complex, enterprise-scale data warehouse environment.You will contrib...Show more
Last updated: 1 day ago • Promoted
Data Engineer
Technogen India Pvt. Ltd. • Hyderabad, India
Design, build, and measure complex ELT jobs to process disparate data sources and form a high integrity, high quality, clean data asset.
Executes and provides feedback for data modeling policies, pr...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Smart Moves Consultants • Hyderabad, Republic Of India, IN
We are looking for candidates with strong Domo and Python / Kafka connect experience.Please find the JD below and share suitable and strong profiles at the earliest.
Design, build, and maintain scala...Show more
Last updated: 20 hours ago • Promoted • New!
Data Engineer
Thermo Fisher Scientific • Hyderabad, Telangana, India
This job is with Thermo Fisher Scientific, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community.
Please do not contact the recruiter directly...Show more
Last updated: 25 days ago • Promoted
Data Engineer
Kyndryl • Hyderabad, Telangana, India
This job is with Kyndryl, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community.
Please do not contact the recruiter directly.Who We Are At Ky...Show more
Last updated: 10 hours ago • Promoted • New!
Data Engineer 2
Confidential • Hyderabad / Secunderabad, Telangana
Data Architecture & Solution Development.Translate business and functional requirements into robust, scalable data solutions aligned with overall architecture.
Design and implement data integrations...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Insight Global • Hyderabad, IN
GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts.
Proficiency in ETL / ELT pipeline development...Show more
Last updated: 30+ days ago • Promoted
Data Engineer
Travel & Payments • Hyderabad, Telangana, India
At Yuno, we are building the payment infrastructure that enables all companies to participate in the global market.Founded by a team of seasoned experts in the payments and IT industries, Yuno prov...Show more
Last updated: 30+ days ago • Promoted