Data Engineer Ii

ClearDemandSecunderabad, Republic Of India, IN

2 days ago

Job description

Job Summary :

Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.

Key Responsibilities

Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.

Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).

Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.

Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet / Avro file formats for efficient querying and cost management.

Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.

Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.

Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.

Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.

Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.

Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.

Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform’s capabilities

Qualifications & Experience :

4-5+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.

Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.

Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.

Proficiency in web crawling frameworks, including Node.Js, HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.

Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.

Strong understanding of infrastructure as code using Terraform, automated CI / CD pipelines with Jenkins, and event-driven architecture with Kafka.

Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.

Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.

Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.

Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.

Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.

Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

Create a job alert for this search

Data Engineer Ii • Secunderabad, Republic Of India, IN

Related jobs

Promoted

Data Engineer

Optumhyderabad, telangana, in

Support the full data engineering lifecycle including research, proof of concepts, design, development, testing, deployment, and maintenance of data management solutions. Utilize knowledge of variou...Show moreLast updated: 23 days ago

Promoted

Data Engineer Ii

ClearDemandHyderabad, Republic Of India, IN

Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership. You'll play a crucial role in driving the evolution and efficiency of our...Show moreLast updated: 2 days ago

Promoted

Data Engineer III

ConfidentialHyderabad / Secunderabad, Telangana, India

Be part of a dynamic team where your distinctive skills will contribute to a winning culture and team.As a Data Engineer III at JPMorgan Chase within the Consumer and Community Banking, you will be...Show moreLast updated: 5 days ago

Promoted

Data Engineer II, Intl. Seller Growth

AmazonHyderabad, Telangana, India

This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.DESCRIPTION : Come...Show moreLast updated: 10 days ago

Promoted

Data Engineer III [T500-19720]

McDonald'shyderabad, telangana, in

One of the world’s largest employers with locations in more than 100 countries, McDonald’s Corporation has corporate opportunities in Hyderabad. Our global offices serve as dynamic innovation and op...Show moreLast updated: 23 days ago

Promoted

Data Engineer

Prudent Technologies and Consulting, Inc.Hyderabad, Telangana, India

Job Summary : Seeking an experienced Senior Data Engineer with over 9 years of industry experience to architect, build, and manage advanced data pipelines and platforms in a cloud environment.The id...Show moreLast updated: 30+ days ago

Promoted

Data Engineer II

ClearDemandHyderabad, IN

Promoted

Data Science Engineer - II [LLM]

Tanla Platforms Limitedhyderabad, telangana, in

We are seeking an experienced AI / ML Engineer to design and implement cutting-edge solutions in the field of Generative AI and Large Language Models (LLMs). This role involves leading the development...Show moreLast updated: 21 days ago

Promoted

Data Engineer Iii T500-19720

McDonald'sHyderabad, Republic Of India, IN

Promoted

Senior Data Engineer (AWS / Databricks)

Accoladehyderabad, telangana, in

The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago

Promoted

Data Engineer II

AmazonHyderabad, Telangana, India

Promoted

Data Engineer II - ETL

SystechcorpHyderabad

Responsibilities : - Develop, optimize, and maintain ETL processes using Teradata.Write and troubleshoot complex Teradata SQL and BTEQ scripts for data extract...Show moreLast updated: 20 days ago

Promoted

Data Science Engineer - Ii Llm

Tanla Platforms LimitedHyderabad, Republic Of India, IN

Promoted

Celebal Technologies - AWS Data Engineer

Celebal TechnologiesHyderabad

DATA ENGINEER (Databricks & AWS) Overview : As a Data Engineer, you will work with multiple teams to deliver solutions on the AWS Cl...Show moreLast updated: 30+ days ago

Promoted

Data Engineer II

ConfidentialHyderabad / Secunderabad, Telangana, India

You thrive on diversity and creativity, and we welcome individuals who share our vision of making a lasting impact.Your unique combination of design thinking and experience will help us achieve new...Show moreLast updated: 5 days ago

Promoted

Data Engineer - Associate II

ConfidentialHyderabad / Secunderabad, Telangana, India

Vialto Partners is a market leader in global mobility services.Our purpose is to 'Connect the world'.We are unique and the only stand-alone global mobility business. This presents a rare opportunity...Show moreLast updated: 5 days ago

Promoted

Data Engineer II

McDonalds in IndiaHyderabad, India

Builds and maintains relevant and reliable data products that support Finance Analytics.Develops and implements new technology solutions as needed to ensure ongoing improvement with data reliabilit...Show moreLast updated: 30+ days ago

Promoted

Data Engineer II, Close Systems

AmazonHyderabad, Telangana, India