No longer accepting applications

Data Engineer II

ClearDemandBengaluru, IN

6 days ago

Job description

Job Summary :

Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.

Key Responsibilities

Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.

Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).

Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.

Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet / Avro file formats for efficient querying and cost management.

Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.

Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.

Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.

Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.

Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.

Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.

Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform’s capabilities

Qualifications & Experience :

4-5+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.

Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.

Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.

Proficiency in web crawling frameworks, including Node.js, HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.

Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.

Strong understanding of infrastructure as code using Terraform, automated CI / CD pipelines with Jenkins, and event-driven architecture with Kafka.

Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.

Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.

Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.

Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.

Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.

Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

Create a job alert for this search

Data Engineer Ii • Bengaluru, IN

Related jobs

Promoted

Senior Data Engineer – II

MakeMyTripBangalore Urban, Karnataka, India

At MakeMyTrip (MMT), technology is at the heart of everything we do.As a leading player in the travel industry, we leverage cutting-edge solutions like AI, machine learning, and cloud infrastructur...Show moreLast updated: 6 days ago

Promoted

Engineer II - Data Engineering [T500-20681]

lululemonbangalore, karnataka, in

Setting the bar in technical fabrics and functional design, we create transformational products and experiences that support people in moving, growing, connecting, and being well.We owe our success...Show moreLast updated: 30+ days ago

Promoted

Data & Analytics Engineer

APPIT Software Inchosur, tamil nadu, in

Data Engineer : Snowflake -Mandatory – Hands -on Experience.ETL Tool -Informatica [IVS version],BDT.GCP : Big query – Mandatory -handson experience. Data Modelling & Data Warehouse -Mandatory -Hands...Show moreLast updated: 3 days ago

Promoted

Data Engineer II [T500-21346]

ADMBengaluru, Karnataka, India

We are one of the world’s largest nutrition companies and a global leader in human and animal nutrition.We unlock the power of nature to provide nourishing quality of life by transforming crops int...Show moreLast updated: 2 days ago

Promoted

Software Engineer II (Data) [T500-20473]

Best Buy IndiaBengaluru, Karnataka, India

Contribute to the delivery of complex solutions, breaking down big problems into smaller pieces.Actively participate in team planning activities. Help ensure the quality and integrity of the SDLC fo...Show moreLast updated: 5 days ago

Promoted

Data Engineer

Recrohosur, tamil nadu, in

Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer

Pro5.aihosur, tamil nadu, in

You’ll play a key role in ensuring reliable data flows, integrations, and preprocessing frameworks that power advanced GenAI systems. This is a high-impact opportunity within an.Design, build, and o...Show moreLast updated: 18 hours ago

Promoted

Data Engineer

thinkbridgehosur, tamil nadu, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

AWS Data Engineer

Tata Consultancy Serviceshosur, tamil nadu, in

TCS has always been in the spotlight for being adept in the next big technologies.What we can offer you is a space to explore varied technologies and quench your techie soul.Required Skill Set : AWS...Show moreLast updated: 14 days ago

Promoted

Data Engineer

Insight Globalhosur, tamil nadu, in

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer (AWS / Databricks)

AccoladeBengaluru, Karnataka, India

The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Tata Consultancy Serviceshosur, tamil nadu, in

Required Technical Skill Set -.Create and maintain optimal data pipeline architecture,.Assemble large, complex data sets that meet functional / non-functional business requirements.Identify, design...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Veraxionhosur, tamil nadu, in

Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions. Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 16 days ago

Promoted

Senior Data Engineer

Mastekhosur, tamil nadu, in

We are looking for a Senior Data Engineer with good exposure to Cloud Services in AWS(IAM, RDS, Lambda, DMS) along with Snowflake / Postgresql, ETL and Streaming Services like Kafka.Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer

meanSquare.aihosur, tamil nadu, in

Data Engineer (onsite | Offshore | PST Overlap Required).We’re looking for an experienced and independent.If you enjoy working with modern data tools, solving real-world data problems, and collabor...Show moreLast updated: 2 hours ago

Promoted

Data Engineer

Response Informaticshosur, tamil nadu, in

AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture. CICD - Terraform or CloudFo...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

IntraEdgehosur, tamil nadu, in

Python, PySpark, AWS services (Glue, Lambda), and Snowflake.The ideal candidate will design, build, and maintain scalable data pipelines, ensure efficient data integration, and enable advanced anal...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

DraconXhosur, tamil nadu, in

DraconX specializes in creating intelligent, scalable digital solutions that drive growth and innovation for startups and enterprises. As pioneers in AI business automation and SaaS platforms, we ex...Show moreLast updated: 1 day ago