Data Engineer II

ClearDemandvapi, gujarat, in

1 day ago

Job description

Job Summary :

Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.

Key Responsibilities

Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.

Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).

Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.

Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet / Avro file formats for efficient querying and cost management.

Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.

Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.

Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.

Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.

Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.

Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.

Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform’s capabilities

Qualifications & Experience :

4-5+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.

Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.

Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.

Proficiency in web crawling frameworks, including Node.js, HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.

Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.

Strong understanding of infrastructure as code using Terraform, automated CI / CD pipelines with Jenkins, and event-driven architecture with Kafka.

Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.

Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.

Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.

Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.

Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.

Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

Create a job alert for this search

Data Engineer Ii • vapi, gujarat, in

Related jobs

Promoted

AWS Data Engineer

Tata Consultancy Servicesvapi, gujarat, in

Aws data engineer having strong experience of Python.Technical / Behavioral Competency.Proficient in Python, with experience in deploying Python packages and OOP, Experience in ingesting data from di...Show moreLast updated: 10 days ago

Promoted

Lead Data Engineer

Cimpressvapi, gujarat, in

Our Team : Enterprise Business Solutions.Vista’s Enterprise Business Solutions (EBS) domain is working to make our company one of the most data-driven organizations to support Finance, Supply Chain,...Show moreLast updated: 1 day ago

Promoted

Data Engineer

Digitalzonevapi, gujarat, in

As a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics. Develop and maintain scalable ETL / ELT pipelines using Py...Show moreLast updated: 1 day ago

Promoted

Data Engineer

Philodesign Technologies Incvapi, gujarat, in

The ideal candidate will have strong hands-on expertise in.Azure Data Factory (ADF), Synapse, and Databricks.ETL, SQL, and data warehousing concepts. Design, develop, and implement ETL solutions usi...Show moreLast updated: 1 day ago

Promoted

Azure Data Engineer

LTIMindtreevapi, gujarat, in

Role Senior Data Engineer 8 years of experience.Build reusable utilities templates and automation pipelines.Design scalable data engineering frameworks standards and best practices.Provide architec...Show moreLast updated: 30+ days ago

Promoted

Senior Data Engineer (AWS / Databricks)

Accoladevapi, gujarat, in

The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago

Promoted

AWS Data Engineer

TerraGiGvapi, gujarat, in

Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR. Writing reusable, testable, and efficient code.Integration of data storage ...Show moreLast updated: 12 days ago

Promoted

Data Engineer

LTIMindtreevapi, gujarat, in

We are seeking a skilled Data Engineer with hands-on experience in.The ideal candidate will be responsible for developing scalable data pipelines, transforming and ingesting large-scale data, and e...Show moreLast updated: 30+ days ago

Promoted

Data Engineer(Azure / AWS, Python / Pyspark, SQL)

Sail Analyticsvapi, gujarat, in

Architect, develop, test and maintain scalable data warehouses and data pipelines.Expertise in SQL, PySpark / Python and Azure(ADB, ADF) or AWS(Glue, Lambda, Redshift). Bachelor's degree or equivalent...Show moreLast updated: 30+ days ago

Promoted

Data Engineer – Azure & AWS

Yoda Techvapi, gujarat, in

We are seeking a skilled and motivated Data Engineer to join our team and help build scalable, secure, and efficient data pipelines and platforms. The ideal candidate will have 2 to 4 years of hands...Show moreLast updated: 1 day ago

Promoted

Data Engineer

thinkbridgevapi, gujarat, in

We are a global digital product development firm that helps growth-stage companies gain the technology sophistication and maturity of leading modern digital businesses. We differentiate ourselves by...Show moreLast updated: 30+ days ago

Promoted

AI / ML & Data Engineer

Mindfire Solutionsvapi, gujarat, in

We are looking for an experienced AI / ML & Data Engineer to design, develop, and deploy scalable machine learning models and data infrastructure on AWS. You will work closely with cross-functional te...Show moreLast updated: 14 days ago

Promoted

Data Engineer

Insight Globalvapi, gujarat, in

GCP DATA ENGINEER - Contract (Long term).Data Engineer with hands-on support for Google Looker.Strong experience in data modeling and building data marts. Proficiency in ETL / ELT pipeline development...Show moreLast updated: 30+ days ago

Promoted
New!

Data Engineer

BeGigvapi, gujarat, in

Work Mode- Remote(6 days working in a week).You’ll design scalable data models, automate ETL / ELT workflows, and collaborate on ML deployments in a fast-paced startup environment.Design and maintain...Show moreLast updated: 11 hours ago

Promoted

Senior Data Engineer

Mastekvapi, gujarat, in

We are looking for a Senior Data Engineer with good exposure to Cloud Services in AWS(IAM, RDS, Lambda, DMS) along with Snowflake / Postgresql, ETL and Streaming Services like Kafka.Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Recrovapi, gujarat, in

Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago

Promoted

Data Engineer

Dexian Indiavapi, gujarat, in

Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Constructing infrastructure for efficient ETL processes from vari...Show moreLast updated: 1 day ago

Promoted
New!

AWS Data Engineer / Snowflake Data Engineer

Numeric Technologiesvapi, gujarat, in

Please apply only if you are comfortable to work in rotational shift.Apply only if you are an immediate to 15 days joiner. Work Mode - Monthly once to office in Bangalore.Years of experience - 2 to ...Show moreLast updated: 11 hours ago