Data Engineer II
ClearDemandKalyan-Dombivli, IN21 hours ago
Job descriptionLead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more. Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO). Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka. Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet / Avro file formats for efficient querying and cost management. Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance. Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices. Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment. Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency. Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health. Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase. Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform’s capabilities 4-5+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines. Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases. Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean. Proficiency in web crawling frameworks, including Node.js, HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction. Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems. Strong understanding of infrastructure as code using Terraform, automated CI / CD pipelines with Jenkins, and event-driven architecture with Kafka. Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC. Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale. Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments. Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments. Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems. Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.
Job Summary :
- Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
- You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.
Key Responsibilities
Qualifications & Experience :
Create a job alert for this search
Data Engineer Ii • Kalyan-Dombivli, IN
Related jobs
- Promoted
- New!
Data Engineer II
ClearDemandThane, IN Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our...Show moreLast updated: 21 hours ago
- Promoted
Data Engineer
RecroThane, IN Data Pipeline Engineering : Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake.
AI Data Enablement : Collaborate wi...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Straivemumbai, maharashtra, in Data Engineer will be responsible for designing, developing, and maintaining data pipelines and architectures that support the organization's analytics and reporting needs.The ideal candidate will ...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer - Snowflake / AWS
ResourcetreeMumbai About the Role : A "Senior Data Engineer" is mid-level professional leading the design, build and evolution of the inhouse data platforms.You lead the const...Show moreLast updated: 27 days ago
- Promoted
Data Engineer – Azure & AWS
Yoda TechKalyan-Dombivli, IN We are seeking a skilled and motivated Data Engineer to join our team and help build scalable, secure, and efficient data pipelines and platforms.
The ideal candidate will have 2 to 4 years of hands...Show moreLast updated: 9 days ago
- Promoted
Data Engineer
Veraxionmumbai city, India Python, Spark, DBT, and AWS-native services.Agile environment to deliver scalable, secure, and high-performance data solutions.
Python, DBT, and AWS services (Data Ops Live).Deliver end-to-end data ...Show moreLast updated: 11 days ago
- Promoted
AWS Data Engineer
Tata Consultancy ServicesDombivali, Maharashtra, India Role • • - AWS Data Engineer Technical Skill Set -Aws data engineer having strong experience of Python Experience Range -6 to 8 Technical / Behavioral Competency 1.
Proficient in Python, with exper...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Response Informaticsthane, maharashtra, in AWS services : Must be proficient in building scalable data pipelines and managing cloud-native ETL workflows.Snowflake : Moderate understanding of Snowflake architecture.
CICD - Terraform or CloudFo...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Envuthane, maharashtra, in At Envu, we partner with our customers to design world-class, forward-thinking innovations that protect and enhance the health of environments around the world.
We offer dedicated services in : Profe...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Fornaxmumbai, maharashtra, in If solving business challenges drives you.Fornax is a team of cross-functional individuals who solve critial business challenges using core concept of analytics, critical thinking.We are seeking a ...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer II
ConfidentialMumbai Data Engineer I - Data Engineering.We make foodthe world loves : 100 brands.With iconic brands like Cheerios, Pillsbury, Betty Crocker, Nature Valley, and Häagen-Dazs, we've been serving up food the...Show moreLast updated: 4 days ago
- Promoted
Data Engineer III
ConfidentialMumbai, India Be part of a dynamic team where your distinctive skills will contribute to a winning culture and team.As a Data Engineer III at JPMorgan Chase within the Asset & Wealth Management, you serve as a s...Show moreLast updated: 4 days ago
- Promoted
AI / ML & Data Engineer
Mindfire SolutionsKalyan-Dombivli, IN We are looking for an experienced AI / ML & Data Engineer to design, develop, and deploy scalable machine learning models and data infrastructure on AWS.
You will work closely with cross-functional te...Show moreLast updated: 13 days ago
- Promoted
- New!
Snowflake Data Engineer
Newpage SolutionsThane, IN Location : Remote | Type : Contract.Newpage Solutions is a global digital health innovation company helping people live longer, healthier lives.
We partner with life sciences organizations—including p...Show moreLast updated: 21 hours ago
- Promoted
Senior Data Engineer (AWS / Databricks)
Accolademumbai city, maharashtra, in The multifamily real estate industry is undergoing a massive transformation, and Accolade is at the forefront.We are building the industry's first AI-native Operations Centralization Platform, desi...Show moreLast updated: 30+ days ago
- Promoted
- New!
Data Engineer Ii
ClearDemandDombivli, Republic Of India, IN Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership.
You'll play a crucial role in driving the evolution and efficiency of our...Show moreLast updated: 14 hours ago
- Promoted
- New!
Software Engineer Ii (Data) T500-20473
Best Buy IndiaDombivli, Republic Of India, IN Contribute to the delivery of complex solutions, breaking down big problems into smaller pieces.Actively participate in team planning activities.
Help ensure the quality and integrity of the SDLC fo...Show moreLast updated: 14 hours ago
- Promoted
AWS Data Engineer
ConfidentialMumbai City, Pune Data Engineer with strong ETL experience and hands-on expertise in AWS Glue, Airflow, Python, and SQL-based systems.Design, develop, and maintain robust ETL workflows using AWS Glue and Airflow.Wri...Show moreLast updated: 4 days ago