This job offer is not available in your country.
Associate Data Engineer
ConfidentialMumbai30+ days ago
Job description0-6months of hands-on experience Designing and developing Data Pipelines for Data Ingestion or Transformation using Python (PySpark) / Spark SQL in AWS cloud Experience in design and development of data pipelines and processing of data at scale. Advanced experience in writing and optimizing efficient SQL queries with Python and Hive handling Large Data Sets in Big-Data Environments Experience in debugging, tunning and optimizing PySpark data pipelines Should have implemented concepts and have good knowledge of Pyspark data frames, joins, caching, memory management, partitioning, parallelism etc. Understanding of Spark UI, Event Timelines, DAG, Spark config parameters, in order to tune the long running data pipelines. Experience working in Agile implementations Experience with building data pipelinesin streaming and batch mode. Experience with Git and CI / CD pipelines to deploy cloud applications Good knowledge of designing Hive tables with partitioning for performance. Experience in data modelling Hands on creating workflows on any Scheduling Tool like Autosys, CA Workload Automation Proficiency in using SDKsfor interacting with native AWS services Strong understanding of concepts of ETL, ELT and data modeling.
Responsibilities :
- Proven success in communicating with users, other technical teams, and senior management to collect requirements, describe data modeling decisions and develop data engineering strategy.
- Ability to work with business owners to define key business requirements and convert to user stories with required technical specifications.
- Communicate results and business impacts of insight initiatives to key stakeholders to collaboratively solve business problems.
- Working closely with the overall Enterprise Data & Analytics Architect and Engineering practice leads to ensure adherence with the best practices and design principles.
- Assures quality, security and compliance requirements are met for supported area.
- Design and create fault-tolerance data pipelines running on cluster
- Excellent communication skills with the ability to influence client business and IT teams
- Should have design data engineering solutions end to end. Ability to come up with scalable and modular solutions
Required Qualification :
Desired Qualification :
Skills Required
Advanced Sql, Numpy, Python, Pyspark, Shell Scripting, Data Modelling
Create a job alert for this search
Data Engineer • Mumbai
Related jobs
- Promoted
Data Engineer
Manuh TechnologiesMumbai, IN Strong proficiency in Python programming (Pandas, NumPy, PySpark, or similar).Hands-on experience with Dask for large-scale distributed data processing.
Proven expertise as a Data Modeler (conceptua...Show moreLast updated: 30+ days ago
- Promoted
- New!
Senior Generative AI Engineer (Databricks Data Lake)
AmpstekMumbai, IN Title : Senior Generative AI Engineer (Databricks Data Lake).We are seeking an experienced Senior Generative AI Engineer with a strong background in Databricks and data lake architectures.This indiv...Show moreLast updated: 14 hours ago
- Promoted
Senior Data Engineer
EvolentKalyan-Dombivli, IN Lead the design, development, and deployment of complex data solutions, ensuring scalability and maintainability.Architect and implement high-performance data pipelines and data warehousing solutio...Show moreLast updated: 24 days ago
- Promoted
Senior Data Engineer
InfogainThane, IN Big Data Engineer (Lead) : As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers.
You will work closely with clients and team members to understand their req...Show moreLast updated: 30+ days ago
- Promoted
Data Engineer
Envuthane, maharashtra, in At Envu, we partner with our customers to design world-class, forward-thinking innovations that protect and enhance the health of environments around the world.
We offer dedicated services in : Profe...Show moreLast updated: 30+ days ago
- Promoted
- New!
Data Engineer
Kline + CompanyThane, IN Here at Kline our data capabilities have grown exponentially over the last four years.Having gone through a rapid digitization process and becoming a cloud-native corporation, we are looking for to...Show moreLast updated: 14 hours ago
- Promoted
Data Engineer
Kumaran SystemsThane, IN Strong hands-on experience in Databricks.Proven expertise in building and managing data ingestion pipelines.Exposure to Databricks data ingestion jobs along with incident management frameworks.Expe...Show moreLast updated: 8 days ago
- Promoted
Rocket UniData Engineer
Mitra AIdombivli, maharashtra, in The ideal candidate will have a deep understanding of UniData and UniVerse environments, with a proven track record in developing, maintaining, and optimizing complex legacy systems.Design, develop...Show moreLast updated: 18 days ago
- Promoted
Data Engineer
IntraEdgeThane, IN Snowflake, AWS (Lambda, Glue), DBT, and SQL.The ideal candidate will be responsible for enabling seamless data integration, transformation, and analytics to support business intelligence and advanc...Show moreLast updated: 30+ days ago
- Promoted
Lead Data Engineer
Eucloid Data SolutionsThane, IN Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 8 days ago
- Promoted
Data Engineer
XebiaKalyan-Dombivli, IN We’re Hiring : Data Engineer | Xebia.Any Xebia location (Hybrid, 3 days office per week).Immediate to 2 weeks – only apply if you can join early.
Databricks, Python, SQL, and Postgres.The ideal candi...Show moreLast updated: 5 days ago
- Promoted
Data Engineer
RevXMumbai, IN RevX helps app businesses acquire and reengage users via programmatic to retain, monetize, and accelerate revenue.We're all about taking your app businesses to a new growth level.We rely on data sc...Show moreLast updated: 30+ days ago
- Promoted
- New!
Senior Generative AI Engineer (Databricks,Data Lake)
AmpstekMumbai, IN Title : Senior Generative AI Engineer (Databricks,Data Lake).Design and implement GenAI models (LLMs, multimodal, embeddings, and fine-tuning) for enterprise use cases.
Architect and optimize data pi...Show moreLast updated: 14 hours ago
- Promoted
Data Engineer
AceolutionThane, IN We are looking for a freelancer to engage with us for 20-40 hours per week.Kindly find the JD below for your reference.Design, develop, and maintain scalable data pipelines and workflows.Work exten...Show moreLast updated: 8 days ago
- Promoted
Data / ML Engineer
GleantapThane, IN Gleantap is a customer engagement platform powering fitness, wellness, and service businesses.Python (scikit-learn, LightGBM, XGBoost).
Set up retraining and monitoring pipelines (Airflow, MLflow, o...Show moreLast updated: 22 days ago
- Promoted
Associate Architect - Data Engineering
Response Informaticsmumbai, maharashtra, in We are seeking an experienced Data Architect to lead the transformation of enterprise data.Alteryx workflows into Azure Databricks.
Microsoft Azure ecosystem, including Azure.Data Factory, Databrick...Show moreLast updated: 26 days ago
- Promoted
Data Engineer
Innodata Inc.Mumbai, IN CI / CD practices, Databricks (Spark), Python, Github and SQL.The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and...Show moreLast updated: 25 days ago
- Promoted
Data Engineer
EveriseThane, IN Join us on our mission to elevate customer experiences for people around the world.As a member of the Everise family, you will be part of a global experience company that believes in being people-f...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
HISH IT SERVICESThane, IN Location : Remote(Banglore,Chennai,Pune).Pay : 14LPA - 18 LPA(Based on Experience).Timings : A couple of hours overlap with EST, as the client is Canada-based (till 12AM IST).Start Date : 20th Octob...Show moreLast updated: 6 days ago
- Promoted
Data Engineer
ACL Digitalmumbai city, maharashtra, in Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.Design, develop, and optimize AWS pipeline as applicable.
Implement and manage GitHub asset bun...Show moreLast updated: 30+ days ago