Talent.com
This job offer is not available in your country.
Lead Data Engineer - Python / Big Data

Lead Data Engineer - Python / Big Data

HyreSnapMumbai
10 days ago
Job description

Responsibilities :

  • Design, execution, and management of a large and complex distributed data system.
  • Monitoring of performance and optimizing existing projects.
  • Researching and integrating any Big Data tools and frameworks required to provide the requested capabilities.
  • Understanding Business / Data requirements and implementing scalable solutions.
  • Creating reusable components and data tools that help all the teams in the company to integrate with our data platform.

Requirements :

  • 6-9 years of experience in big data technologies (Apache Hadoop) and relational databases (MS SQL Proficiency in at least one of the following programming languages : Java, Python, or Scala.
  • Expertise in SQL Proficiency in Apache Spark.
  • Hands-on knowledge of working with Data Frames, Data Sets, RDDs, Spark SQL / PySpark / Scala APIs, with a deep understanding of Performance Optimizations.
  • Good Understanding of Distributed Storage (HDFS / S3).
  • Strong analytical / quantitative skills and comfortable working with very large sets of data.
  • Experience with the integration of data across multiple data sources.
  • Good understanding of distributed computing principles.
  • It would be good if you have the following skills :

  • Experience with Message Queues (e. g., Apache Kafka).
  • Experience with MPP systems (e. g., Redshift / Snowflake).
  • Experience with NoSQL storage (e. g., MongoDB).
  • (ref : hirist.tech)

    Create a job alert for this search

    Lead Data Engineer • Mumbai

    Related jobs
    • Promoted
    Data Engineer

    Data Engineer

    Canopus Infosystems - A CMMI Level 3 CompanyThane, IN
    Python expertise and hands-on experience in handling large datasets, data cleaning, analysis, and visualization.The ideal candidate should be capable of building data pipelines, performing web scra...Show moreLast updated: 18 days ago
    • Promoted
    Scala Big Data Lead Engineer - 7 YoE - Immediate Joiner - Any UST Location

    Scala Big Data Lead Engineer - 7 YoE - Immediate Joiner - Any UST Location

    USTnavi mumbai, maharashtra, in
    CTC, notice period, and current location details to.Apache Hadoop, Airflow, Kubernetes, and Containers.Troubleshoot Hadoop log files and work with. Required Skills & Qualifications : .Scala, Spark, Py...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Media.netmumbai, maharashtra, in
    Our proprietary contextual technology is at the forefront of enhancing Programmatic buying, the latest industry standard in ad buying for digital platforms. HQ is based in New York, and the Global H...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    VAANTECHdombivli, maharashtra, in
    Immediate Joiner (or within 15 days).Includes Night Shifts (US Shift).Preferred Candidates : From Chennai.Data Pipeline Optimization & Tuning. Hadoop Infra / Cloud Platforms.Lead and mentor a team of...Show moreLast updated: 28 days ago
    • Promoted
    ETL Data Engineer - Python / PySpark

    ETL Data Engineer - Python / PySpark

    Information TechnologyMumbai
    Key Responsibilities : - Design, develop, and maintain robust data pipelines and ETL / ELT workflows using PySpark, Python, and SQL. Build and manage data ingestion and ...Show moreLast updated: 1 day ago
    • Promoted
    Data Engineer

    Data Engineer

    INFEC Servicesthane, maharashtra, in
    Design, develop, and optimize data pipelines and ETL processes on GCP or Azure.Work with structured and unstructured data, integrating sources such as databases, APIs, and streaming platforms.Imple...Show moreLast updated: 4 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Bahwan CyberTeknavi mumbai, maharashtra, in
    Job Title : Data Engineer – Google Cloud Platform (GCP).We are seeking a skilled and motivated Data Engineer with hands-on experience in building scalable data pipelines and cloud-native data soluti...Show moreLast updated: 17 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DeltacubesThane, IN
    Build and maintain scalable ETL / ELT pipelines.Work with Snowflake and BigQuery for data storage.Implement orchestration with Airflow or Prefect. Integrate data workflows with Python.Optimize data pi...Show moreLast updated: 15 days ago
    • Promoted
    Lead Azure Data Engineer

    Lead Azure Data Engineer

    RandomTreesThane, IN
    We’re a leading software company specializing in Artificial Intelligence, Machine Learning, Data Analytics, Innovative data solutions, Cloud-based technologies. If you're passionate about building r...Show moreLast updated: 26 days ago
    • Promoted
    Azure Data Engineer

    Azure Data Engineer

    Tata Consultancy Servicesthane, maharashtra, in
    Job Title : Azure Data Engineer.Required Skillset : Azure Databricks(Strong),scala(strong), Pyspark(Good to have).Azure Databricks using Scala, PySpark. Software Development Life Cycle experience.Work...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    AceolutionKalyan-Dombivli, IN
    We are looking for a freelancer to engage with us for 20-40 hours per week.Kindly find the JD below for your reference.Design, develop, and maintain scalable data pipelines and workflows.Work exten...Show moreLast updated: 8 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Innodata Inc.Kalyan-Dombivli, IN
    CI / CD practices, Databricks (Spark), Python, Github and SQL.The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and...Show moreLast updated: 25 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data SolutionsMumbai, IN
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 7 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Kanerika Incthane, maharashtra, in
    Databricks, Microsoft Fabric, or Snowflake.You will also play a key role in enabling effective data visualization and reporting for business stakeholders. Databricks, Fabric, or Snowflake.Power BI d...Show moreLast updated: 19 days ago
    • Promoted
    Data Engineer Team Lead

    Data Engineer Team Lead

    SGImumbai city, maharashtra, in
    To be discussed based on your skills and experience.Strong hands-on data engineering experience with a proven ability to design, build, and optimize scalable data pipelines in .Deep technical exper...Show moreLast updated: 6 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    SAIVA AIThane, IN
    We are building the future of healthcare analytics.Join us to design, build, and scale robust data pipelines that power nationwide analytics and support our machine learning systems.Our goal : pipel...Show moreLast updated: 7 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Manuh TechnologiesKalyan-Dombivli, IN
    Strong proficiency in Python programming (Pandas, NumPy, PySpark, or similar).Hands-on experience with Dask for large-scale distributed data processing. Proven expertise as a Data Modeler (conceptua...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    HISH IT SERVICESKalyan-Dombivli, IN
    Location : Remote(Banglore,Chennai,Pune).Pay : 14LPA - 18 LPA(Based on Experience).Timings : A couple of hours overlap with EST, as the client is Canada-based (till 12AM IST).Start Date : 20th Octob...Show moreLast updated: 6 days ago