Talent.com
Cloud Data Engineer - Databricks on AWS/Scala
Cloud Data Engineer - Databricks on AWS/ScalaConfidential • Chennai, Hyderabad / Secunderabad, Telangana, Pune
Cloud Data Engineer - Databricks on AWS / Scala

Cloud Data Engineer - Databricks on AWS / Scala

Confidential • Chennai, Hyderabad / Secunderabad, Telangana, Pune
30+ days ago
Job description

Key Responsibilities :

Design and Development  :

  • Design, develop, and maintain scalable ETL pipelines using cloud-native tools (AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, etc.).
  • Architect and implement data lakes and data warehouses on cloud platforms (AWS, Azure, GCP).
  • Develop and optimize data ingestion, transformation, and loading processes using Databricks, Snowflake, Redshift, BigQuery and Azure Synapse.
  • Implement ETL processes using tools like Informatica, SAP Data Intelligence, and others.
  • Develop and optimize data processing jobs using Spark Scala.

Data Integration and Management  :

  • Integrate various data sources, including relational databases, APIs, unstructured data, and ERP systems into the data lake.
  • Ensure data quality and integrity through rigorous testing and validation.
  • Perform data extraction from SAP or ERP systems when necessary.
  • Performance Optimization :

  • Monitor and optimize the performance of data pipelines and ETL processes.
  • Implement best practices for data management, including data governance, security, and compliance.
  • Collaboration and Communication  :

  • Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
  • Collaborate with cross-functional teams to design and implement data solutions that meet business needs.
  • Documentation and Maintenance :

  • Document technical solutions, processes, and workflows.
  • Maintain and troubleshoot existing ETL pipelines and data integrations.
  • Education :

  • Bachelor s degree in Computer Science, Information Technology, or a related field. Advanced degrees are a plus.
  • Experience :

  • 7+ years of experience as a Data Engineer or in a similar role.
  • Proven experience with cloud platforms : AWS, Azure, and GCP.
  • Hands-on experience with cloud-native ETL tools such as AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, etc.
  • Experience with other ETL tools like Informatica, SAP Data Intelligence, etc.
  • Experience in building and managing data lakes and data warehouses.
  • Proficiency with data platforms like Redshift, Snowflake, BigQuery, Databricks, and Azure Synapse.
  • Experience with data extraction from SAP or ERP systems is a plus.
  • Strong experience with Spark and Scala for data processing.
  • Skills :

  • Strong programming skills in Python, Java, or Scala.
  • Proficient in SQL and query optimization techniques.
  • Familiarity with data modeling, ETL / ELT processes, and data warehousing concepts.
  • Knowledge of data governance, security, and compliance best practices.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.
  • Preferred Qualifications :

  • Experience with other data tools and technologies such as Apache Spark, or Hadoop.
  • Certifications in cloud platforms (AWS Certified Data Analytics - Specialty, Google Professional Data Engineer, Microsoft Certified : Azure Data Engineer Associate).
  • Experience with CI / CD pipelines and DevOps practices for data engineering
  • Selected applicant will be subject to a background investigation, which will be conducted and the results of which will be used in compliance with applicable law.
  • Skills Required

    Apache Spark, Hadoop, Devops, Sql, Programming Skills

    Create a job alert for this search

    Cloud Data Engineer • Chennai, Hyderabad / Secunderabad, Telangana, Pune

    Related jobs
    Data Engineer

    Data Engineer

    IntraEdge • Pune, IN
    We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team. You will be responsible for building scalable and reli...Show more
    Last updated: 30+ days ago • Promoted
    AWS Cloud Engineer

    AWS Cloud Engineer

    Tata Consultancy Services • Pune, IN
    Greetings from TATA Consultancy Services!!.Thank you for expressing your interest in exploring a career possibility with the TCS Family. Hiring For : -AWS Cloud Engineer.We are seeking a highly skille...Show more
    Last updated: 15 days ago • Promoted
    AWS Database Eng

    AWS Database Eng

    Tata Consultancy Services • Pune, Maharashtra, India
    Desired Competencies (Technical / Behavioral Competency).Should have expertise in creating data warehouses in AWS utilizing the following tools : EC2, S3, EMR, Athena, Sagemaker, Aurora and Snowflake....Show more
    Last updated: 30+ days ago • Promoted
    AWS Data Engineer

    AWS Data Engineer

    Capgemini Invent • Pune, Maharashtra, India
    At Capgemini Invent, we believe difference drives change.As inventive transformation consultants, we blend our strategic, creative and scientific capabilities, collaborating closely with clients to...Show more
    Last updated: 8 days ago • Promoted
    AWS Cloud Engineer

    AWS Cloud Engineer

    Volkswagen Group Digital Solutions [India] • Pune, Maharashtra, India
    Great work will eventually happen!.ShapingFutureTechnologists is our POV.If technology is your passion and innovation is your fuel, then this is your workplace. More than a Great Place to Work, we a...Show more
    Last updated: 5 days ago • Promoted
    AWS Data Platform Engineer

    AWS Data Platform Engineer

    Persistent Systems • Pune, Maharashtra, India
    We’re looking for an AWS Data Platform Engineer to help automate and scale our cloud-based analytics environment.You’ll work with our BI and Data Engineering teams to build secure, automated, and r...Show more
    Last updated: 16 days ago • Promoted
    Databricks Data Engineer

    Databricks Data Engineer

    Ascendion • Pune, India
    Job Title : Senior Data Engineer.Location : Gurgaon / Pune / Bangalore.Skills : PySpark, SQL, Databricks, AWS.We are looking for 3–4 experienced Databricks Developers to support a fast-paced, high-impa...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer - Google Cloud Platform

    Data Engineer - Google Cloud Platform

    SkillKart • Pune
    Job Summary : We are looking for an experienced Data Engineer with strong hands-on expertise in Google Cloud Platform (GCP) and BigQuery to design, ...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer - AWS / ETL / Python

    Data Engineer - AWS / ETL / Python

    NonStop • Pune
    Description : About the Role : We are hiring an experienced Data Engineer to design and maintain scalable data pipelines and architectures.Youl...Show more
    Last updated: 30+ days ago • Promoted
    AWS Cloud Engineer

    AWS Cloud Engineer

    Proglite • Pune, IN
    Infrastructure & System Administration : .Deploy, manage, and optimize EC2 instances across dev, test, and production environments. Perform system administration and troubleshooting for Linux and Wind...Show more
    Last updated: 30+ days ago • Promoted
    AWS Data Engineer (Remote)

    AWS Data Engineer (Remote)

    Mindcraft Labs • Pune, IN
    Remote
    This role focuses on building and maintaining data pipelines and analytics infrastructure on AWS.You will work daily with S3, Glue, Redshift, Athena, Lake Formation, Airflow, SNS / SQS, and Postgres ...Show more
    Last updated: 6 hours ago • Promoted • New!
    Cloud Engineer

    Cloud Engineer

    GoML • Pune, IN
    At goML, we design and build cutting-edge Generative AI, AI / ML, and Data Engineering solutions that help businesses unlock the full potential of their data, drive intelligent automation, and create...Show more
    Last updated: 1 day ago • Promoted
    Data Engineer

    Data Engineer

    Tata Consultancy Services • Pune, Maharashtra, India
    TCS is hiring for AWS Data engineer, please find the below JD.Experience range – 6 to 8 years.Location - Hyderabad / Pune / Indore. Skills Required - AWS, Pyspark, EMR, Glue.AMI Rehydration, Cloud for...Show more
    Last updated: 30+ days ago • Promoted
    AWS Data Engineer

    AWS Data Engineer

    Tata Consultancy Services • Pune, Maharashtra, India
    Required Technical Skill Set AWS Redshift, Glue, PySpark.Desired Experience Range 4-10 Years.Location of Requirement Chennai / Pune. Desired Competencies (Technical / Behavioral Competency).Must-Have • ...Show more
    Last updated: 30+ days ago • Promoted
    Senior Data Engineer- AWS

    Senior Data Engineer- AWS

    InfoCepts • Pune, India
    Key Result Areas and Activities : .Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements. Should be able to design comple...Show more
    Last updated: 30+ days ago • Promoted
    MAN - Cloud Engineer - AWS

    MAN - Cloud Engineer - AWS

    MAN TRUCK & BUS INDIA PVT. LTD • Pune
    Description : Qualification : - Degree in Computer Science (or similar), alternatively well-founded profe...Show more
    Last updated: 30+ days ago • Promoted
    AWS Data Engineer - Python / Scala / Java

    AWS Data Engineer - Python / Scala / Java

    HyrEzy Talent Solutions • Pune
    Job Title : AWS Data Engineer Location : Pune , Bangalore, Greater Noida, Trichy Responsibilities : Show more
    Last updated: 30+ days ago • Promoted
    Senior Data Engineer - Snowflake / AWS

    Senior Data Engineer - Snowflake / AWS

    Resourcetree • Pune
    About the Role : A "Senior Data Engineer" is mid-level professional leading the design, build and evolution of the inhouse data platforms.You lead the const...Show more
    Last updated: 30+ days ago • Promoted