Talent.com
Senior Data Engineer - Python / Spark

Senior Data Engineer - Python / Spark

GetinzChennai
6 days ago
Job description

Description : Role Overview :

We are looking for a highly skilled and motivated Senior Data Engineer to join our clients team. You will play a key role in designing, implementing, and optimizing data architectures and pipelines to support scalable data solutions for our business.

Key Responsibilities :

  • Design, build, and maintain efficient and scalable data pipelines to support data integration and transformation across various sources.
  • Work with relational databases (e.g., MySQL, PostgreSQL, etc.) and NoSQL databases (e.g., MongoDB, Cassandra, etc.) to manage and optimize large datasets.
  • Utilize Apache Spark for distributed data processing and real-time analytics.
  • Implement and manage Kafka for data streaming and real-time data integration between

systems.

  • Collaborate with cross-functional teams to gather and translate business requirements into

    • technical solutions.

  • Monitor and optimize the performance of data pipelines and architectures, ensuring high

    • availability and reliability.

  • Ensure data quality, consistency, and integrity across all systems.
  • Stay up-to-date with the latest trends and best practices in data engineering and big data Skills :
  • 6-9 years of experience in data engineering, with a focus on building and managing data pipelines.
  • Strong proficiency in relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases

    • (e.g., MongoDB, Cassandra).

  • Experience in building data pipelines with data warehouses like Snowflake, Redshift
  • Experience in processing unstructured data stored from S3 using Athena, Glue etc.
  • Hands-on experience with Kafka for real-time data streaming and messaging.
  • Solid understanding of ETL processes, data integration, and data pipeline optimization.
  • Proficiency in programming languages like Python, Java, or Scala for data processing.
  • Experience with Apache Spark for big data processing and analytics is an advantage
  • Familiarity with cloud platforms like AWS, GCP, or Azure for data infrastructure is a plus.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration skills, with the ability to work effectively in a team

    • environment.

    (ref : hirist.tech)

    Create a job alert for this search

    Senior Data Engineer • Chennai