Data Engineer - ETL / PySpark

Role : Data Engineer

Location : Chennai (Hybrid).

Job Description :

Key Responsibilities :

• Lead the design, development, and deployment of scalable data pipelines and ETL processes using Scala and Apache Spark (Databricks).
• Develop and maintain data processing workflows using PySpark and Python.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust data solutions.
• Architect and implement data ingestion, transformation, and integration solutions for structured and unstructured data.
• Optimize Spark jobs for performance and cost efficiency on cloud platforms.
• Lead, mentor, and guide a team of data engineers, ensuring best practices in coding, testing, and deployment.
• Establish and enforce data engineering standards, code reviews, and documentation.
• Troubleshoot and resolve complex data pipeline issues.
• Stay current with emerging technologies and recommend improvements to existing data infrastructure.
• Work closely with DevOps and cloud teams to ensure smooth deployment and monitoring of data pipelines.

Required Qualifications :

(ref : hirist.tech)