Data Engineer

Job Description :

• Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
• Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
• Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
• Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
• Design and implement data CI / CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
• Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
• Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
• Stay up-to-date with emerging technologies and best practices in data engineering.
• Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Responsibilities :

Required Skills -

Python  ,  Pyspark  ,  Sql  ,Airflow,  Trino  , Hive,  Snowflake  , Agile  Scrum

Good to have-

Linux,  Openshift  ,  Kubernentes  , Superset

Role :   Data Engineer

Industry Type :   IT Services & Consulting

Department :   Engineering - Software & QA

Employment Type :   Full Time, Permanent

Role Category :   Software Development

Education

UG :   Any Graduate

PG :   Any Postgraduate

Skills Required

Sql Queries, Devops, Delta, Scala, Spark, Microsoft Azure, Data Validation, Troubleshooting, Monitoring