Lead Data Pipeline Engineer

Job Description : Tech Lead design, build, and maintain the infrastructure for an organization's data, ensuring it is reliable, accessible, and optimized for use by data scientists and analysts. Key responsibilities include creating and managing data pipelines, optimizing data storage and processing, ensuring data quality and security, and collaborating with other teams to meet data needs.

Responsibilities

Design and build data infrastructure :

Create and maintain scalable data pipelines, databases, and data warehouses to handle large volumes of data from various sources.

Ensure data quality and reliability :

Implement data cleansing, validation, and quality checks to ensure the data is accurate and trustworthy.

Optimize data systems :

Monitor and tune performance of databases and data processing tasks to ensure efficiency and speed.

Collaborate with stakeholders :

Work with data scientists, analysts, and other teams to understand their data requirements and make data accessible for their use.

Key skills :

8+ years of hands on experience in Data Engineering .

Experience on GCP (google cloud Platform)- Mandatory.

Hands on experience on Data bricks and worked on Data Governance (Unity Catalog)

Experience on Pyspark and Python.

Hands on experience on Unit testing.

Working experience on Terraform.