Data Pipeline Engineer

ValueLabsHyderabad, Republic Of India, IN

30+ days ago

Job description

Role : Data Engineer

Experience : 6+ Years

Location : Hyderabad

Working Model : Hybrid

Notice : 0-15 Days

Job Description :

Design, develop, and maintain
ETL / ELT data pipelines
to ingest data from various sources, including databases and flat files, into our data warehouse.
Utilize AWS services such as
AWS Lambda
AWS DMS
, and
AWS Glue
for data ingestion and transformation processes.
Develop and optimize
Snowflake stored procedures
to implement complex business logic and data transformations.
Manage and monitor scheduled data workflows using
Apache Airflow
Work closely with stakeholders to understand project requirements and translate them into technical solutions, ensuring seamless integration with existing data pipelines.
Implement and maintain robust
data quality checks
to ensure accuracy and reliability of all delivered datasets and reports.
Adhere to the full project delivery lifecycle, including requirements gathering, design, implementation, documentation, and obtaining user acceptance testing (UAT) approvals.
Deliver data sets to other AWS accounts and support the creation of Power BI reports.

Roles and Responsibilities :

Proven experience as a Data Engineer or in a similar role.

Strong proficiency in

SQL

scripting.

Proficiency in

Python

for data manipulation and automation.

Experience with

Apache Airflow

for workflow orchestration.

Hands-on experience with

AWS Cloud services

, including Lambda, DMS, and Glue.

Solid understanding of data warehousing concepts, with practical experience using

Snowflake

Excellent problem-solving skills and the ability to design and implement solutions based on business requirements.

Experience with the full software development lifecycle (SDLC) and project documentation.

Familiarity with data quality best practices.

Strong communication and collaboration skills.

Data Pipeline Engineer • Hyderabad, Republic Of India, IN