Data Engineer (1744)

Role: AWS Data Engineer

Preferred Skills

· ETL Development

· AWS ETL Services

· Data Pipeline Development

· SQL & Python Programming

· Data Warehousing

Experience: 4-8 years

Detailed Responsibilities & Skills

Must Have Technical Skills

Strong hands-on experience with:

· AWS ETL Services (Glue Lambda Step Functions)

· AWS AppFlow (mainly ODATA connector)

· AWS Database Migration Service (DMS)

· OData Services and API Integration

· Amazon Redshift

· Data Lake & Amazon S3

· Python PySpark SQL Data Modeling

· Data Modeling & ETL Development

· Data Quality & Performance Optimization

· Enterprise Data Integration (SAP Oracle Azure Oracle Cloud)

Key Responsibilities

· Design develop and maintain scalable ETL pipelines using AWS Glue Lambda Step Functions and Redshift.

· Build and orchestrate AWS workflows using Master Parent and Child Step Functions to execute Glue jobs and Redshift stored procedures.

· Develop optimized Glue jobs using Python and PySpark ensuring efficient DPU utilization and performance tuning.

· Design and implement data ingestion pipelines from multiple enterprise data sources including SAP Oracle Database Oracle Cloud Azure and other heterogeneous systems.

· Capture change data / delta data to ingestion layer.

· Store transform and manage data in Amazon S3 (Object Store Iceberg Tables) and Amazon Redshift.

· Perform data cleansing transformation and validation to ensure high data quality and reliability.

· Identify opportunities to improve ETL performance data quality and overall pipeline efficiency.

· Design and implement data models including fact and dimension tables to support reporting and analytics.

· Transform unstructured and semi-structured data into structured datasets suitable for downstream consumption.

· Develop BI-ready datasets and understand KPI calculations across multiple business domains.

· Collaborate with business and analytics teams to understand reporting requirements and translate them into technical solutions.

· Troubleshoot production issues and support enhancements for existing data pipelines.

Additional Responsibilities

· Design and develop data ingestion pipelines using AWS AppFlow AWS DMS OData services APIs and batch ingestion frameworks.

· Integrate enterprise applications such as SAP (OData/CDS Services) Oracle Database Oracle Cloud Azure and other heterogeneous sources with AWS data platforms.

· Configure and optimize AWS DMS for full load and Change Data Capture (CDC) migration scenarios.

· Build and maintain AWS AppFlow integrations for secure and automated data movement between SaaS applications and AWS services.

· Troubleshoot and optimize data ingestion performance across AppFlow DMS Glue and Step Functions.

Required Experience

· 4–8 years of experience in Data Engineering with strong AWS expertise.

· Proven experience building enterprise-scale ETL pipelines on AWS.

· Strong proficiency in Python PySpark and SQL.

· Experience working with Amazon Redshift as a developer including stored procedures and performance optimization.

· Hands-on experience with AWS Step Functions for workflow orchestration.

· Experience in AWS Glue job development optimization and Glue Crawlers.

· Experience handling large datasets and optimizing ETL workloads.

· Good understanding of data modeling concepts including star schema fact tables and dimension tables.

· Familiarity with cloud-based data lakes and modern data architectures.

· Experience integrating data from enterprise applications such as SAP Oracle Azure and Oracle Cloud.

· Understanding of KPI calculations and preparation of analytics-ready datasets.

Good to Have

· Knowledge of Apache Iceberg on Amazon S3.

· Experience with CI/CD for AWS data pipelines.

· Exposure to Infrastructure as Code (CloudFormation/Terraform).

· Understanding of data governance data cataloging and security best practices.

· Experience with monitoring and troubleshooting AWS data services.

Soft Skills

· Strong analytical and problem-solving skills.

· Good communication and stakeholder management.

· Ability to work independently as well as within cross-functional teams.

· Strong documentation and knowledge-sharing practices.

· Ability to work in an Agile development environment.

Employment Type : Full Time
Experience: years
Vacancy: 1