We are seeking a skilled and experienced AWS Data Engineer to join our team.
The ideal candidate will be a hands-on expert in designing, building, and managing robust and scalable data pipelines on the Amazon Web Services (AWS) platform.
This role is crucial for our data-driven initiatives, requiring a strong understanding of cloud-based data solutions and a passion for optimizing data workflows.
You will work on complex ETL (Extract, Transform, Load) processes, ensuring data integrity, security, and performance.
Key Responsibilities :
- Data Pipeline Development : Design, build, and maintain scalable and efficient data pipelines using a range of AWS services.
- ETL Processes : Implement and manage ETL processes to extract data from various sources, transform it for analysis, and load it into data lakes or data warehouses.
- Data Workflow Optimization : Continuously monitor and optimize data workflows and processing jobs to improve performance, reduce latency, and ensure cost-effectiveness.
- Collaboration : Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality, clean, and accessible data.
- Cloud Security and Governance : Implement and maintain data security best practices on AWS, including access control, encryption, and compliance with data governance policies.
- Automation : Automate data-related tasks and infrastructure management using scripting and Infrastructure as Code (IaC) principles.
- Troubleshooting : Identify, diagnose, and resolve complex data-related issues and system failures.
Required Skills & Experience :
Hands-on AWS Expertise : Strong experience with core AWS data services, including AWS Glue, Amazon S3, and AWS Lambda.Programming Proficiency : Expert-level skills in Python for scripting, automation, and data manipulation.Distributed Data Processing : Proven experience with PySpark for processing large-scale datasets.Data Engineering Fundamentals : Solid understanding of data warehousing, data modeling, and ETL / ELT concepts.SQL Knowledge : Proficient in SQL for data querying and manipulation.Familiarity with other AWS Services : Experience with other AWS services such as Amazon Redshift, Amazon Athena, and AWS Step Functions is a plus.Security and Performance : Knowledge of AWS security best practices (e.g., IAM, KMS) and experience with performance tuning and cost optimization.Problem-Solving : Excellent analytical and problem-solving skills with a meticulous attention to detail.(ref : hirist.tech)