Data Engineer

The Data Engineer – MWAA & Redshift will design, build, and maintain data pipelines orchestrated through AWS Managed Workflows for Apache Airflow (MWAA) to ingest, transform, and optimize data in Amazon Redshift. This role is central to implementing automated, scalable, and reliable data workflows that support analytics, reporting, and downstream data products within a modern cloud data platform.

About the Role

Key Responsibilities

• Pipeline Design & Development
• Design, build, and maintain data ingestion and transformation pipelines using MWAA (Apache Airflow) and AWS Redshift.
• Develop Airflow DAGs to orchestrate data extraction from diverse sources (e.g., Oracle EBS, APIs, flat files, or S3).
• Implement transformation logic using SQL, Python, or Redshift stored procedures to curate data for analytical consumption.
• Create and manage staging, curated, and semantic data layers in Redshift aligned with the Medallion architecture principles.
• Integrate data ingestion pipelines with AWS services (e.g., S3, Glue, Lambda, Secrets Manager).
• Performance, Optimization & Maintenance
• Tune Redshift queries and schema for high performance and cost efficiency.
• Implement Airflow monitoring, logging, and alerting using CloudWatch or similar tools.
• Maintain and version-control Airflow DAGs and Redshift scripts in Git.
• Continuously improve data pipeline reliability, error handling, and observability.
• Establish data validation checks and ensure pipeline SLAs are met.
• Integration & Security
• Configure and manage secure data access between MWAA, S3, and Redshift using IAM roles and Secrets Manager.
• Collaborate with data architects and DevOps teams to implement CI / CD automation for pipeline deployment.
• Work closely with business analysts and data consumers to ensure datasets meet functional and performance requirements.
• Testing & Validation
• Perform unit, integration, and regression testing on data pipelines and transformations.
• Validate data quality, schema consistency, and referential integrity between source and Redshift layers.
• Participate in User Acceptance Testing (UAT) and resolve issues collaboratively with stakeholders.
• Document data flows, dependencies, and operational runbooks for pipeline management.

Qualifications

Required Skills

Preferred Skills

Pay range and compensation package

[Pay range or salary or compensation]

Equal Opportunity Statement

[Include a statement on commitment to diversity and inclusivity.]