Cloud Consultant

We need an experienced AWS Data Engineer / Performance Optimization Consultant to help us analyze, tune, and optimize our existing AWS Glue ETL, Step Functions workflows and Amazon Redshift data warehouse. The goal is to reduce job runtimes, improve query performance, and lower AWS operational costs.

Key Responsibilities :

Glue Job Optimization :

Review and analyze existing AWS Glue jobs, including global / local variables, job parameters, and configurations.

Evaluate Glue job scripts, reference framework Python modules, and orchestration via Step Functions.

Consolidate multi-layered Glue jobs where feasible to improve efficiency and maintainability.

Identify and resolve performance bottlenecks across the ETL pipeline.

Optimize job configurations such as worker types, parallelism, partitioning strategies, Spark tuning, and incremental load logic.

Review Spark UI and job metrics to identify opportunities for runtime and cost improvements.

Redshift Performance Tuning :

Analyze query plans and stored procedures for optimization opportunities.

Tune sort keys, distribution styles, and table design.

Optimize WLM (Workload Management), concurrency scaling, vacuum / analyze strategies.

Data Pipeline & Cost Optimization :

Improve data load patterns from source EBS to Glue to Redshift.

Recommend and implement best practices for partitioning, caching, and resource management.

Evaluate and implement cost-saving strategies