AWS Data Engineer

Role

AWS Data Engineer

Experience

8+ years

Location

Remote

Time Zone

UK

Duration

2 months (Extendable)

Job Description

• Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR.
• Writing reusable, testable, and efficient code
• Integration of data storage solutions in spark – especially with AWS S3 object storage. Performance tuning of pySpark scripts.
• Need to ensure overall build delivery quality is good and on-time delivery is done at all times.
• Should be able to handle meetings with customers with ease.
• Need to have excellent communication skills to interact with the customer.
• Be a team player and willing to work in an onsite-offshore model, mentor other folks in the team (onsite as well as offshore)
• 5+ years of experience in programming with python. Strong proficiency in python
• Familiarity with functional programming concepts
• 3+ years of hands-on experience in developing ETL data pipelines using pySpark on AWS EMR
• Experience in building pipelines and data lake for large enterprises on AWS
• Good understanding of Spark’s Dataframe and API
• Experience in configuring EMR clusters on AWS
• Experience in dealing with AWS S3 object storage from Spark.
• Experience in troubleshooting spark jobs. Knowledge of monitoring spark jobs using Spark UI
• Performance tuning of Spark jobs.
• Understanding fundamental design principles behind business processes

Process Knowledge and Expertise :

Databricks-Specific Skills :

Important Pointers :