Talent.com
AWS Data Engineer

AWS Data Engineer

TerraGiGMadurai, IN
13 hours ago
Job description

Role

AWS Data Engineer

Experience

8+ years

Location

Remote

Time Zone

UK

Duration

2 months (Extendable)

Job Description

  • Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR.
  • Writing reusable, testable, and efficient code
  • Integration of data storage solutions in spark – especially with AWS S3 object storage. Performance tuning of pySpark scripts.
  • Need to ensure overall build delivery quality is good and on-time delivery is done at all times.
  • Should be able to handle meetings with customers with ease.
  • Need to have excellent communication skills to interact with the customer.
  • Be a team player and willing to work in an onsite-offshore model, mentor other folks in the team (onsite as well as offshore)
  • 5+ years of experience in programming with python. Strong proficiency in python
  • Familiarity with functional programming concepts
  • 3+ years of hands-on experience in developing ETL data pipelines using pySpark on AWS EMR
  • Experience in building pipelines and data lake for large enterprises on AWS
  • Good understanding of Spark’s Dataframe and API
  • Experience in configuring EMR clusters on AWS
  • Experience in dealing with AWS S3 object storage from Spark.
  • Experience in troubleshooting spark jobs. Knowledge of monitoring spark jobs using Spark UI
  • Performance tuning of Spark jobs.
  • Understanding fundamental design principles behind business processes

Process Knowledge and Expertise :

  • Demonstrated experience in change management processes, including understanding of governance frameworks and preparation of supporting artefacts required for approvals.
  • Strong clarity on the path to production, with hands-on involvement in deployments, testing cycles, and obtaining business sign-offs.
  • Proven track record in technical solution design, with the ability to provide architectural guidance and support implementation strategies.

    • Databricks-Specific Skills :

  • Experience in at least developing and delivering end-to-end Proof of Concept (POC) solutions covering the below :
  • Basic proficiency in Databricks, including creating jobs and configuring clusters.
  • Exposure to connecting external data sources (e.g., Amazon S3) to Databricks.
  • Understanding of Unity Catalog and its role in data governance.
  • Familiarity with notebook orchestration and implementing modular code structures to enhance scalability and maintainability.

    • Important Pointers :

  • Candidates must have actual hands-on work experience , not just home projects or academic exercises.
  • Profiles should clearly state how much experience they have in each skill area , as this helps streamline the interview process.
  • Candidates must know their CV / profile inside out , including all projects and responsibilities listed. Any ambiguity or lack of clarity on the candidate’s part can lead to immediate rejection, as we value accuracy and ownership.
  • They should be able to confidently explain their past experience, challenges handled, and technical contributions.
    • Create a job alert for this search

    Aws Data Engineer • Madurai, IN