Senior Data Engineer- AWS

Key Result Areas and Activities :

Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.

Should be able to design complex Big Data use cases using AWS Services and Databricks, under guidance of Architect

Should be able to assist the architect in reasoning about various architecture choices in terms of cost, performance and durability.

Should be able to suggest optimizations in existing designs.

Project documentation adheres to quality guidelines & schedules.

Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.

Articulate design decisions very clearly with stakeholders.

Perform Proof-of-Concepts and document all observations before proposing a new solution.

Conduct design review sessions with other teams and identify scope of improvement.

Suggests automation to improve existing processes.

Work and Technical Experience :

Must-Have :

In-depth knowledge of the following AWS services : S3, EC2, EMR, Athena, AWS Glue, Lambda

Experience with at least one MPP database : AWS Redshift, Snowflake, SingleStore

Proficiency in Big Data technologies : Apache Spark, Databricks

Must have strong programming skills in Python

Responsible for building data pipelines in AWS And Databricks

Experience with Big Data table formats, such as Delta Lake (open source)

Must have very strong SQL skills

Experience with orchestration tools like Apache Airflow

Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.

Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required

Strong understanding of data warehousing concepts

Good to have :

Cloud Databases – Snowflake, AWS Aurora

Big Data – Hadoop, Hive

Cloud Databases– AWS Aurora

Associate Level or Professional Level AWS Certification or Databricks Certification

Qualifications :

Overall, 7+ years of IT Exp

5+ years of relevant experience in AWS related project

Bachelor’s degree in computer science, engineering, or related field (master's degree is a plus)

Demonstrated continued learning through one or more technical certifications or related methods.

Qualities :

Hold strong technical knowledge and experience

Should have the capability to deep dive and research in various technical related fields

Self-motivated and focused on delivering outcomes for a fast-growing team and firm

Able to communicate persuasively through speaking, writing, and client presentations

Able to consult, write, and present persuasively

Able to work in a self-organized and cross-functional team

Able to iterate based on new information, peer reviews, and feedback

Prior experience of working in a large media company would be added advantage