AWS Data Engineer

Role and responsibilities

• Understands the process flow and the impact on the project module outcome.
• Works on coding assignments for specific technologies basis the project requirements and documentation available
• Debugs basic software components and identifies code defects.
• Focusses on building depth in project specific technologies.
• Expected to develop domain knowledge along with technical skills.
• Effectively communicate with team members, project managers and clients, as required.
• A proven high-performer and team-player, with the ability to take the lead on projects.
• Design and create S3 buckets and folder structures (raw, cleansed_data, output, script, temp-dir, spark-ui)
• Develop AWS Lambda functions (Python / Boto3) to download Bhav Copy via REST API and ingest into S3
• Author and maintain AWS Glue Spark jobs to :

– partition data by scrip, year and month

– convert CSV to Parquet with Snappy compression

Technical skills requirements

The candidate must demonstrate proficiency in,

Nice-to-have skills