Senior Data Engineer - ETL / PySpark

Essential Duties and Responsibilities :

• This section contains a list of five to eight primary responsibilities of this role that account for 5% or more of the work. The incumbent will perform other duties assigned.
• Development of new ETL / data transformation jobs, using PySpark or Python in AWS.
• Enhancement and support on existing ETL / data transformation jobs.
• Can explain technical solutions and resolutions with internal customers and communicate feedback to the ETL team.
• Perform technical code reviews for peers moving code into production.
• Perform and review integration testing before production migrations.
• Provide high level of technical support and perform root cause analysis for problems experienced within area of functional responsibility.
• Can document technical specs from business :
• To perform this job successfully, an individual must be able to perform each essential duty satisfactorily.
• List knowledge, skills, and / or abilities required. Reasonable accommodations may be made to enable individuals with disabilities to perform essential functions.
• 6+ years of ETL experience.
• Experience with core Python programming for data transformation.
• Intermediate-level Python and PySpark skills. Can write, read, understand and debug existing code Python and ySpark code from scratch.
• Strong knowledge of SQL fundamentals and snowflake experience, understanding of subqueries, can tune queries with execution hints to improve performance.
• Able to write SQL code sufficient for most business requirements for pulling data from sources, applying rules to the data, and stocking target data
• Proven track record in troubleshooting ETL jobs and addressing production issues like performance tuning, reject handling, and ad-hoc reloads.
• Proficient in developing optimization strategies for ETL processes.
• Basic AWS technical support skills. Has ability to log in, find existing jobs and check run status and logs
• Will run and monitor jobs running via Control-M
• Can create clear and concise documentation and communications.
• Can document technical specs from business communications.
• Ability to coordinate and aggressively follow up on incidents and problems, perform diagnosis, and provide resolution to minimize service interruption
• Ability to prioritize and work on multiple tasks simultaneously
• Effective in cross-functional and global environments to manage multiple tasks and assignments concurrently with effective communication skills.
• A self-starter who can work well independently and on team projects.
• Experienced in analyzing business requirements, defining the granularity, source to target mapping of the data elements, and full technical specification.
• Understands data dependencies and how to schedule jobs in Control-M.
• Experienced working at the command line in various flavors of UNIX, with basic understanding of shell scripting in bash and korn and / or Experience :
• Include the education and experience that is necessary to perform the job satisfactorily.
• Bachelors of Science in computer science or equivalent
• 7+ years of ETL and SQL experience
• 3+ years of python and PySpark experience
• 3+ Snowflake experience
• 3+ years of AWS and unix experience

Preferred certifications :

ref : hirist.tech)