EY - GDS Consulting - AI and DATA - Hadoop - Senior

The opportunity

We are looking for a Senior Data Engineer for Senior technology Consultant role with 6+ years of hands-on experience in developing and maintaining large-scale data processing systems. Candidate will have deep, practical expertise in the big data ecosystem, specifically in Spark, Hive, Hadoop, and Apache Iceberg. Candidate should be a key contributor on the team, responsible for building efficient, reliable, and scalable data pipelines that feed our data Lakehouse and power analytics.

Your key responsibilities

• Experience of 6+ years in data engineering with a proven track record of building and maintaining large-scale data processing systems.
• Hands-on programming experience in Apache Spark using Spark SQL, Pyspark SQL and HQL.
• Hands on experience with major components like Hive, PIG, Spark, MapReduce.
• Hands on writing HiveQL queries and Spark jobs to ensure high performance and cost-effectiveness. Tune jobs for efficient resource utilization in a distributed cluster environment.
• Hands on Building and maintaining ETL / ELT processes to ingest data from a variety of sources (databases, APIs, logs, streams) into our central data platform.
• Strong understanding & familiarity with all Hadoop Ecosystem
• Experienced in Hadoop clustering and Auto scaling.
• Knowledge of Apache Oozie based workflow.
• Solid understanding of ETL methodologies in a multi-tiered stack, integrating with Big Data systems like Hadoop and Cassandra.
• Define and develop client specific business rules around data within a Hadoop environment on Azure cloud
• Design and develop data ingestion programs to process large data sets in Batch mode using HIVE, Pig and Sqoop technologies.
• Experience on clean, maintainable, and well-documented code with best practices, including version control (Git), code reviews, unit testing, and CI / CD.
• Strong UNIX operating system concepts and shell scripting knowledge.
• Flexible and proactive / self-motivated working style with strong personal ownership of problem resolution.
• Knowledge on Anti Money Laundering- Transaction Monitoring will be an added advantage
• Experience in converting business problems / challenges to technical solutions considering security, performance, scalability etc.
• Excellent communicator (written and verbal formal and informal).
• Ability to multi-task under pressure and work independently with minimal supervision.
• Strong verbal and written communication skills.
• Must be a team player and enjoy working in a cooperative and collaborative team environment.
• Adaptable to new technologies and standards.
• Minimum 6 years hand-on experience in one or more of the above areas.

Skills and attributes for success

Ideally, you’ll also have

What we look for

What working at EY offers

At EY, we’re dedicated to helping our clients, from startups to Fortune 500 companies — and the work we do with them is as varied as they are.

You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees, and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer :

EY | Building a better working world