Spark Cluster Admin

Introduction

Joining the IBM Technology Expert Labs teams means you'll have a career delivering world-class services for our clients. As the ultimate expert in IBM products, you'll bring together all the necessary technology and services to help customers solve their most challenging problems. Working in IBM Technology Expert Labs means accelerating the time to value confidently and ensuring speed and insight while our clients focus on what they do best—running and growing their business.

Excellent onboarding and industry-leading learning culture will set you up for a positive impact, while advancing your career. Our culture is collaborative and experiential. As part of a team, you will be surrounded by bright minds and keen co-creators—always willing to help and be helped—as you apply passion to work that will positively impact the world around us.

Your Role And Responsibilities

We are seeking a highly skilled and experienced Spark Cluster Admin to join our dynamic team. This role is critical for building, optimizing, and supporting our cutting-edge data platform, leveraging Apache Spark , Apache Iceberg , and a robust DevOps approach within an OpenShift environment. The ideal candidate will be adept at both developing high-performance data solutions and ensuring their stability and reliability in a production setting.

Key Responsibilities

• Design, develop, and optimize scalable and resilient data processing applications using Apache Spark (batch, streaming, and real-time).
• Implement and manage data pipelines, ensuring data quality, consistency, and performance.
• Perform Spark job performance tuning and optimization to handle large-scale datasets efficiently.
• Manage and automate the deployment of Spark applications within OpenShift clusters, utilizing Docker and Kubernetes.
• Establish and maintain CI / CD pipelines for automated testing, deployment, and release management of Spark workloads.
• Provide comprehensive production support for critical Spark jobs, including proactive monitoring, troubleshooting, debugging, and participation in on-call rotations.
• Work extensively with Apache Iceberg table format, leveraging its capabilities for schema evolution, time travel, hidden partitioning, and ACID transactions.
• Collaborate closely with data scientists, other data engineers, and operations teams to deliver robust and integrated solutions.
• Develop and maintain documentation for data pipelines, job configurations, and operational procedures.

Preferred Education

Master's Degree

Required Technical And Professional Expertise

Preferred Technical And Professional Experience

Skills Required

Java, Devops, Docker, Pyspark, Openshift, Scala, Apache Spark, Sql, Python, Kubernetes