Apache Spark Developer

Responsibilities

• Develop and maintain high-quality data processing applications using Apache Spark and related technologies.
• Design and implement data transformations with a focus on performance and scalability.
• Collaborate with data engineers and analysts to integrate data sources and services.
• Write and execute unit tests to ensure code quality and reliability.
• Participate in code reviews and adhere to best coding practices.
• Work within an Agile development framework to deliver features and improvements.
• Troubleshoot and resolve issues in existing data processing applications.
• Stay up-to-date with the latest industry trends and technologies.
• Mentor junior developers and provide technical guidance.
• Communicate effectively with team members and stakeholders to ensure project success.
• Manage multiple tasks and meet deadlines in a fast-paced environment.
• Utilize tools like JIRA, Confluence, and Git/Bitbucket for project management and version control.
• Experience with testing frameworks like ScalaTest or JUnit.

Qualifications

• Bachelor s degree in Computer Science, Software Engineering, or a closely related field.
• Technical Requirement:
• 4+ years of experience with Apache Spark (preferably Spark 3.0 or above).
• Proficiency in Scala or Java programming languages.
• Experience with big data technologies like Hadoop, Kafka, or HDFS.
• Experience with Orchestration tool like airflow.
• Experience working with Managed cloud Spark infrastructure like AWS EMR and in general AWS tools like S3, SQS, VPC and API Gateway.
• Familiarity with data processing frameworks and libraries.
• Strong understanding of distributed computing and parallel processing.
• Knowledge of Agile methodologies and good coding practices.
• Ability to work both independently and as part of a team.
• Strong communication and interpersonal skills.
• Experience with version control systems like Bit Bucket/Git.
• Experience with CI/CD tools like Jenkins, Docker, or Kubernetes.

Skills Required
Apache Spark, Hadoop, Apache Airflow, Kafka