PySpark Developer

Key Skills & Responsibilities

• Strong expertise in PySpark and Apache Spark for batch and real-time data processing.
• Experience in designing and implementing ETL pipelines, including data ingestion, transformation, and validation.
• Proficiency in Python for scripting, automation, and building reusable components.
• Hands-on experience with scheduling tools like Airflow or Control-M to orchestrate workflows.
• Familiarity with AWS ecosystem, especially S3 and related file system operations.
• Strong understanding of Unix / Linux environments and Shell scripting.
• Experience with Hadoop, Hive, and platforms like Cloudera or Hortonworks.
• Ability to handle CDC (Change Data Capture) operations on large datasets.
• Experience in performance tuning, optimizing Spark jobs, and troubleshooting.
• Strong knowledge of data modeling, data validation, and writing unit test cases.
• Exposure to real-time and batch integration with downstream / upstream systems.
• Working knowledge of Jupyter Notebook, Zeppelin, or PyCharm for development and debugging.
• Understanding of Agile methodologies, with experience in CI / CD tools (e.g., Jenkins, Git).

Preferred Skills

Skills Required

Pyspark, Hadoop, Spark SQL, Aws, Scala