Talent.com
This job offer is not available in your country.
Lead Data Engineer - Scala / Spark

Lead Data Engineer - Scala / Spark

Ixceed SolutionsChennai
20 days ago
Job description

About the Role :

We are seeking a highly skilled and experienced Lead Data Engineer / Scala Developer with strong hands-on expertise in Scala programming, Apache Spark, Databricks, Python, and PySpark. The ideal candidate will lead the design, development, and deployment of scalable big data solutions, working closely with cross-functional teams to deliver high-quality data processing pipelines and analytics platforms.

Key Responsibilities :

  • Lead the development and implementation of big data solutions using Scala and Apache Spark on Databricks.
  • Design, develop, and optimize Spark jobs and ETL pipelines using Scala, Python, and PySpark.
  • Collaborate with data engineers, data scientists, and business stakeholders to understand requirements and translate them into technical solutions.
  • Mentor and guide junior developers and team members on best practices in Scala and Spark development.
  • Ensure code quality, performance tuning, and scalability of Spark applications.
  • Participate in architectural discussions and contribute to technology roadmap decisions.
  • Troubleshoot and resolve production issues related to Spark jobs and data pipelines.
  • Implement CI / CD pipelines for Spark applications and automate deployment processes.
  • Stay updated with the latest trends and advancements in Scala, Spark, and big data technologies.

Required Skills and Qualifications :

  • Strong hands-on experience in Scala programming with a deep understanding of functional programming concepts.
  • Extensive experience with Apache Spark (RDDs, DataFrames, Datasets) and Databricks platform.
  • Proficient in Python and PySpark for big data processing.
  • Experience in building and optimizing large-scale ETL pipelines and data workflows.
  • Solid understanding of distributed computing and big data ecosystem.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP, especially Databricks on cloud.
  • Experience with version control systems (Git) and CI / CD tools.
  • Strong problem-solving skills and ability to work in an agile environment.
  • Excellent communication and leadership skills to manage and mentor a development team.

    • (ref : hirist.tech)

    Create a job alert for this search

    Lead Data Engineer • Chennai