Talent.com
This job offer is not available in your country.
Big Data Engineer - Spark / Scala

Big Data Engineer - Spark / Scala

IT FirmIndia
10 days ago
Job description

Position : Big Data Engineer

Industry : Information Technology / Data Engineering

Department : Data Engineering

Job Overview :

We are looking for a highly skilled and motivated Big Data Engineer with expertise in Apache Spark, Scala, and Java to join our dynamic team. The ideal candidate will have experience in designing, developing, and implementing scalable and high-performance data processing systems. The successful candidate will work on large-scale data sets and build robust data pipelines to extract, transform, and load (ETL) complex data from various sources to facilitate analytics and :

  • Design and develop scalable, high-performance data pipelines using Apache Spark.
  • Write efficient code in Scala and Java for distributed data processing tasks.
  • Collaborate with data scientists, analysts, and other engineers to integrate various data sources and ensure smooth data flow.
  • Create and maintain ETL processes that handle structured, semi-structured, and unstructured data.
  • Optimize performance of data processing tasks, identify bottlenecks, and implement solutions to improve efficiency.
  • Perform data ingestion from various sources, including relational databases, APIs, and file systems.
  • Develop and maintain real-time data streaming solutions using Apache Kafka and Spark Streaming.
  • Ensure data quality and integrity throughout the data pipeline.
  • Troubleshoot and debug issues across the data pipeline.
  • Work with cloud platforms (e.g., AWS, Azure, or GCP) to deploy and scale Big Data solutions.
  • Build and maintain data models and assist with data architecture decisions.
  • Conduct unit tests and ensure the deployment of high-quality code.
  • Stay up to date with industry trends and emerging technologies in the Big Data ecosystem.

Required Skills and Qualifications :

  • Bachelor's or Master's degree in Computer Science, Engineering, or related field.
  • 5+ years of hands-on experience in Big Data technologies, specifically with Apache Spark.
  • Proficient in Scala and Java programming languages.
  • Strong knowledge of Hadoop ecosystem (Hive, HBase, Pig, etc.).
  • Expertise in Spark SQL and working with structured and unstructured data.
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra).
  • Understanding of distributed computing concepts and parallel processing.
  • Hands-on experience with Apache Kafka, Flume, or similar messaging platforms.
  • Familiarity with cloud-based Big Data solutions (AWS, Google Cloud, or Microsoft Azure).
  • Experience in data integration and transformation using ETL tools.
  • Strong problem-solving skills with the ability to troubleshoot complex data-related issues.
  • Familiarity with data warehousing concepts and tools.
  • Experience with version control tools (e.g., Git).
  • Ability to work in an agile, fast-paced environment.
  • Preferred Skills :

  • Knowledge of Apache Hudi or Delta Lake for managing large datasets.
  • Experience with Kubernetes and Docker for containerized deployment.
  • Familiarity with Apache Airflow for orchestrating data workflows.
  • Understanding of Machine Learning concepts and integration with Big Data platforms.
  • Prior experience working in an Agile team environment.
  • Benefits :

  • Competitive salary and benefits package.
  • Work with cutting-edge technologies in a collaborative and fast-paced environment.
  • Opportunity for career growth and development in Big Data and cloud technologies.
  • Flexible working hours and remote work options.
  • Health and wellness programs.
  • ref : hirist.tech)

    Create a job alert for this search

    Big Data Engineer • India