Big Data Engineer

K&K Talents is an international recruiting agency that has been providing technical resources globally since 1993. This position is with one of our clients in India , who is actively hiring candidates to expand their teams.

Title : Big Data Engineer (Scala & Spark)

Location : PAN India (Hybrid)

Employment Type : Permanent

Notice Period : 15 Days

Required Experience : 8 Years and above

Role

We are seeking a highly skilled Senior Data Engineer with strong expertise in Azure cloud, Big Data technologies, and distributed data processing frameworks . The ideal candidate will design, build, optimize, and manage large-scale data pipelines that support both batch and real-time analytics. This role requires hands-on experience in Azure Databricks, ADF, Synapse, Spark (Scala / PySpark) , and modern ETL / data integration practices.

Responsibilities

• Design and build scalable, high-performance data pipelines using Azure Databricks, ADF, and Spark.
• Develop and optimize Spark (Scala / PySpark) jobs for large-volume data processing and transformation.
• Manage and integrate data from diverse sources including ADLS, Blob Storage, SQL databases, NoSQL stores, and streaming platforms .
• Implement and support real-time data streaming using Kafka, Spark Streaming, or Flink.
• Create and maintain ETL workflows , ensuring data quality, reliability, and consistency.
• Work with Azure Synapse Analytics, Snowflake, or BigQuery for warehousing and analytics solutions.
• Build and maintain CI / CD pipelines using Azure DevOps / Jenkins for automated deployments.
• Collaborate with data architects, analysts, and data scientists to enable analytics, ML workflows, and business insights.
• Optimize query performance, storage, cost, and compute usage across pipelines and cloud environments.
• Ensure data governance , security, lineage, and compliance best practices are followed.
• Participate in Agile processes, code reviews, documentation, and production support.

Skills & Experience

Preferred Qualifications