Talent.com
Softility - Big Data Engineer - Apache Spark
Softility - Big Data Engineer - Apache SparkSoftility • Hyderabad
Softility - Big Data Engineer - Apache Spark

Softility - Big Data Engineer - Apache Spark

Softility • Hyderabad
30+ days ago
Job description

Big Data Engineer :

  • Design, build, and optimize robust and scalable data pipelines for both batch and real-time data ingestion using Apache Spark (Streaming + Batch), Apache Nifi, and Apache Kafka.

Data Storage and Management :

  • Manage and maintain data storage solutions on Hadoop / HDFS. Implement data models and schemas in Hive for a data warehouse and reporting layer.
  • Work with HBase for specific use cases requiring fast, random-access to large datasets, leveraging Phoenix / SQL line for SQL-based access.
  • Workflow Orchestration : Develop and manage complex data workflows and dependencies using Apache Oozie.
  • ETL and Data Integration :

  • Utilize Informatica for traditional ETL workflows and Apache Sqoop to efficiently transfer data between RDBMS and the Hadoop ecosystem.
  • Resource Management :

  • Work with YARN to manage cluster resources, monitor job execution, and ensure high availability and fault tolerance.
  • Monitoring and Maintenance :

  • Monitor the health and performance of the data platform using tools like Hue and New Relic. Proactively identify and resolve issues.
  • Collaboration :

  • Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
  • Cloud and Advanced Skills (Good to Have) :

  • Experience with containerization and cloud-native solutions, particularly Anthos, for deploying and managing applications.
  • Familiarity with data observability and logging platforms like Cribl for advanced data collection and routing.
  • Qualifications :

    Experience : Proven experience as a Big Data Engineer or a similar role.

    Technical Skills :

  • Strong expertise in Hadoop and HDFS.
  • Proficiency in Apache Spark for both batch and stream processing.
  • Hands-on experience with Apache Hive and HBase.
  • Knowledge of data ingestion tools like Apache Kafka and Apache Nifi.
  • Experience with Apache Oozie or other workflow schedulers.
  • Familiarity with Apache Sqoop for RDBMS integration.
  • Understanding of YARN for cluster resource management.
  • Proficiency in at least one scripting language (e.g., Python, Scala).
  • Familiarity with monitoring tools like Hue and New Relic.
  • Experience with Informatica is a plus.
  • Soft Skills :

  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration abilities.
  • Ability to work in a fast-paced, agile environment.
  • Education :

  • Bachelors or Masters degree in Computer Science, Data Science, or a related field.
  • (ref : hirist.tech)

    Create a job alert for this search

    Big Data Engineer • Hyderabad

    Related jobs
    Data Engineer – Databricks Platform

    Data Engineer – Databricks Platform

    Amicon Hub Services • Hyderabad, Telangana, India
    Delta Lake, Spark, PySpark, SQL).SQL Server, MongoDB, InfluxDB).Kafka, Azure Event Hubs, or similar).Excellent problem-solving skills and the ability to work in a fast-paced environment.Familiar wi...Show more
    Last updated: 25 days ago • Promoted
    Big Data Solutions Engineer

    Big Data Solutions Engineer

    Tata Consultancy Services • Hyderabad, Republic Of India, IN
    Advanced working Scala ,SQL,Python / PySpark knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.Experience bu...Show more
    Last updated: 14 days ago • Promoted
    Big Data Engineer - Scala / PySpark

    Big Data Engineer - Scala / PySpark

    Sheryl strategic solutions Pvt. LTD. • Hyderabad
    Description : Position : Big Data Engineer Location : Hyderabad, India (Hybrid, 2-3 times per week on any weekdays) Duratio...Show more
    Last updated: 30+ days ago • Promoted
    Big Data Engineer

    Big Data Engineer

    LiveRamp • Hyderabad, Republic Of India, IN
    LiveRamp is the data collaboration platform of choice for the world’s most innovative companies.A groundbreaking leader in consumer privacy, data ethics, and foundational identity, LiveRamp is sett...Show more
    Last updated: 14 days ago • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Straive • Hyderabad, Telangana, India
    The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show more
    Last updated: 30+ days ago • Promoted
    Senior Data Engineer

    Senior Data Engineer

    SGS & Co • hyderabad, telangana, in
    Position Title : Senior Data Engineer.Experience Required : 8 to 12 Years.We are looking for a highly skilled and experienced Data Engineer with strong expertise in. The ideal candidate will play a ke...Show more
    Last updated: 14 days ago • Promoted
    Big Data Developer

    Big Data Developer

    Kresta Softech Private Limited • Hyderabad, India
    Notice Period : Immediate / 7 days joiners mandate.Must have skills- Big Data, Scala, AWS and Python.Show more
    Last updated: 30+ days ago • Promoted
    Big Data Engineer - Python / PySpark

    Big Data Engineer - Python / PySpark

    People Prime World Wide • Hyderabad
    Key Responsibilities : - Design, develop, and maintain scalable ETL / ELT pipelines using Python and PySpark.Work with large-scale datasets across di...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer - Scala / Apache Spark

    Data Engineer - Scala / Apache Spark

    Ixceed Solutions • Hyderabad
    Responsibilities : - Design, develop, and maintain robust and scalable data pipelines using Apache Spark and Scala on the Databricks platform. Implement ETL (Extract, Transform,...Show more
    Last updated: 26 days ago • Promoted
    Data Engineer (Spark / Scala)

    Data Engineer (Spark / Scala)

    Tata Consultancy Services • Hyderabad, Telangana, India
    TCS is Hiring Data Engineer (Spark / Scala)!!.Location : Hyderabad, Bangalore, Chennai, Pune, Gurugram.Experience : 7 - 10 yrs (Accurate). Date of Interview : 30th October,2025.Notice Period : 0-30 Days / ...Show more
    Last updated: 24 days ago • Promoted
    Big Data Engineer

    Big Data Engineer

    People Prime Worldwide • Hyderabad, Telangana, India
    Our client is a trusted global innovator of IT and business services, present in 50+ countries.They specialize in digital & IT modernization, consulting, managed services, and industry-specific sol...Show more
    Last updated: 2 days ago • Promoted
    Data Engineer

    Data Engineer

    One Tapp Consulting • Hyderabad, IN
    Design and Development : Design, build, and maintain robust, scalable, and optimized ETL / ELT.Python and standard data warehousing principles. Data Platform Management : Implement and manage data proce...Show more
    Last updated: 12 hours ago • Promoted • New!
    Senior Data Engineer

    Senior Data Engineer

    Adaptive Technology Insights • hyderabad, telangana, in
    We are looking for an experienced.You will be responsible for designing, building, and optimizing data pipelines and architecture to support analytics, reporting, and data-driven decision-making ac...Show more
    Last updated: 1 day ago • Promoted
    Big Data Engineer (Spark / Scala)

    Big Data Engineer (Spark / Scala)

    Tata Consultancy Services • Hyderabad, Republic Of India, IN
    TCS is Hiring Data Engineer (Spark / Scala)!!.Location : Hyderabad, Bangalore, Chennai, Pune, Gurugram.Experience : 7 - 10 yrs (Accurate). Date of Interview : 30th October,2025.Notice Period : 0-30 Days / ...Show more
    Last updated: 24 days ago • Promoted
    Lead Data Pipeline Engineer

    Lead Data Pipeline Engineer

    Straive • Hyderabad, Republic Of India, IN
    The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer

    Data Engineer

    Straive • Hyderabad, Telangana, India
    Key Responsibilities : • Design, build and maintain scalable batch and (optionally) streaming data pipelines using Apache Spark 3. Scala, Python & SQL • Implement core ETL / ELT logic in Scala and Py...Show more
    Last updated: 30+ days ago • Promoted
    Senior Databricks Engineer (AWS,Spark / databricks, SQL)

    Senior Databricks Engineer (AWS,Spark / databricks, SQL)

    SID Information Technologies • hyderabad, telangana, in
    Role : Senior Data Engineer (Python, Spark / Databricks, SQL, AWS).Hybrid (3 days / week in-office).S3, Glue, Redshift, EMR, Lambda. Strong SQL skills for data transformation and analytics.Expertise in s...Show more
    Last updated: 11 hours ago • Promoted • New!
    Spark / Scala Data Solutions Engineer

    Spark / Scala Data Solutions Engineer

    Tata Consultancy Services • Hyderabad, Republic Of India, IN
    TCS is Hiring Data Engineer (Spark / Scala)!!.Location : Hyderabad, Bangalore, Chennai, Pune, Gurugram.Experience : 7 - 10 yrs (Accurate). Date of Interview : 30th October,2025.Notice Period : 0-30 Days / ...Show more
    Last updated: 24 days ago • Promoted