This job offer is not available in your country.
Softility - Big Data Engineer - Apache Spark
SoftilityHyderabad10 days ago
Job descriptionManage and maintain data storage solutions on Hadoop / HDFS. Implement data models and schemas in Hive for a data warehouse and reporting layer. Work with HBase for specific use cases requiring fast, random-access to large datasets, leveraging Phoenix / SQL line for SQL-based access. Workflow Orchestration : Develop and manage complex data workflows and dependencies using Apache Oozie. Utilize Informatica for traditional ETL workflows and Apache Sqoop to efficiently transfer data between RDBMS and the Hadoop ecosystem. Work with YARN to manage cluster resources, monitor job execution, and ensure high availability and fault tolerance. Monitor the health and performance of the data platform using tools like Hue and New Relic. Proactively identify and resolve issues. Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions. Experience with containerization and cloud-native solutions, particularly Anthos, for deploying and managing applications. Familiarity with data observability and logging platforms like Cribl for advanced data collection and routing. Strong expertise in Hadoop and HDFS. Proficiency in Apache Spark for both batch and stream processing. Hands-on experience with Apache Hive and HBase. Knowledge of data ingestion tools like Apache Kafka and Apache Nifi. Experience with Apache Oozie or other workflow schedulers. Familiarity with Apache Sqoop for RDBMS integration. Understanding of YARN for cluster resource management. Proficiency in at least one scripting language (e.g., Python, Scala). Familiarity with monitoring tools like Hue and New Relic. Experience with Informatica is a plus. Excellent problem-solving and analytical skills. Strong communication and collaboration abilities. Ability to work in a fast-paced, agile environment. Bachelors or Masters degree in Computer Science, Data Science, or a related field.
Big Data Engineer :
- Design, build, and optimize robust and scalable data pipelines for both batch and real-time data ingestion using Apache Spark (Streaming + Batch), Apache Nifi, and Apache Kafka.
Data Storage and Management :
ETL and Data Integration :
Resource Management :
Monitoring and Maintenance :
Collaboration :
Cloud and Advanced Skills (Good to Have) :
Qualifications :
Experience : Proven experience as a Big Data Engineer or a similar role.
Technical Skills :
Soft Skills :
Education :
(ref : hirist.tech)
Create a job alert for this search
Big Data Engineer • Hyderabad
Related jobs
- Promoted
Senior Big Data Engineer
Grid Dynamicshyderabad, telangana, in Proven experience leading data engineering teams, including distributed teams across multiple geographies and time zones.
Effective in managing cross-team collaboration with architects, product mana...Show moreLast updated: 20 days ago
- Promoted
Big Data Developer - Scala / Spark
SourcingxpressHyderabad Company : : Visit Type : Consulting Firm Company Type : Model : ...Show moreLast updated: 8 days ago
- Promoted
Data Engineer - Databricks
Tata Consultancy ServicesHyderabad, Telangana, India TCS is hiring Data Engineer (Databricks).Years of Experience – 10-16 years.Senior Data Engineer – Databricks.Databricks, PySpark, Delta Lake, MLflow, Azure / AWS / GCP, Data Lakehouse architecture, CI / ...Show moreLast updated: 26 days ago
- Promoted
Dun & Bradstreet - Data Engineer - Spark / Hadoop
DUN BRADSTREET TECHNOLOGY AND CORPORATE SERVICESHyderabad Why We Work at Dun & Bradstreet : Dun & Bradstreet unlocks the power of data through analytics, creating a better tomorrow.Each da...Show moreLast updated: 12 days ago
- Promoted
Data Engineer
Otomeyt AIHyderabad, IN We are seeking a highly skilled 5+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 30+ days ago
- Promoted
Senior Data Engineer
StraiveHyderabad, Telangana, India The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show moreLast updated: 3 days ago
- Promoted
Big Data Engineer - Hadoop / Spark
ImpacteersHydrebad Position Overview : We are looking for a skilled and detail-oriented Big Data Engineer to design, develop, and maintain scalable data pipelines and architectures.The ...Show moreLast updated: 13 days ago
- Promoted
Big Data Engineer - Python / PySpark
People Prime World WideHyderabad Key Responsibilities : - Design, develop, and maintain scalable ETL / ELT pipelines using Python and PySpark.Work with large-scale datasets across di...Show moreLast updated: 10 days ago
- Promoted
Senior Big Data Engineer
VeltrisHyderabad, IN Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries.
We specialize in delivering next-gene...Show moreLast updated: 14 days ago
- Promoted
Senior Data Engineer
Otomeyt AIHyderabad, IN We are seeking a highly skilled 7+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 18 days ago
- Promoted
Big Data Engineer
Rapinno Tech Solutions Pvt.Ltd.hyderabad, telangana, in Locations : Hyderabad | Chennai | Pune | Bangalore.Skills : Scala, Python, Apache Spark, PySpark, AWS, CI / CD.Strong hands-on experience in Scala & distributed data processing.Show moreLast updated: 7 days ago
- Promoted
Data Engineer
ACL Digitalsecunderabad, telangana, in Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.Design, develop, and optimize AWS pipeline as applicable.
Implement and manage GitHub asset bun...Show moreLast updated: 30+ days ago
- Promoted
Scala Big Data Lead Engineer - 7 YoE - Immediate Joiner - Any UST Location
USTsecunderabad, telangana, in CTC, notice period, and current location details to.Apache Hadoop, Airflow, Kubernetes, and Containers.Troubleshoot Hadoop log files and work with.
Required Skills & Qualifications : .Scala, Spark, Py...Show moreLast updated: 30+ days ago
- Promoted
- New!
Big Data with Quantexa-Pan India
Tata Consultancy Serviceshyderabad, telangana, in We are looking for Big data developer with Quantexa as mandatory skill.Experience Range- 5 to 12 years.Data Engineer - Big Data Hadoop, Spark and Scala.
Spark, Scala, Hadoop, HIVE, Oozie, Shell Scri...Show moreLast updated: 6 hours ago
- Promoted
Bigdata Engineer
TAO Digital SolutionsHyderabad, Telangana, India Highly skilled Big Data Engineer with expertise in distributed systems, and advanced programming techniques.Candidate will possess experience in developing highly scalable Big Data pipelines, stron...Show moreLast updated: 7 days ago
- Promoted
GCP Big Data Engineer
Talentmaticssecunderabad, telangana, in The ideal candidate will bring strong leadership capabilities, technical depth, and a proven track record of delivering end-to-end big data solutions in cloud environments.Lead and mentor teams in ...Show moreLast updated: 5 days ago
- Promoted
- New!
Data Engineer
TechVeritohyderabad, telangana, in You will play a critical role in designing, building, and optimizing data workflows that enable scalable analytics and real-time insights.
The ideal candidate is hands-on, detail-oriented, and passi...Show moreLast updated: 6 hours ago
- Promoted
Big Data Developer
Kresta Softech Private LimitedHyderabad, Telangana, India Immediate / 7 days joiners mandate.Must have skills- Big Data, Scala, AWS and Python or Java.Show moreLast updated: 20 days ago