Talent.com
This job offer is not available in your country.
Lead Big Data Engineer - Spark

Lead Big Data Engineer - Spark

QUALYS SECURITY TECHSERVICES PRIVATE LIMITEDPune
14 days ago
Job description

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!.

Description :

We are seeking a talented Lead Big Data Engineer to deliver roadmap features of Enterprise TruRisk Platform which would help customers to Measure, Communicate and Eliminate Cyber Risks.

Working with a team of engineers and architects, you will be responsible for prototyping, designing, developing and supporting a highly scalable, distributed SaaS based Security Risk Prioritization product.

This is a fantastic opportunity to be an integral part of a team building Qualys next generation platform using Big Data & Micro-Services based technology to process over billions of transactions data per day, leverage open-source technologies, and work on challenging and business-impacting :

  • Be the thought leader in data platform and pipeline along with Risk Evaluation.
  • Provide technical leadership to the engineering organization on data platform design, roll out and evolution.
  • Liason to product teams, professional services and sales engineers on solution and trade-off reviews and represent engineering in such conversations.
  • Drive technology explorations and roadmaps.
  • Serve as a technical lead on our most demanding, cross-functional departments.
  • Ensure the quality of architecture and design of systems.
  • Functionally decompose complex problems into simple, straight-forward solutions.
  • Fully and completely understand system interdependencies and limitations.
  • Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices.
  • Leverage knowledge of internal and industry prior art in design decisions.
  • Effectively research and benchmark cloud technology against other competing systems in the industry.
  • Able to document the details so it will be easy for developers to understand the requirements.
  • Assisting developers with proper requirements and directions.
  • Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues and helping managers guide the career growth of their team members.
  • Exert technical influence over multiple teams, increasing their productivity and effectiveness by sharing your deep knowledge and experience.
  • Able to share knowledge and train others.

Qualifications :

  • Bachelors degree in computer science or equivalent.
  • 8+ years of total experience.
  • 4+ years of relevant experience in design and architecture Big Data solutions using Spark.
  • 3+ years experience in working with engineering resources for innovation.
  • 4+ years experience in understanding Big Data events flow pipeline.
  • 3+ years experience in performance testing for large infrastructure.
  • 3+ In depth experience in understanding various search solutions solr / elastic.
  • 3+ years experience in Kafka.
  • In depth experience in Data lakes and related ecosystems.
  • In depth experience of messing queue.
  • In depth experience in giving requirements to build a scalable architecture for Big data and Micro-services environments.
  • In depth experience in understanding caching components or services.
  • Knowledge in Presto technology.
  • Knowledge in Airflow.
  • Hands-on experience in scripting and automation.
  • In depth understanding of RDBMS / NoSQL, Oracle , Cassandra , Kafka , Redis, Hadoop, lambda architecture, kappa , kappa ++ architectures with flink data streaming and rule engines.
  • Experience in working with ML models engineering and related deployment.
  • Design and implement secure big data clusters to meet many compliances and regulatory requirements.
  • Experience in leading the delivery of large-scale systems focused on managing the infrastructure layer of the technology stack.
  • Strong experience in doing performance benchmarking testing for Big data technologies.
  • Strong troubleshooting skills.
  • Experience leading development life cycle process and best practices.
  • Experience in Big Data services administration would be added value.
  • Experience with Agile Management (SCRUM, RUP, XP), OO Modeling, working on internet, UNIX, Middleware, and database related projects.
  • Experience mentoring / training the engineering community on complex technical issue.
  • Project management experience.
  • (ref : hirist.tech)

    Create a job alert for this search

    Big Data Engineer • Pune

    Related jobs
    • Promoted
    Lead Aws Data Engineer

    Lead Aws Data Engineer

    ACL Digitalpune, maharashtra, in
    Multi-year work experience as a Platform Lead, Delivery Leader or similar role.Good working knowledge of Aws Services, Python, Pyspark, data modelling. Knowledge of columnar and NoSQL databases, pre...Show moreLast updated: 17 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Otomeyt AIPune, IN
    We are seeking a highly skilled 7+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 17 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    DeltacubesPune, IN
    Build and maintain scalable ETL / ELT pipelines.Work with Snowflake and BigQuery for data storage.Implement orchestration with Airflow or Prefect. Integrate data workflows with Python.Optimize data pi...Show moreLast updated: 14 days ago
    • Promoted
    Data Engineer

    Data Engineer

    ACL Digitalpune, maharashtra, in
    Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing.Design, develop, and optimize AWS pipeline as applicable. Implement and manage GitHub asset bun...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Big Data Engineer

    Lead Big Data Engineer

    Qualyspune, maharashtra, in
    Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!.We are seeking a talented Lead Big Data Engineer to deliver roadmap features of ...Show moreLast updated: 16 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Alp Consulting Ltd.pune, maharashtra, in
    Data Software Engineer - Spark, Python, (Azure Databricks ).Years of in Big Data & Data related technology experience.Expert level understanding of distributed computing principles.Expert level kno...Show moreLast updated: 18 days ago
    • Promoted
    Lead Big Data Engineer

    Lead Big Data Engineer

    WorkiFicientPune
    Job Description : Roles And Responsibility : - Collaborate closely with Product Management and Engineering leadership to devis...Show moreLast updated: 30+ days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Eucloid Data SolutionsPune, IN
    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications.The ideal candidate will support development of data infrastructure on Databricks...Show moreLast updated: 7 days ago
    • Promoted
    Senior Big Data Engineer

    Senior Big Data Engineer

    VeltrisPune, IN
    Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries. We specialize in delivering next-gene...Show moreLast updated: 14 days ago
    • Promoted
    Big Data Developer

    Big Data Developer

    USTpune, maharashtra, in
    The ideal candidate should demonstrate a deep understanding of big data concepts, programming fundamentals, and the ability to solve complex problems related to scalability, failure handling, and o...Show moreLast updated: 4 days ago
    • Promoted
    Big Data Engineer - Apache Spark

    Big Data Engineer - Apache Spark

    Merito.aiPune
    Job Description : We are looking for a Big Data Engineer who will work on building, and managing Big Data Pipelines for us to deal with the huge structured data sets ...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    Straivepune, maharashtra, in
    The ideal candidate is a strong software engineer with hands-on experience in Spark (3.You'll be responsible for designing and implementing ETL / ELT solutions, collaborating with teams to deliver da...Show moreLast updated: 4 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    ACL Digitalpune, maharashtra, in
    Experience : - 10 years to 13 years.Location : Pune,Bangalore,Chennai.Notice Period : - Immediate Joiner only.Experience in Python & pyspark. Experience in ETL Development, ETL pipeline.Experience in Dat...Show moreLast updated: 7 days ago
    • Promoted
    GCP Big Data Engineer

    GCP Big Data Engineer

    Talentmaticspune, maharashtra, in
    The ideal candidate will bring strong leadership capabilities, technical depth, and a proven track record of delivering end-to-end big data solutions in cloud environments.Lead and mentor teams in ...Show moreLast updated: 4 days ago
    • Promoted
    Big Data Engineer - Spark / Scala

    Big Data Engineer - Spark / Scala

    Leinex ConsultingPune
    Role & responsibilities : - Evaluate domain, financial and technical feasibility of solution ideas with help of all key stakeholders < / p&...Show moreLast updated: 6 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Otomeyt AIPune, IN
    We are seeking a highly skilled 5+.The ideal candidate will have strong technical expertise in Azure, Data Engineering tools, and advanced ETL design along with excellent communication and problem-...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    InfogainPune, IN
    Big Data Engineer (Lead) : As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers. You will work closely with clients and team members to understand their req...Show moreLast updated: 30+ days ago
    • Promoted
    Scala Big Data Lead Engineer - 7 YoE - Immediate Joiner - Any UST Location

    Scala Big Data Lead Engineer - 7 YoE - Immediate Joiner - Any UST Location

    USTpune, maharashtra, in
    CTC, notice period, and current location details to.Apache Hadoop, Airflow, Kubernetes, and Containers.Troubleshoot Hadoop log files and work with. Required Skills & Qualifications : .Scala, Spark, Py...Show moreLast updated: 30+ days ago