Talent.com
Sr. Software Development Engineer - Scala, Spark
Sr. Software Development Engineer - Scala, SparkIBM • bangalore, India
No longer accepting applications
Sr. Software Development Engineer - Scala, Spark

Sr. Software Development Engineer - Scala, Spark

IBM • bangalore, India
15 hours ago
Job description

What you’ll do (responsibilities) :

Responsibilities include :

  • Design, develop, and enhance components related to database server internals, including storage, indexing, query execution, and transaction processing.
  • Implement and refine query planners, optimizers, and execution engines with a focus on performance and scalability.
  • Analyze and optimize complex SQL and distributed queries, ensuring minimal latency and resource efficiency.
  • Contribute to Apache Spark or related open-source ecosystems, including performance improvements, extensions, and debugging.
  • Build and maintain large-scale distributed data processing pipelines.
  • Perform deep query analysis, profiling, troubleshooting, and root cause investigation for performance bottlenecks.
  • Design cloud-native microservices using Kubernetes and Docker.
  • Troubleshoot and debug production issues using advanced Linux debugging tools, logs, and metrics.
  • Collaborate with cross-functional engineering teams to define technical strategies and architectural improvements.
  • Mentor junior developers, conduct code reviews, and contribute to development best practices.

What you’ll need :

  • 9+ years of professional software development experience.
  • Strong expertise in :

  • Database server internals
  • Query planners & optimizers
  • Query execution frameworks
  • Hands-on experience with query optimization and SQL performance tuning.
  • Proven contributions to Spark open-source, or strong experience working with Spark internals.
  • Strong proficiency in Scala and / or Java, with deep understanding of concurrency, memory management, and functional programming concepts.
  • Solid experience with Kubernetes (K8s) and Docker for container orchestration and deployment.
  • Strong Linux fundamentals and hands-on experience with :
  • Linux profiling tools (perf, strace, lsof, etc.)
  • Kernel-level debugging (preferred)
  • Deep knowledge of distributed system design (networking, partitioning, replication, fault tolerance).
  • Experience with CI / CD pipelines and version control (Git).
  • What's nice to have (preferred qualifications)

    Icing on the cake…

  • Experience contributing to large-scale open-source projects (Apache Spark, Presto, Trino, etc.).
  • Familiarity with columnar formats (Parquet, ORC) and vectorized execution engines.
  • Experience with cloud platforms (AWS, Azure, GCP).
  • Knowledge of JVM performance tuning and GC optimization.
  • Exposure to big data query engines or OLAP systems.
  • Create a job alert for this search

    Sr Software Engineer • bangalore, India

    Related jobs
    Sr. Software Development Engineer

    Sr. Software Development Engineer

    Amazon • Bengaluru, Karnataka, India
    This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.DESCRIPTION : Join...Show more
    Last updated: 30+ days ago • Promoted
    Sr. Advanced Software Engineer

    Sr. Advanced Software Engineer

    Honeywell • Bengaluru, Karnataka, India
    Honeywell is charging into the Industrial IoT revolution with the establishment of Honeywell Connected Enterprise (HCE), building on our heritage of invention and deep, on-the-ground industry exper...Show more
    Last updated: 15 days ago • Promoted
    Software Development Engineer III - Full Stack Technologies

    Software Development Engineer III - Full Stack Technologies

    Confidential Company • Bangalore
    RESPONSIBILITIES : - Drive discussions to create / improve product, process and technology.Building abstractions and contracts with separation of concerns for a larger scope.Rapi...Show more
    Last updated: 30+ days ago • Promoted
    Software Development Engineer I

    Software Development Engineer I

    Joveo • Bengaluru, Karnataka, India
    About Joveo As the global leader in AI-powered, high-performance recruitment marketing, Joveo is transforming talent attraction and recruitment media buying for the world’s largest employers, staf...Show more
    Last updated: 18 days ago • Promoted
    Sr. Software Development Engineer, Alexa Smart Properties

    Sr. Software Development Engineer, Alexa Smart Properties

    Confidential • Bengaluru / Bangalore, India
    Are you looking for an opportunity to build an LLM based enterprise-grade, highly available, large scale, software solution Does it excite you to find patterns and build generic, composable softwar...Show more
    Last updated: 24 days ago • Promoted
    Sr. Lead Software Development Eng.

    Sr. Lead Software Development Eng.

    Confidential • Bengaluru / Bangalore, India
    WHAT YOU DO AT AMD CHANGES EVERYTHING.At AMD, our mission is to build great products that accelerate next-generation computing experiences—from AI and data centers, to PCs, gaming and embedded syst...Show more
    Last updated: 27 days ago • Promoted
    2576634 Software Development Engineer III, Aurora

    2576634 Software Development Engineer III, Aurora

    half the sky • Bengaluru, KA, India
    Are you interested in building hyper-scale database services in the cloud? Do you want to revolutionize the way databases are built for the cloud ? Do you want to have direct and immediate impact o...Show more
    Last updated: 30+ days ago • Promoted
    Software Development Engineer 3

    Software Development Engineer 3

    Prosperr.io • Bengaluru, Karnataka, India
    About the Company : At Prosperr.As a pioneering fintech SaaS platform, we empower employees to maximize their in-hand salaries and enable businesses to design smarter, compliant, and efficient tax s...Show more
    Last updated: 21 days ago • Promoted
    Sr. Software Development Engineer

    Sr. Software Development Engineer

    Confidential • Bengaluru / Bangalore, India
    Serving thousands of enterprise customers around the world including 45% of Fortune 500 companies, Zscaler (NASDAQ : ZS) was founded in 2007 with a mission to make the cloud a safe place to do busin...Show more
    Last updated: 27 days ago • Promoted
    Sr. Software Development Engineer, Device OS

    Sr. Software Development Engineer, Device OS

    Confidential • Bengaluru / Bangalore, India
    As a Software Development Engineer in the Amazon Device OS organization, you will own the definition and implementation of components used to build applications on millions of devices.We are buildi...Show more
    Last updated: 30+ days ago • Promoted
    Spark / Scala Engineer

    Spark / Scala Engineer

    E-Solutions • Bengaluru, India
    Job Role : Spark / Scala Engineer.Good hands-on experience of working on the Hadoop Eco-system ( HDFS, Hive, Spark ).Good understanding of the Hadoop File Formats. Good Expertise on REST API , SQL , GI...Show more
    Last updated: 3 hours ago • Promoted • New!
    Sr Software Development Engineer

    Sr Software Development Engineer

    Confidential • Bengaluru / Bangalore, India
    Designs / develops software solutions and business process solutions based on business needs and requirements using specialized domain knowledge and developed business expertise.Focuses on software q...Show more
    Last updated: 19 days ago • Promoted
    Sr. Software Development Engineer, Digital Acceleration

    Sr. Software Development Engineer, Digital Acceleration

    Amazon • Bengaluru, Karnataka, India
    This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.DESCRIPTION : Do y...Show more
    Last updated: 30+ days ago • Promoted
    Sr. Software Development Engineer, Device OS

    Sr. Software Development Engineer, Device OS

    Amazon • Bengaluru, Karnataka, India
    This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.DESCRIPTION : As a...Show more
    Last updated: 30+ days ago • Promoted
    Sr. Software Development Engineer, Digital Acceleration

    Sr. Software Development Engineer, Digital Acceleration

    Confidential • Bengaluru / Bangalore, India
    Do you thrive on solving complex problems with simple solutions Would you like to drive innovation for Amazon's Digital Businesses influenced by Gen AI solutions Join Amazon Digital Acceleration (D...Show more
    Last updated: 17 days ago • Promoted
    Sr. Software Development Engineer - Scala, Spark

    Sr. Software Development Engineer - Scala, Spark

    IBM • Bengaluru, India
    What you’ll do (responsibilities) : .Design, develop, and enhance components related to database server internals, including storage, indexing, query execution, and transaction processing.Implement a...Show more
    Last updated: 3 hours ago • Promoted • New!
    Software Development Engineer - III

    Software Development Engineer - III

    Capillary Technologies • Bengaluru, India
    Innovative and self-motivated with a passion to develop complex and scalable applications.Strong coding and design skills with prior experience in developing scalable & high availability applicatio...Show more
    Last updated: 5 days ago • Promoted
    Sr. Staff Software Development Engineer

    Sr. Staff Software Development Engineer

    Confidential • Bengaluru / Bangalore, India
    Serving thousands of enterprise customers around the world including 45% of Fortune 500 companies, Zscaler (NASDAQ : ZS) was founded in 2007 with a mission to make the cloud a safe place to do busin...Show more
    Last updated: 17 days ago • Promoted