Sr. Engineering Manager - Serverful Compute

P-932

The last 10 years was the decade of the cloud. The next 10 years will be the decade of Data and AI. This market will be over $3T by 2030. In a market this big, the winner(s) will win on the strength of a rich product portfolio combined with strong platform play.

Databricks is uniquely positioned in this market as both a powerful platform built on open source and open standards, and boasting of a rich product portfolio spanning verticals such as SQL, model lifecycle, data engineering, analytics, and observability for data and AI. Databricks has always been a visionary in this space, embracing AI before it was cool. Our platform story is the result of years of investment in a general purpose platform that is driving the hockey stick growth in our user community and customers.

The Compute Fabric team in Databricks powers this rocketship. Compute Fabric is organized into serverful and serverless . Serverful Compute is Databricks original Compute offering, still powering most of Databricks revenue at EOY 2024. Its the most sophisticated, powerful, and scalable platform for orchestrating data-centric compute on the planet.

What We Do

The charter of the Serverful Compute team, aka Clusters Compute Team, or Clusters Team, is to create the most reliable, cost-competitive, and easy to use compute platform for the modern data cloud, which just works . Towards that goal, we are scaling our platform for the next order of magnitude of growth, while raising the bar significantly on reliability, TCO, capabilities, and performance. We work on complex problems at the forefront of distributed systems, cloud computing, ML infrastructure, and compute.

In serverful, the end user is typically doing two things : (1) using a Databricks product (e.g. jobs, notebooks), and (2) managing compute (clusters) for the product. In serverless, compute is completely transparent to the user and is automatically managed through the user-facing product (e.g. SQL). Although this distinction is orthogonal to hosting model, for historic and business reasons we offer serverful in customer account (customer-hosted) and serverless in Databricks account (self-hosted). Serverful has thus gotten coupled with the customer-hosted offering.

Serverful Compute Team in BLR will fully own the following areas from the above table by end of FY26 :

• Compute Plane (aka Data Plane).
• Stability

The impact you will have :

What we look for :

About Databricks

Databricks is the data and AI company. More than 10,000 organizations worldwide including Comcast, Cond Nast, Grammarly, and over 50 PERCENT of the Fortune 500 rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter , LinkedIn , and Facebook .

Benefits

At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region, please visit mybenefitsnow.com / databricks .

Our Commitment to Diversity and Inclusion

At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics.

Compliance

If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.

Locations : Bengaluru, India