Who Is Matei Zaharia? Spark, Databricks and AI

Matei Zaharia is the computer scientist who started Apache Spark, co-founded Databricks, and helped build open-source systems including MLflow, Delta Lake, ColBERT, and DSPy. He is currently chief technology officer of Databricks and an associate professor at UC Berkeley, working where distributed systems, data infrastructure, and artificial intelligence meet.

That list explains who he is. It does not quite explain why his career matters. Zaharia has repeatedly worked on the layer beneath a technology boom: first the engine that made large-scale data processing faster and easier, then the tools that turned machine learning experiments into production systems, and now the retrieval and programming infrastructure around large language models. The recurring move is to take a difficult systems problem, turn the research into open-source software, and build an ecosystem around it.

Who is Matei Zaharia?

UC Berkeley’s official biography describes Zaharia as an associate professor of electrical engineering and computer sciences, a Databricks co-founder and CTO, and the person who started Apache Spark during his Berkeley PhD in 2009. His research spans cloud computing, database systems, artificial intelligence, and information retrieval. He earned his PhD in computer science from Berkeley in 2013.

The job titles cross boundaries that usually stay separate. As a professor, Zaharia works on publishable computer-systems research. As Databricks CTO, he helps turn data and AI infrastructure into commercial products. As an open-source developer, he has helped make ideas from both settings available well beyond one company or laboratory.

His best-known creation is Spark, but his wider body of work includes Apache Mesos and Spark Streaming, the machine-learning platform MLflow, the storage layer Delta Lake, the retrieval model ColBERT, and the LLM programming framework DSPy. His current academic homepage emphasizes systems for large-scale AI and analytics, tools for improving AI quality, and ways to connect language models with external data.

Apache Spark: the project that changed data processing

Spark began in 2009 in UC Berkeley’s AMPLab. At the time, Hadoop MapReduce was the defining open-source model for processing large datasets across clusters of machines. It was dependable, but multi-stage jobs repeatedly wrote intermediate results to storage. That made iterative work such as machine learning and interactive analysis awkward and slow.

Zaharia’s central contribution was a more general model for sharing data across stages without giving up fault tolerance. Spark’s original abstraction, the resilient distributed dataset or RDD, let a cluster keep working data in memory and reconstruct lost partitions from their history when a machine failed. The result was not simply “Hadoop, but faster.” It made batch jobs, interactive queries, streaming, machine learning, and graph processing composable on a common engine.

The authors of the definitive Apache Spark: A Unified Engine for Big Data Processing paper framed this unification as the real advantage. Organizations no longer needed a separate engine for every type of workload, with data and code stitched between them. Spark could expose specialized libraries while keeping one execution model underneath.

The distinction between “creator” and “sole author” matters. Zaharia started Spark and led its early design, but the system became important through a large research and open-source community. The unified-engine paper alone has fourteen authors. Spark was open-sourced in 2010, entered the Apache Software Foundation in 2013, and became a top-level Apache project in 2014. The Apache Spark project now lists a broad international group of committers and project-management committee members.

That growth is part of the achievement. Many academic systems prove an idea and then disappear. Spark crossed the gap from dissertation research to durable infrastructure used by data engineers, analysts, and machine-learning teams. It also became the technical foundation around which Databricks was built.

From an open-source project to Databricks

In 2013, Zaharia co-founded Databricks with six other members of the Berkeley data-systems community: Ali Ghodsi, Andy Konwinski, Ion Stoica, Patrick Wendell, Reynold Xin, and Arsalan Tavakoli-Shiraji. The company took shape around a practical question: if Spark was powerful enough to become a standard, could a managed platform make it usable by organizations that did not want to assemble and operate the stack themselves?

That is the commercial logic behind Databricks. Spark remained an Apache project rather than proprietary company code. Databricks sold the managed environment, collaboration, governance, reliability, and enterprise support around data workloads. The strategy widened over time from Spark hosting to the “lakehouse”: a platform intended to combine the flexible storage of a data lake with the management and query behavior associated with a data warehouse.

Zaharia’s role in this story is technical, not just ceremonial. Databricks currently lists him as co-founder and CTO, and his research appears throughout the platform’s architecture. He has worked on Delta Lake, which adds transaction and reliability features to data-lake storage, and MLflow, which manages the increasingly messy lifecycle of machine-learning and AI applications. Databricks also lists him among the people behind projects such as the Dolly open language model and the ColBERT retrieval system.

This matters to the business story because Databricks did not commercialize Spark by closing it. It used open source to create a shared technical standard, then competed on the operational and enterprise layers above that standard. The same pattern now appears throughout AI infrastructure: make the core widely adoptable, then sell the managed system that removes the hard parts. Databricks is one of the major enterprise platforms in the site’s AI inference-provider directory, but its roots are in this earlier data-platform playbook.

MLflow and the move from big data to AI infrastructure

Spark solved how to execute different kinds of data work across a cluster. MLflow addressed a different bottleneck: how to keep track of machine-learning work once an organization had many datasets, models, parameters, experiments, deployment targets, and teams.

Databricks launched MLflow in June 2018. The original paper, Accelerating the Machine Learning Lifecycle with MLflow, lists Zaharia as its first author and identifies three problems: tracking experiments, reproducing results, and moving models into production. Its most important design choice was openness. MLflow was meant to work across libraries, languages, and deployment environments rather than force every team into one model framework.

That design has aged well because the number of tools did not shrink. It exploded. MLflow now covers model registries, evaluation, tracing, observability, prompt management, and gateways for LLM applications and agents. As of July 2026, the MLflow project reports more than 30 million monthly downloads and integrations with more than 100 tools. Those are project-reported adoption figures, but they show how far it moved beyond the original experiment tracker.

The transition from Spark to MLflow also maps the industry’s transition. In the early 2010s, the scarce capability was processing huge datasets reliably. By the late 2010s, it was managing the models trained on that data. In the current generative-AI cycle, the hard problem is evaluating, observing, and governing probabilistic systems that call models and tools. Zaharia’s work has moved up that stack without leaving the systems problems underneath it.

Research beyond Spark: ColBERT, DSPy, and reliable AI systems

Zaharia’s academic work is broader than the Databricks product line. At Stanford, he co-started the DAWN lab in 2016 to work on usable machine-learning infrastructure. His group helped develop DAWNBench, a competition for measuring end-to-end training time and cost that influenced MLPerf, as well as open-source research systems including ColBERT and DSPy.

ColBERT is a retrieval model designed to search large text collections efficiently while retaining richer token-level matching than a single vector per document. DSPy treats prompts and model calls less like handcrafted strings and more like programs whose components can be optimized against examples and metrics. Both sit at the intersection Zaharia’s biography predicts: models are important, but the surrounding retrieval, evaluation, and programming system determines whether they work reliably.

He also appears on current AI-systems research that tests the economics of model behavior. A 2026 preprint co-authored by Zaharia found a “price reversal” effect in which a model with a lower listed token price can cost more to complete a task because it consumes more tokens. The site’s analysis of why cheaper reasoning models can cost more explains the result and its limitations. The question is classic systems thinking applied to AI: measure the whole job, not one attractive number on the rate card.

Zaharia is also a founding advisor to DBOS, the durable-execution company built around research by Michael Stonebraker and others. DBOS uses Postgres to record workflow progress so applications can recover from failures without repeating completed work. That project sits naturally beside his earlier interests in fault tolerance, distributed execution, and dependable infrastructure. For the application to AI agents, see the site’s guide to DBOS and durable agent harnesses.

Why Matei Zaharia won the ACM Prize in Computing

On April 8, 2026, the Association for Computing Machinery named Zaharia the recipient of the 2025 ACM Prize in Computing. The year in the award’s name and the announcement year are different: it is the 2025 prize, announced and presented in 2026. The ACM announcement recognized his development of distributed data systems and infrastructure that enabled large-scale analytics, machine learning, and AI.

The prize fits the shape of the work. Spark is not an end-user application, and MLflow is not a model that produces a dramatic demo. They are infrastructure: the layer that lets thousands of other teams build, repeat, and operate their own systems. Infrastructure is easy to ignore precisely when it works. The ACM award makes the case that changing the substrate can have more reach than creating one application on top of it.

It was not Zaharia’s first major recognition. He received the 2014 ACM Doctoral Dissertation Award for his work on large-scale computing, the U.S. Presidential Early Career Award for Scientists and Engineers in 2019, the ACM SIGMOD Systems Award for Spark in 2022, and the ACM SIGOPS Mark Weiser Award in 2023. Together, the awards span databases, operating systems, and the wider computing field, mirroring how his work crosses the usual category lines.

What his career says about the AI business

The useful business lesson is not “start an open-source project and become a founder.” Most research software never becomes a standard, and most standards do not produce a major company. The more specific lesson is that infrastructure businesses become valuable when they reduce a problem that nearly every participant in a growing market must solve.

Spark arrived when companies had more data than their existing workflows could comfortably process. Databricks reduced the cost of operating the resulting stack. MLflow arrived when companies could train models but struggled to reproduce and deploy them. ColBERT and DSPy address a newer constraint: getting language models to use external information and behave consistently enough for real applications.

The products change, but the economic position stays similar. Zaharia works below the most visible application layer, where a technical abstraction can become a standard used by many competing products. That is a high-leverage place to build. It is also why his name surfaces across data engineering, cloud infrastructure, machine learning, and generative AI rather than belonging to one cycle of the industry.

Bottom line

Matei Zaharia is best known as the creator of Apache Spark and a co-founder of Databricks, but that shorthand leaves out the pattern that makes his work significant. He has spent his career turning difficult data and AI systems problems into open software that other people can build on.

Spark unified large-scale data workloads. Databricks made that infrastructure easier to operate as a platform. MLflow brought order to the machine-learning lifecycle. His newer research applies the same systems discipline to retrieval, LLM programming, evaluation, and reliability. The connecting idea is simple: breakthroughs matter more when the infrastructure lets everyone else use them.

Frequently asked questions

Who is Matei Zaharia?

Matei Zaharia is a computer scientist, the creator of Apache Spark, a co-founder and CTO of Databricks, and an associate professor at UC Berkeley. His research focuses on data systems, cloud computing, artificial intelligence, and information retrieval.

What did Matei Zaharia create?

Zaharia started Apache Spark during his PhD at UC Berkeley in 2009. He also helped develop or lead work on MLflow, Delta Lake, Apache Mesos, Spark Streaming, ColBERT, DSPy, and other open-source data and AI systems.

Does Matei Zaharia still work at Stanford?

No. Zaharia previously served on the Stanford computer science faculty and co-started its DAWN lab. He is now an associate professor of electrical engineering and computer sciences at UC Berkeley.

What is Matei Zaharia's role at Databricks?

Zaharia is a co-founder and the chief technology officer of Databricks. His work connects the company to open-source projects and research in distributed data processing, machine learning, and AI infrastructure.