Linus Torvalds is a giant of open source. And the story of how he created Linux as a student (Just For Fun, as the title of his book would have it), and then ruled (somewhat grumpily) over the community that developed the Linux kernel, epitomizes the hacker spirit of the early open source movement.
But open source has undeniably changed: It’s been embraced by the corporate world, and that’s transformed both businesses and the open source community. Today’s most influential open source projects are more likely to emerge from collaborative processes within big companies than from lone gunmen. But because they’re open source, the projects end up helping the whole community—and their creators often move on to reap the benefits of their newly raised profile.
Let’s take a look at the innovators behind some of today’s most important open source projects.
Data streaming may sound dull and technical, like plumbing for the digital age. But as companies need to ingest and process more and more data from more and more sources to compete, the infrastructure necessary to send those firehoses to the right places is ever more crucial, and traditional solutions often became bottlenecks.
In the late 2000s and early 2010s, LinkedIn was one of the first companies to start bumping up against this problem. Three of LinkedIn’s engineers—Jay Kreps (center), Neha Narkhede (right), and Jun Rao (left), who had worked on a team that among other things created the Who’s Viewed My Profile Feature—worked on a messaging system project that eventually became Kafka, named by Kreps after one of his favorite writers.
Released as open source, Apache Kafka has become an important part of the plumbing for a vast array of companies and projects. Kreps, Narkhede, and Rao, meanwhile, moved on to found Confluent, a company that offers commercial and cloud-based editions of Kafka.
Redis is a protean marvel: a NoSQL in-memory data structure store that can persist on disk and function as a database, a cache, or a message broker. It started, the way a lot of open source projects do, with an annoyance. Specifically, Salvatore Sanfilippo, who was working at an Italian startup trying to develop a real-time weblog analysis tool, was frustrated by the inadequacies of various databases he was trying out for the job.
Sanfilippo open sourced Redis not long after launching it, the project quickly grew, and soon big players like GitHub were using it. Sanfilippo’s career then followed a not uncommon trajectory: While staying in his native Sicily, he was hired by VMware to continue to work on the Redis project. Large companies serving as patrons of open source have become an important part of the professional ecosystem. Sanfilippo has also written a novel, coming soon!
Kubernetes is a container orchestration system that has become crucial infrastructure for big and small companies looking to move to a container-based architecture. It was born inside Google, and Joe Beda (left), one of its creators, tells the story of how it came together, as he worked together with Brendan Burns (not shown) and Craig McLuckie (right), building on concepts from the company’s earlier “Borg” cluster manager. Kubernetes’ original name was “Project 7,” a reference to the Borg character Seven of Nine from Star Trek: Voyager.
Probably the most important part of Beda’s story is how he, Burns, and McLuckie all pushed hard to open source their embryonic project—because, he said, they “really wanted people to use this” and wanted to “bring the best ideas from Borg out into the open.” The rest is open source history. Beda and McLuckie both now work on cloud tech at VMware, while Burns is a VP at Microsoft’s Azure division.
Apache Spark, the data processing platform that displaced MapReduce as the top dog of big data, is an outlier among the projects we’re discussing in that it emerged from academia. Its creator, Matei Zaharia, gold medaled at the International Collegiate Programming Contest, then went on to get his PhD at UC Berkeley, where the foundations for Spark were laid as part of his research. Spark was open source almost from the start, as befits its non-corporate background.
As Spark took the big data world by storm, Zaharia kept a foot in both business and academic worlds. He founded Databricks, a company that provides cloud service offerings around Spark and other data engineering, data science, and machine learning projects, and he’s a professor at Stanford, where his research group works closely with the open source community.
Ansible is a powerful automation platform and a key part of the infrastructure as code movement. YAML-based Ansible Playbooks define server configurations that Ansible automatically pushes out via SSH to the target systems. Along with Puppet, Chef, and a few other tools, Ansible played a large role in bringing infrastructure management into the DevOps world.
Michael DeHaan is Ansible’s creator, and his career trajectory offers another path that an open source project and creator might follow. DeHaan worked at Red Hat’s R&D group in the mid 2000s, then moved on to a few different companies where he worked with clients, all in the realm of various kinds of infrastructure provisioning—and, classically, growing increasingly frustrated with the tools available.
DeHaan launched Ansible as an open source project, then built a company around it, which was soon acquired by his old employers at Red Hat. The Linux and open source giant was a natural fit.
Ryan Dahl can boast of a rare achievement in the open source world: a pair of huge hits. His first influential project was Node.js, a runtime that brought server-side JavaScript execution mainstream, then took JavaScript into more and more non-browser niches. Node and Dahl followed a not unfamiliar open source trajectory: Once the project hit a critical mass, both project and creator moved to a for-profit company that could incubate it. Dahl stepped aside after a few years, and Node.js was shifted to a foundation.
Then, for his next act, Dahl presented Deno ... another JavaScript runtime (and anagram of Node). Provocatively, Dahl unveiled his new project at a talk called “10 Things I Regret About Node.js.” The upshot: Deno aims to be more secure and more productive for programmers than Node.js. Deno has seen rapid growth, and Dahl has founded a new company to serve as its home (though it remains open source).
Spencer Kimball (left) and Peter Mattis (center) have the distinction of straddling two eras of open source history. While undergraduate roommates at UC Berkeley in the mid 1990s, they developed the GNU Image Manipulation Program and GTK, two foundational pieces of software that helped kickstart the viability of open source, although they didn’t maintain a presence in the projects’ ongoing development.
Instead, Kimball and Mattis spent the 2000s and early 2010s in corporate Silicon Valley, starting out as early Google employees and then dabbling in various social media adjacent projects. But eventually they joined force with fellow Googler Ben Darnell (right) to roll out CockroachDB, a SQL database so named because it’s built on a distributed architecture that makes it extremely hard to kill.
Drawing on Kimball, Mattis, and Darnell’s earlier Google experience, CockroachDB goes head to head with Google Cloud Spanner. But as an open source project, it’s freely available for anyone to use.
So far we’ve discussed big open source projects launched from Silicon Valley giants and startups and research labs and dorm rooms. Let’s visit a final unassuming origin of open source innovation: a local newspaper in a college town in Kansas. Simon Willison (shown) and Adrian Holovaty (next slide) worked for the Lawrence Journal-World in the early 2000s when they were in their early twenties, and, as Willison put it, needed to be able to “create reasonably complex database driven web applications in as little time as possible.” Django was the Python-based web application framework Willison and Holovaty designed to do it.
Continues...
Not only did Django power the newspaper but it became wildly popular for Python web development, still holding an important place today. While Willison (previous slide) and Holovaty (shown) haven’t produced another big hit like some of the folks on this list, they both have interesting and productive careers—Holovaty has worked for the Washington Post and other publications on tech journalism projects, while Willison teamed up with his wife to work on an events platform that was acquired by Eventbrite. It’s surely a source of satisfaction to them—like everyone else on this list—that so many have benefitted from their open source work.