Many years ago Linux founder Linus Torvalds borrowed Sir Isaac Newton’s “standing on the shoulders of giants” phrase when discussing his open source work with Linux. While it’s a nice sentiment — “Hey, I’ve only been able to do great work because of a few others’ great work” — that may have applied to Linux in 1991, it doesn’t do a good job of describing open source in 2020. When someone releases open source code today, after all, they’re drawing on much more than a tiny set of genius developers.
No, as the latest release of Gatsby.js shows, modern open source projects depend on thousands of other projects. Or, as Gatsby founder Kyle Mathews put it in an interview, “We’re standing on the shoulders of thousands of [ordinary] people.”
1000x faster builds? Yes, please!
In the beginning (of the web) was the static site generator. Over time, as Mathews described in his post, organizations turned to database-driven websites to support increasingly dynamic features. As websites grew, static site generation resulted in slow build speeds, even as database-driven tools like WordPress grew in favor with the marketing folks that didn’t want to bother coding in Markdown. Pre-rendering helped the site generators, but not enough to make them the preferred option for large sites.
But that was then; this is now.
Earlier in 2020 Gatsby introduced Gatsby Builds, which made build times 60X faster than standard continuous deployment solutions through a clever combination of distributed computing and sophisticated caching features. Nice. But with the recent release of Incremental Builds, Gatsby is getting builds under 10 seconds for data edits, which can represent a 1000x improvement over existing build solutions.
How? “The biggest thing we’ve done,” said Mathews in an interview, “is created a dependency tracking system that allows us to cheaply figure out what needs updating between builds.” If this sounds like the approach taken by build tools like Bazel, or data processing tools like Apache Spark and Apache Flink, it’s because it is. “Gatsby treats website change events from data/code like other stream processors—we figure out what’s changed and cheaply update it.” Traditional static site generators have long had a batch processing approach to data, but Gatsby’s new approach is real-time stream processing.
It’s a big deal, making the Gatsby approach usable by any website instead of a tiny fraction of the web. It’s also dependent on amazing open source technology like GraphQL, which allows Gatsby to track data dependencies between pages and the data sources so that when some piece of data changes, they can calculate what pages need updated.
So I asked Mathews just how dependent Gatsby is on other open source….
Standing on the shoulders of React, GraphQL, Webpack, Babel...
“We’re heavily dependent on a lot of great projects,” Matthews said. And then he went on:
The phrase could be updated perhaps that instead of standing on the shoulders of giants, we’re standing on the shoulders of thousands of people. Which is one of the cool things about open source—it’s not for geniuses only! Ordinary people can build and/or help maintain bite-sized packages which still solve critical needs.
Some of those “ordinary people” have built extraordinary code upon which Gatsby depends. Given that Gatsby is based on React, React obviously immediately comes to mind. But there’s more. Much more. Gatsby also relies on Webpack and Babel to do the heavy lifting around processing and preparing JavaScript, CSS, and other assets to be deployed. These projects are “phenomenal with years of great work by thousands of engineers,” noted Mathews.
In fact, if you check the Babel NPM package page, you’ll find 136 dependencies, but this omits all the dependencies of other packages Gatsby releases, not to mention the dependencies of Gatsby’s dependencies. All in all, said Mathews, “A typical Gatsby project might use thousands of packages maintained by tens of thousands of engineers all over the world, which is astounding.”
It’s this dependence on open source that “helps [Gatsby] move so quickly and do so many things as a relatively small team.” It’s “the amazingly broad and deep Node.js ecosystem that has almost everything [Gatsby] needs,” he concluded. Nor is he alone. The next time you use Linux, contribute to Kubernetes, or otherwise engage with open source software, remember that in today’s world we are all hopelessly (and hopefully!) dependent on millions of “ordinary developers,” doing extraordinary things.