What the web platform can learn from Node.js

The Darwinism of small modules

November 18th 2015 Will Binns-Smith in Node.js, Bitbucket

As web developers, we've all come to appreciate the layer of sanity that libraries like jQuery slather atop the inconsistencies and awkwardness of what the platform provides. What was once constructing an XMLHTTPRequest object over a handful of lines becomes a single-line invocation of $.ajax, and interaction with the DOM through jQuery rarely involves platform-specific hacks and workarounds.

Anyone who's used languages like Python1 or Java knows what it's like to have a batteries-included standard library available. In the browser, jQuery was (and to many, still is) that standard library, along with toolkit libraries like underscore. Like most standard libraries, everyone's code has a tendency to become deeply coupled to it. And with so much code tied to these APIs, like the platform itself, it becomes increasingly difficult to move forward without breaking the web. And forget about including multiple versions of these libraries for compatibility; they're often larger than 30KB and write to the global window object by default.

I once accepted this problem as an unfortunate truth of developing software. That all changed, for me at least, with the Node.js ecosystem.

The rise of small modules and mass composability

Node, a JavaScript runtime built upon Chrome's V8 engine, barely has much of a standard library. Instead, most of what you'd find in the standard libraries of other ecosystems is found in "userland", or outside of the core platform, which in the case of Node is the npm registry.

On npm, small modules have become the norm, with users like substack and Sindre Sorhus publishing upwards of 685 and 760 modules respectively, each following the UNIX way of doing one thing and doing it well. Modules such as array-union, which returns the union of two arrays, and svg-create-element, which provides a great API for creating SVGs in the DOM, seem so absurdly obvious and small that they ought to ship with the language or platform.

Sindre even has a module named negative-zero, which simply returns whether or not a value is -0, and is effectively one line long. While it may appear extreme to create a package out of such simple functionality instead of inlining the function into one's own code, a problem can be solved once and then later expressed by what one means, rather than repeating implementation details. Sindre speaks to this in an incredible reply to an "Ask me anything" question regarding small modules. I highly recommend giving it a read.

Even the incredibly popular Express web framework is distributed as the kernel of a web application. In contrast to large frameworks like Ruby on Rails or Django which come bundled with templating, ORMs, csrf protection, and other features, Express only ships with middleware to host static files. Instead, the application developer is free to use whichever implementation of these features they like, and compose them together to create their application. Many middleware packages come and go as ideas are improved upon and others dismissed. It's the Node way.

As a result, small modules — and apps composed of them — tend to have extremely large dependency graphs (Bitbucket's frontend, for example, contains more than one thousand JavaScript modules — many internal and many from npm).

A directed acyclic graph representing a small module's dependencies.

The greatest quality of these modules is that they aren't bound tightly to their platform: they're free from being frozen in time by the curse of the standard library, and with the help of Semantic Versioning can freely iterate on their API without breaking all of their dependent users.

A story of streams

Node includes streams, an abstraction over asynchronously flowing data, and is most often used for connecting and transforming sources of I/O. Node's initial implementation of streams, which shipped in Node 0.4, left a lot to be desired and led to potentially lost data if used incorrectly. In response, Node 0.10 shipped with a revision of streams (also known as Streams2). But Streams2 wasn't a simple iteration to the streams pattern, and actually went through a number of changes before shipping in Node 0.10. When it did ship, it was compatible with apps running in Node 0.8 without being backported into a release of Node 0.8.

How could this be possible? Well, Streams2 was born from the readable-stream module, which began life as an independent module by Isaac Schlueter in July of 2012 long before it shipped with Node 0.10 in March of 2013. There, it went through iteration as its API and functionality matured, and the Node community found itself with a far superior implementation of streams.

Even today, the latest implementation of readable-stream is maintained as a userland module in npm for use in Node versions dating back to 0.8. Many users use the userland module rather than the bundled one altogether to ensure compatibility with the ecosystem.

A series of unfortunate APIs

In contrast to this, existing APIs in JavaScript and the web platform are frozen solid. Iterations to the JavaScript language, which cannot have any backwards-incompatible changes lest they risk breaking every existing site, must have entirely additive changes. After the performance woes of the Mutation Events API, for example, Mutation Observers were introduced to smooth things over. WebSQL was deprecated in favor of the lower-level, but more awkward IndexedDB. Application Cache is being phased out in favor of the lower-level and more versatile ServiceWorker. Object.observe, an ES2016 proposal which allowed observation of an object's properties, was recently withdrawn after React's preference for unidirectional data-flow and immutability took the web by storm.

Confused by all of this yet? We shouldn't be expected to get things right the first time, but we need a platform that lets us get it wrong first, and then iterate towards perfection.

Evolving the platform

Proponents of the Extensible Web Manifesto want to make the web resilient to userland trial and error the same way that Node is. Their mission is to have the platform provide as many low-level building blocks as possible, so that libraries outside of the browser can experiment freely and avoid the costly and lengthy process that formal standardization must follow. One such low-level primitive is the set of APIs for web components, which provide developers with the means to create dynamic custom elements and attributes through JavaScript, complete with encapsulation2. It's no secret that libraries, big and small, have implemented dialog functionality, but now the iteration of the API can be handled in userland, rather than needing to get it right the first time in the platform. With the low-level primitives in place, userland is free to explore higher-level abstractions in the form of small modules. In other words, no more AppCache.

Thankfully, we've already seen the benefits of this approach. No longer do we have to be stuck with features like AppCache that are implemented without much real-world usage. Instead, we have a standardized implementation of the Promises A+ Specification, which was proven in npm modules like Q and shipped as part of ES2015 earlier this year. The WHATWG is also working on a specification for streams, which is heavily influenced by the evolution of streams that developed from Node.

Like the rest of the platform, these new standards are admittedly difficult to change, but luckily their ideas and APIs were iterated upon in userland. Thanks Node!

[1] For a great example of the curse of the standard library in Python, check out the saga of urllib, urllib2, and the excellent requests library, which would rather not be included in the standard library

[2] Be sure to check out Skate from Atlassian's own Trey Shugart and the Polymer Project from Google for excellent abstractions over the low-level web components API.


Have thoughts on small modules? Want to sing their praises or have second thoughts? Reply below, ping me at @wbinnssmith on Twitter, or contact me via wbinnssmith.com.

Many thanks to my fellow Atlassians who reviewed and provided feedback for this series: Tim Pettersen, Chris Darroch, Travis Smith, Marcin Szczepanski, Trey Shugart, and Jon Mooring.

And an equally special shout-out to my small module heroes, Matt DesLauriers and Sindre Sorhus for their review, input, and overall inspiration for this series.