Bitbucket's new npm integration: a tale of three XHRs

I'm a huge fan of Node.js and npm, so I've built a little npm for Bitbucket add-on that adds module metadata, dependency information and download statistics to the npm modules hosted on Bitbucket. What makes the add-on special is that it's built in a slightly peculiar way: it's 100% static HTML & client-side JavaScript. However, it uses a variety of interesting XHR techniques (CORS, window.postMessage, and API proxying) to exhibit some pretty powerful dynamic behaviour.

What does it do?

npm for Bitbucket provides two additional bits of UI to your Bitbucket experience.

First, it adds a statistics panel on the overview page for any repository containing a package.json in the root directory:

npm statistics

This shows the version of the npm module in your repository, the latest version that's been published to the npm registry, and some download statistics. The statistics are just vanity metrics, but the Repo version and Published version are handy if you or your co-workers have the habit of invoking npm publish but forgetting to git push (or vice-versa), which I have been personally guilty of on occasion.

Second, the add-on provides a new Package Info view mode to Bitbucket's source screen:

npm package.json viewer

Package Info renders your package.json file as a table of dependencies with some additional bits of information pulled from the npm registry.

The Version and Latest columns are useful for keeping your project up to date with the latest and greatest npm module releases, and the License column is useful for making sure your project conforms to your dependencies' licensing requirements (e.g. making sure you're not violating the GPL). I also decided to inline the Description field for each dependency to give casual observers a bit more information about each module. With 200K+ modules in the registry named things like gulp, grunt, yo, and q it's not always immediately obvious what each one actually does.

I've put together a short demo video here if you want to see it in action:

The add-on is open source and free. If you use npm in your projects, you should give it a try! It's hosted on Aerobatic, a static website hosting service who happen to have excellent Bitbucket integration.

The rest of this post details how the add-on works under the hood.

Why a static web application?

Actually, "under the hood" is a bit misleading, as the add-on doesn't have a traditional application backend. Instead, the add-on is basically a static website that makes use of several flavors of XMLHttpRequest to source data from Bitbucket and npm. A static web application imposes some interesting design constraints that I'll get to in a minute, but it does provide some seriously attractive benefits:

  • performance & caching: static HTML & JavaScript can be spooled off disk or from cache with little or no additional processing, which results in a very low time-to-first-byte. Being static, it can also be served from a CDN and aggressively cached by clients, leading to a nice fast user experience.
  • scalability: CDNs are very good at serving high request volumes, and since all the actual code is running as JavaScript in the user's browser it scales - and I do not use this term lightly - near infinitely.
  • low cost: CDN hosting is extremely cheap when compared to the cost of cloud PaaS/IaaS solutions, or running your own servers.
  • simpler security: because there is no backend, there is no persistence, which means far fewer concerns about securing user data (this one is particularly attractive for Bitbucket add-ons that work with users' source code).

Of course, there are some big drawbacks to using the static application approach too:

  • persistence is trickier: the flipside of the "simpler security" win above is that your options for persisting state are pretty limited. You can store data per client using the Web Storage API or potentially share state across clients using tools like Firebase, but you don't have the luxury of a traditional RDBMS or NOSQL datastore that you can query at will as you build the page.
  • everything is JavaScript: or something like TypeScript or CoffeeScript that compiles to JavaScript and runs in the browser. I don't personally see this as a drawback, but it may not be every developer's cup of tea.
  • using remote APIs is tough: you can't safely distribute secret credentials with a static application. Also, due to browser same-origin policies, you can't query REST or other web services hosted outside of the domain from which your site is served. Even if they don't require authentication.

These limitations make it hard or impossible to build many applications in a static fashion. However since my npm add-on is basically stateless and I'm comfortable with JavaScript, the only real challenge I had to overcome was that pesky browser same-origin policy. The rest of this post is about how I've used (and possibly abused) the browser XMLHttpRequest API to retrieve data from Bitbucket and npm and synthesize it into the integration above.

XHR #1: fetching package.json from Bitbucket

In order to display the version information and download statistics, we first need to determine the name of the npm module contained in the repository that the user is viewing.

npm statistics xhr #1

To get this data, we need to retrieve the package.json file from the Bitbucket repository. Bitbucket has a convenient REST API for retrieving raw file content. For example, you can retrieve the add-on's own package.json file here:

GET /1.0/repositories/tpettersen/bitbucket-npm/raw/HEAD/package.json

To make the REST request we're going to need a couple of things:

  1. a way to identify the repository that the user is currently viewing; and
  2. a way to authenticate the request to the Bitbucket REST API.

Fortunately the Bitbucket Connect framework provides both of these things. The npm statistics bar is integrated into the Bitbucket UI as a "web panel", which is basically a souped-up <iframe> that targets our add-on server.

npm Bitbucket iframe

The excerpt from our add-on's JSON descriptor for configuring the web panel looks like this:

...
"webPanel": [{
  "key": "npm-stats",
  "url": "/stats.html?repo={repo_uuid}&repo_path={repo_path}",
  "location": "org.bitbucket.repository.overview.informationPanel",
  "conditions": [{
    "condition": "has_file",
    "params": {
      "filename": "package.json"
    }
  }]
}]
...

At render time, the {repo_uuid} and {repo_path} parameters in the url property are substituted with values based on the repository that the user is currently viewing. This allows our add-on to pluck the UUID and repository path from the iframe's query string and use them to identify the repository in our REST request.

Bitbucket Connect also provides a cross-frame JavaScript API that allows us to make authenticated requests to the Bitbucket API. To use it we need to include a special JS file from Bitbucket into our iframe:

<script src='https://bitbucket.org/atlassian-connect/all.js'></script>

all.js provides a rich API with various modules for interacting with Bitbucket. We can use the request module that it provides (which wraps the popular xhr npm module and mimics its API) to retrieve the package.json file from Bitbucket:

AP.require('request', function(request) {
  request({
    url: '/1.0/repositories/{}/' + repoUuid + '/raw/HEAD/package.json',
    responseType: "text/plain",
    success: function(data) { ... },
    error: function(err) { ... }
  });
});

As an aside: the all.js functionality is namespaced under AP, which originally stood for Atlassian Plugins in an earlier version of the Connect framework, but is now kept around for backwards compatibility reasons. The correct nomenclature for an application built with Bitbucket Connect is a Bitbucket Add-on.

Of course, letting the add-on make requests on behalf of any user who browses to that page would be a security problem. So the first time a user views the integration, they'll see a little prompt asking for the permissions we've configured in our add-on's OAuth consumer:

Bitbucket Connect OAuth

The OAuth dialog prompt is actually handled by Bitbucket Connect framework. All we have to do is invoke request and Bitbucket handles the authentication dance for us. Neat!

Assuming the user clicks "Grant access", our success function should be called with the contents of the module's package.json.

{
  "name": "git-guilt",
  "version": "0.1.1",
  "description": "Social blame tool for Git.",
  ...

Then it's simply a matter of parsing the JSON and picking out the name property. Now that we know the npm module's name, we can retrieve some interesting information about it from npm!

XHR #2: fetching download counts from npm

npm has a really nice API for querying package download statistics.

npm statistics xhr #2

From the package name, we can construct a request to get the downloads for the last seven days:

GET api.npmjs.org/downloads/point/last-week/git-guilt

This gives us the (fairly modest) download counts for my git-guilt npm module:

{
  downloads: 19,
  start: "2015-11-26",
  end: "2015-12-02",
  package: "git-guilt"
}

Typically, same-origin restrictions would prevent us from querying this API directly, except that the npm download API returns CORS headers in their responses!

$ curl -I https://api.npmjs.org/downloads/point/last-week/git-guilt
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 03 Dec 2015 01:30:24 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 78
Connection: keep-alive
access-control-allow-origin: *
access-control-max-age: 86400
access-control-allow-methods: GET, HEAD, POST, PUT, DELETE, OPTIONS
access-control-allow-headers: Authorization, Content-Type, If-None-Match
access-control-expose-headers: WWW-Authenticate, Server-Authorization
cache-control: no-cache
...

That beautiful "access-control-allow-origin: *" HTTP header means we can hit the API from client-side JavaScript served from any domain matching *. That is, from anywhere!

I decided to make the request using superagent, a nice HTTP request library that works in both Node.js and the browser. It make what looks like an ordinary AJAX request, but with an absolute URL targeting the npm API:

superagent
  .get("https://api.npmjs.org/downloads/point/last-week/" + packageName)
  .set('Accept', 'application/json')
  .end(function (err, res) {
    if (err) {
      opts.error(err);
    } else {
      opts.success(res.body.downloads);
    }
  });

Well that was easy! The next step is to retrieve some version metadata from the npm registry.

XHR #3: fetching metadata from the npm registry

Unfortunately, npm doesn't have a formal API for retrieving package metadata.

npm statistics xhr #3

However, you can mimic the npm command line tool and just hit their CouchDB web app directly:

GET registry.npmjs.org/git-guilt

It returns a nice JSON representation containing all published versions of your package.json:

{
  _id: "git-guilt",
  _rev: "23-f9a088dedc3892e8747eab722bce7364",
  name: "git-guilt",
  description: "Social blame tool for Git",
  dist-tags: {
    latest: "0.1.1"
  },
  versions: {
    0.0.1: {},
    0.0.2: {},
    0.0.3: {},
    0.0.4: {},
    0.0.5: {},
    0.0.6: {
      name: "git-guilt",
      version: "0.0.6",
      description: "Social blame tool for Git",
...

Unfortunately, there's one small problem:

$ curl -I https://registry.npmjs.org/git-guilt
HTTP/1.1 200 OK
server: CouchDB/1.5.0 (Erlang OTP/R16B03)
etag: "32BZGZJ1ER99XCW4KLPN51Y7V"
Content-Type: application/json
Cache-Control: max-age=60
Content-Length: 12760
Accept-Ranges: bytes
Date: Thu, 03 Dec 2015 03:57:49 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-sjc3120-SJC
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1449115069.445930,VS0,VE46
Vary: Accept

No CORS headers! This means that, despite the fact that the API is public and unauthenticated, we can't hit it from client-side JavaScript.

A quick aside: developers, if you maintain an anonymously accessible API, please consider adding CORS headers for the sake of us static application developers! :)

However, with a bit of googling I did stumble across an intriguing web app living at npm-registry-cors-proxy.herokuapp.com. Sure enough:

$ curl -I https://npm-registry-cors-proxy.herokuapp.com/git-guilt
HTTP/1.1 200 OK
Server: Cowboy
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8
Content-Length: 16697
Date: Thu, 03 Dec 2015 04:06:41 GMT
Via: 1.1 vegur

Huzzah! The magical "Access-Control-Allow-Origin: *" HTTP header. Unfortunately I couldn't figure out who owned the proxy (tweet me if you know) so it seemed unwise to build a production application on top of it. However, it did give me an idea.

Aerobatic, the hosting solution I'm using for npm for Bitbucket, allows you to register proxy routes as part of your static web app. These allow static app developers to hit APIs that either require authentication or don't support CORS as if they were regular AJAX end points running on their own domains. Configuring it is simple, you just add a _virtualApp section to your package.json mapping local routes to remote API paths:

  ...
  "_virtualApp": {
    "router": [{
      "module": "express-request-proxy",
      "path": "/registry/:package",
      "method": "get",
      "options": {
        "url": "https://registry.npmjs.org/:package"
      }
    },{
      "module": "webpage"
    }]
  },
  ...

This maps any AJAX requests made by our add-on to /registry/{package_name} through to the npm registry at https://registry.npmjs.org/{package_name}. So now we can retrieve the package metadata using superagent again to make a simple AJAX request:

superagent
  .get("/registry/" + packageName)
  .set('Accept', 'application/json')
  .end(function (err, res) {
    if (err) {
      opts.error(err);
    } else {
      opts.success(res.body);
    }
  });

And there we have it! With the ability to query data from Bitbucket, the npm stats API and the npm registry itself, building the rest of the integration was just a simple matter of HTML and CSS. You can check out the full source code if you like, and as always, contributions are very welcome.

Conclusion

With a variety of XHR based techniques we've built a static application that exhibits some powerful dynamic behaviour, costs next to nothing to host, scales to very large numbers of users, and is extremely secure by virtue of running entirely in the user's browser! Static applications certainly aren't always the answer, but next time you're building a new app or feature, consider whether part of it could be built using static techniques. And please, add CORS headers to your public APIs!

If you'd like to try out npm for Bitbucket, you can use this handy install button:

Install npm for Bitbucket

If you have any feedback on npm for Bitbucket, questions about Bitbucket or add-on development, or just want to chat about static web apps, drop me a line on Twitter! I'm @kannonboy.

If you've enjoyed this post, you might also enjoy my talk about integrating with npm from our Mad Science Node.js meetup in November.