Over the past decade, millions of developers have interacted with the Heroku CLI. In those 10 years, the CLI has gone through many changes. We've changed languages several times; redesigned the plugin architecture; and improved test coverage and the test framework. What follows is the story of our team's journey to build and maintain the Heroku CLI from the early days of Heroku to today.
Ruby (CLI v1-v3)
Our original CLI (v1-v3) was written in Ruby and served us well for many years. Ruby is a great, expressive language for building CLIs, however, we started experiencing enough problems that we knew it was time to start thinking about some major changes for the next version.
For example, the v3 CLI performed at about half the speed on Windows as it did on Unix. It was also difficult to keep the Ruby environment for a user's application separate from the one used by the CLI. A user may be working on a legacy Ruby 1.8.7 application with gems specific to Ruby 1.8.7. These must not conflict with the Ruby version and gem versions the CLI uses. For this reason, commands like heroku local (which came later) would have been hard to implement.
However, we liked the plugin framework of the v3 CLI. Plugins provide a way for us to nurse new features, test them first internally and then later in private and public beta. Not only does this allow us to write experimental code that we don't have to ship to all users, but also, since the CLI is an open-source project, we sometimes don't want to expose products we're just getting started on (or that are experimental) in a public repository. A new CLI not only needed to provide a plugin framework like v3, but also it was something we wanted to expand on as well.
Another reason we needed to rewrite the CLI was to move to Heroku's API v3. At the start of this project, we knew that the old API would be deprecated within a few years, so we wanted to kill two birds with one stone by moving to the new API as we rewrote the CLI.
Go/Node (CLI v4)
When we started planning for v4, we originally wanted the entire CLI to be written in Go. An experimental CLI was even done before I started at the company to rebuild the CLI in Go called hk. hk was a major departure from the existing CLI that not only changed all the internals, but changed all the commands and IO as well.
Parity with CLI v3
We couldn't realistically see a major switch to a new CLI that didn't keep at least a very similar command syntax. CLIs are not like web interfaces, and we learned this the hard way. On the web you can move a button around, and users won't have much trouble seeing where it went. Renaming a CLI command is a different matter. This was incredibly disruptive to users. We never want users to go through frustration like that again. Continuing to use existing syntax and output was a major goal of this project and all future changes to the CLI.
While we were changing things, we identified some commands that we felt needed work with their input or output. For example, the output of heroku addons changed significantly using a new table output. We were careful to display deprecation warnings on significant changes, though. This is when we first started using color heavily in the CLI. We disable color when the output is not a tty to avoid any issues with parsing the CLI output. We also added a --json option to many commands to make it easier to script the CLI with jq.
No Runtime Dependency
In v3, ensuring that we had a Ruby binary that didn't conflict with anything on the user's machine on all platforms was a big headache. The way it was done before also did not allow us to update Ruby without installing a new CLI (so we would've been stuck with Ruby 1.9 forever). We wanted to ensure that the new CLI didn't have a runtime dependency so that we could write code in whatever version of Ruby we wanted to without worrying about compatibility.
So Why Not All Go?
You might still be wondering why we didn’t reimplement both the plugins and core in Go (but maintain the same command syntax) to obviate our runtime dependency concerns. As I mentioned, originally we did want to write the CLI in Go as it provided extremely fast single-file binaries with no runtime dependency. However, we had trouble reconciling this with the goal of the plugin interface. At the time, Go provided no support for dynamic libraries and even today this capability is extremely limited. We considered an approach where plugins would be a set of compiled binaries that could be written in any language, but this didn't provide a strong interface to the CLI. It also begged the question of where they would get compiled for all the architectures.
Node.js for Plugins and Improved Plugin Architecture
This was when we started to think about Node as the implementation language for plugins. The goal was for the core CLI (written in Go) to download Node just to run plugins and to keep this Node separate from any Node binary on the machine. This kept the runtime dependency to a minimum.
Additionally, we wanted plugins to be able to have their own dependencies (library not runtime). Ruby made this hard as it's very difficult to have multiple versions of the same gem installed. If we ever wanted to update a gem in v3, we had to go out of our way to fix every plugin in the ecosystem to work with the new version. This made updating any dependencies difficult. It also didn't allow plugins to specify their own dependencies. For example, the heroku-redis plugin needs a redis dependency that the rest of the CLI doesn't need.
We also wanted to improve the plugin integration process. In v3, when we wanted the functionality from a plugin to go into the core of the CLI, it was a manual step that involved moving the commands and code into the core of the CLI and then deprecating the old plugin. It was fraught with errors and we often had issues come up attempting this. Issues were compounded because it usually wasn't done by a CLI engineer. It was done by a member on another team that was usually moving a plugin for their first time.
Ultimately we decided to flip this approach on its head. Rather than figure out an easy way to migrate plugin commands into the core, we made the CLI a collection of core plugins. In other words, a plugin could be developed on its own and installed as a “user plugin”, then when we wanted to deliver it to all users and have it come preinstalled, we simply declared it as a “core plugin”. No modifications to the plugin itself would be required.
This model provided another benefit. The CLI is now a modular set of plugins where each plugin could potentially be maintained by a separate team. The CLI provides an interface that plugins must meet, but outside of that, individual teams can build their own plugins the way they want without impacting the rest of the codebase.
Allowing these kinds of differences in plugins is actually really powerful. It has allowed developers on other teams and other companies to provide us with clever ideas about how to build plugins. We've continually been able to make improvements to the plugin syntax and conventions by allowing other developers the ability to write things differently as long as they implemented the interface.
Slow Migration
One thing I've learned from doing similar migrations on back-end web services is that it's always easier to migrate something bit-by-bit rather than doing a full-scale replacement. The CLI is a huge project with lots of moving parts. Doing a full-scale replacement would have been a 1+ year project and would have involved a painful QA process while we validated the new CLI.
Instead, we decided to migrate each command individually. We started out by writing a small core CLI with just a few lesser-used commands and migrating them from the v3 CLI to the v4 CLI one at a time. Moving slow allowed us to identify issues with specific commands (whether it was an issue with the core of the CLI, the command itself, or using the new API). This minimized effort on our part and user impact by allowing us to quickly jump on issues related to command conversion.
We knew this project would likely take 2 years or longer when we started. This wasn't our only task during this time though, so it enabled us to make continual progress while also working on other things. Over the course of the project, we sometimes spent more time on command conversion, sometimes less. Whatever made sense for us at the time.
The only real drawback with this approach was user confusion. Seeing two versions of the CLI listed when running heroku version was odd and it also wasn't clear where the code lived for the CLI.
We enabled the gradual migration from v3 to v4 by first having v3 download v4, if it did not exist, into a dotfile of the user's home directory. v4 provides a hidden command heroku commands --json that outputs all the information about every command including the help. When v3 starts, it runs this command so that it knows what commands it needs to proxy to v4 as well as what the full combined help is for both v3 and v4.
For 2 years we shipped our v4 Go/Node CLI alongside v3. We converted commands one by one until everything was converted.
Go/Node (CLI v5)
The v5 release of the CLI was more of an incremental change. Users would occasionally see issues with v4 when first running the CLI because it had trouble downloading Node or the core plugins. v5 was a change from downloading Node when the CLI was first run, to including Node in the initial tarball so it would be available when the CLI first loaded. Another change was that instead of running npm install to install the core plugins on first run, we included all the core plugins' Node files with the initial tarball and kept the user plugins separate.
Ruby to Node Command Conversion Complete
In December 2016, we finally finished converting all the commands into the new plugins-based CLI. At this point we modified our installers to no longer include the v3 CLI and the shim that launched the v4 or v5 CLI. Existing users with the CLI already installed as of this time will still be using the v3 CLI because we can't auto-update all parts of the CLI, but new installers will not include v3 and are fully migrated to v5 (or now, v6). If you still have the Ruby CLI installed (you’ll know if you run ‘heroku version’ and see v3.x.x mentioned), you’ll benefit from a slight speed improvement by installing the current version of the CLI to get rid of these old v3 components.
Pure Node (CLI v6)
In April 2017 we released the next big iteration of the CLI, v6. This included a number of advantages with a lighter and generic core written only in Node that could be used as a template for building other CLIs, and a new command syntax.
Leaving Go
While at Heroku we use Go heavily on the server-side with great success, Go did not work out well for us as a CLI language due to a number of issues. OS updates would cause networking issues and cross-compiling would cause issues where linking to native objects did not work. Go is also a relatively low-level language which increased the time to write new functionality. We were also writing very similar, if not exactly the same, code in Go and Node so we could directly compare how difficult it was to write the same functionality in multiple languages.
We had long felt that the CLI should be written in pure Node. In addition to only having one language used and fewer of the issues we had writing the CLI in Go, it also would allow for more communication between plugins and the core. In v4 and v5, the CLI started a new Node process every time it wanted to request something from a plugin or command (which takes a few hundred ms). Writing the CLI entirely in Node would keep everything loaded in a single process. Among other things, this allowed us to design a dynamic autocomplete feature we had long wanted.
cli-engine
Occasionally we would be asked how other people could take advantage of the CLI codebase for their own use — not just to extend the Heroku CLI, but to write entirely new CLIs themselves. Unfortunately the Node/Go CLI was complicated for a few reasons: it had a complex Makefile to build both languages and the plugins, it was designed to work both standalone as well as inside v3, and there was quite a bit of “special” functionality that only worked with Heroku commands. (A good example is the --app flag). We wanted a general solution to allow other potential CLI writers to be able to have custom functionality like this as well.
CLI v6 is built on a platform we call cli-engine. It's not something that is quite ready for public use just yet, but the code is open sourced if you'd like to take a peek and see how it works. Expect to hear more about this soon when we launch examples and documentation around its use.
New Plugin Interface
Due to the changes needed to support much of the new functionality in CLI v6, we knew that we would have to significantly change the way plugins were written. Rather than look at this as a challenge, we considered it an opportunity to make improvements with new JavaScript syntax.
The main change was moving from the old JavaScript object commands into ES2015 (ES6) class-based commands.
// v5
const cli = require('heroku-cli-util')
const co = require('co')
function * run (context, heroku) {
let user = context.flags.user || 'world'
cli.log(`hello ${user}`)
}
module.exports = {
topic: 'hello',
command: 'world',
flags: [
{ name: 'user', hasValue: true, description: 'who to say hello to' }
],
run: co.wrap(cli.command(run))
}
// v6
import {Command, flags} from 'cli-engine-heroku'
export default class HelloCommand extends Command {
static topic = 'hello'
static command = 'world'
static flags = {
user: flags.string({ description: 'who to say hello to' })
}
async run () {
let user = this.flags.user || 'world'
this.out.log(`hello ${user}`)
}
}
async/await
async/await finally landed in Node 7 while we were building CLI v6. We had been anticipating this since we began the project by using co. Switching to async/await is largely a drop-in replacement:
// co
const co = require('co')
let run = co.wrap(function * () {
let apps = yield heroku.get('/apps')
console.dir(apps)
})
// async/await
async function {
let apps = await heroku.get('/apps')
console.dir(apps)
}
The only downside of moving away from co is that it offered some parallelization tricks using arrays of promises or objects of promises. We have to fall back to using Promise.all() now:
// co
let run = co.wrap(function * () {
let apps = yield {
a: heroku.get('/apps/appa'),
b: heroku.get('/apps/appb')
]
console.dir(apps.a)
console.dir(apps.b)
})
// async/await
async function run () {
let apps = await Promise.all([
heroku.get('/apps/appa'),
heroku.get('/apps/appb')
])
console.dir(apps[0])
console.dir(apps[1])
}
It's not a major drawback, but it does make the code slightly more complicated. Not having to use a dependency and the semantic benefits of using async/await far outweigh this drawback.
Flow
The CLI is now written with Flow. This static type checker makes plugin development much easier as it can enable text editors to provide powerful code autocomplete and syntax checking, verifying that the plugin interface is used correctly. It makes plugins more resilient to change by providing interfaces checked with static analysis.
While learning new tools is a challenge when writing code, we've found that with Flow the difficulty was all in writing the core of the CLI and not as much in plugin writing. Writing plugins involves using existing types and functions so often plugins won't have any type definitions at all, where the core has many. This means we as the CLI engineers have done the hard work to include the static analysis, but plugin developers reap the benefits of having their code checked without having to learn much of a new tool if any.
Babel
Class properties and Flow required us to use Babel in order to preprocess the code. Because the process for developing plugins requires you to “link” plugins into the CLI, this allowed us to check if the code had any changes before running the plugin. This means that we can use Babel without requiring a “watch” process to build the code. It happens automatically and there is no need to setup Babel or anything else. All you need is the CLI to develop plugins. (Note that Node must be installed for testing plugins, but it isn't needed to run a plugin in dev mode.)
Improved Testing
Testing is crucial to a large, heavily-used CLI. Making changes in the core of the CLI can have unexpected impact so providing good test coverage and making tests easy to write well is very important. We've seen what common patterns are useful in writing tests and iterated on them to make them concise and simple.
As part of the new plugin interface, we've also done some work to make testing better. There were some gaps in our coverage before where we would have common issues. We worked hard to fill those gaps, ensuring our tests guaranteed commands were properly implementing the plugin interface while keeping the tests as simple as possible to write. Here is what they look like in comparison from v5 of the CLI to v6:
// v5 mocha test: ./test/commands/hello.js
const cli = require('heroku-cli-util')
const expect = require('chai').expect
describe('hello:world', function () {
beforeEach(() => {
cli.mockConsole()
})
it('says hello to a user', function () {
return cmd.run({flags: {user: 'jeff'}})
.then(() => expect(cli.stdout).to.equal('hello jeff!\n'))
})
})
// v6 jest test: ./src/commands/hello.test.js
import Hello from './hello'
describe('hello:world', () => {
it('says hello to a user', async () => {
let {stdout} = await Hello.mock('--user', 'jeff')
expect(stdout).toEqual('hello jeff!\n')
})
})
The syntax is almost identical, but we're using Jest in v6 and Mocha in v5. Jest comes preloaded with a mocking framework and expectation framework so there is much less to configure than with mocha.
The v6 tests also run the flag parser which is why '--user', 'jeff' has to be passed in. This avoids a common issue with writing v5 tests where you could write a test that works but not include the flag on the command definition. Also, if there is any quirk or change with the parser, we'll be able to catch it in the test since it's running the same parser.
What's Next?
With these changes in place, we've built a foundation for the CLI that's already been successful for several projects at Heroku. It empowers teams to quickly build new functionality that is well tested, easy to maintain, and has solid test coverage. In addition, with our CLI Style Guide and common UI components, we're able to deliver a consistent interface.
In the near future, expect to see more work done to build more interactive interfaces that take advantage of what is possible in a CLI. We're also planning on helping others build similar CLIs both through releasing cli-engine as a general purpose CLI framework, but also through guidelines taken from our Style Guide that we feel all CLIs should strive to meet.