Ben McCann

Co-founder at Connectifier.
ex-Googler. CMU alum.

Ben McCann on LinkedIn Ben McCann on AngelList Ben McCann on Twitter

How SBT does dependency resolution


SBT uses its own fork of Ivy to do dependency resolution. The code to do this resolution is split between a few classes such as ConvertResolver.

I’ve posted below an excerpt of what SBT does. It uses a chained resolver, which I’ve simplified here to use a single resolver for demonstration purposes.

IvySettings settings = new IvySettings();
ResolveEngine engine = new ResolveEngine(settings, new EventManager(), new SortEngine(settings));
ResolveData data = new ResolveData(engine, new ResolveOptions());
IBiblioResolver resolver = new IBiblioResolver();


ModuleRevisionId mrid = ModuleRevisionId.newInstance("commons-cli", "commons-cli", "1.3.1");
ResolvedModuleRevision resolved = resolver.getDependency(new DefaultDependencyDescriptor(mrid, true), data);

System.out.println("Resolved: " + resolved);

Setting up Mac OSX


Install Homebrew. It will fail to install packages by default due to issues writing to /usr/local. To fix this:

sudo chmod -R g+w /usr/local

Make hidden files visible in the Finder:

defaults write com.apple.finder AppleShowAllFiles TRUE
killall Finder

Change the following settings using the Mac system preferences to make the trackpad usable:

  • Key Repeat – all the way long
  • Delay Until Repeat – all the way short
  • Tracking speed – faster

Install Karabiner, so that you can remap keys.

Go to “Misc & Uninstall” then click “Open private.xml”. Paste the code below. Switch back to the main tab “Change Key” and hit “Reload XML”. Now check “Swap Command and Control unless tabbing through windows”. Also check “Disable all settings while you are using Remote Desktop or VNC”.

 The autogen format is:
   new keys
   original keys

<?xml version="1.0"?>
    <name>Swap Command and Control unless tabbing through windows</name>
      KeyCode::TAB, ModifierFlag::CONTROL_L,
      KeyCode::TAB, ModifierFlag::COMMAND_L
      KeyCode::TAB, ModifierFlag::COMMAND_L,
      KeyCode::TAB, ModifierFlag::CONTROL_L
      KeyCode::BACKQUOTE, ModifierFlag::CONTROL_L,
      KeyCode::BACKQUOTE, ModifierFlag::COMMAND_L
      KeyCode::BACKQUOTE, ModifierFlag::COMMAND_L,
      KeyCode::BACKQUOTE, ModifierFlag::CONTROL_L

OAuth in a command line script


Many APIs today use OAuth. If you want to use an OAuth API from the command line, then what I recommend is starting a web server locally to handle the OAuth callback. Here’s a quick and dirty example of doing that in Python.

#!/usr/bin/env python

from flask import Flask,redirect, request
import json
import logging
import threading
import time
from urlparse import urlparse
import urllib
import urllib2
import webbrowser

CLIENT_ID = 'xxxx'
CLIENT_SECRET = 'yyyyyyyy'

SCOPE = 'repo:read'
AUTH_URL = 'https://quay.io/oauth/authorize'
IMAGES_URL = 'https://quay.io/api/v1/repository/myorg/myrepo/image/'

oauth_access_token = None

app = Flask(__name__)

def oauth_request_token():
  url = 'https://quay.io/oauth/authorize?response_type=token&redirect_uri=' + urllib.quote('http://localhost:7777/oauth_callback') + '&realm=realm&client_id=' + urllib.quote(CLIENT_ID) + '&scope=' + urllib.quote(SCOPE)
  print 'Redirecting to ' + url
  return redirect(url)

def oauth_callback():
  result = """
    getHashParams = function() {
      var hashParams = {};
      var e,
        a = /\+/g,  // Regex for replacing addition symbol with a space
        r = /([^&;=]+)=?([^&;]*)/g,
        d = function (s) { return decodeURIComponent(s.replace(a, " ")); },
        q = window.location.hash.substring(1);

      while (e = r.exec(q))
        hashParams[d(e[1])] = d(e[2]);
      return hashParams;
    ajax = function(url, callback, data) {
      try {
        var x = new(this.XMLHttpRequest || ActiveXObject)('MSXML2.XMLHTTP.3.0');
        x.open(data ? 'POST' : 'GET', url, 1);
        x.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
        x.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
        x.onreadystatechange = function () {
            x.readyState > 3 && callback && callback(x.responseText, x);
      } catch (e) {
        window.console && console.log(e);

    hashParams = getHashParams();
    ajax('/receive_token', function() { window.close(); }, 'access_token=' + hashParams['access_token']);
  return result

@app.route('/receive_token', methods=['POST'])
def receive_token():
  global oauth_access_token
  oauth_access_token = request.form['access_token']
  return '{}'

class ServerThread(threading.Thread):

  def __init__(self):

  def run(self):

if '__main__'==__name__:

  thread = ServerThread()
  thread.daemon = True


  while oauth_access_token is None:

  print 'Retreived auth code ' + oauth_access_token

  opener = urllib2.build_opener()
  opener.addheaders = [('Authorization', 'Bearer ' + oauth_access_token)]
  images = opener.open(IMAGES_URL)
  print images.read()

Building Docker images with SBT


A typical way to setup Jenkins is to connect it to your source repository (e.g. with the Git Plugin), run your tests after each commit, and then build a package for deployment when the tests pass. We’ll use SBT’s sbt-native-packager for this last step, which allows you to package your applications in numerous different formats including zip, deb, rpm, dmg, msi, and docker.

To setup sbt-native-packager to publish you Docker images you need to add sbt-native-packager to your project and specify your Docker repo in your build.sbt. E.g. dockerRepository := Some("quay.io/myorganization"). You now need to setup the credentials to publish to your Docker repository. This typically goes in ~/.dockercfg. You can place the .dockercfg in the Jenkins home directory, which on Ubuntu will by default be located at /var/lib/jenkins/.

The next thing you need to setup is the build step to build the Docker image. This can be a bit confusing because Jenkins has build steps and post-build actions and it’s not completely clear what the difference is. I’ve found that the build step does what we want. You can use the Jenkins SBT Plugin to run your sbt tests with each commit. Now, to build a Docker image you can click “Add build step” followed by “Build using sbt” and in the Actions field enter “docker:publish”

Another thing you may need to deal with is having SBT sub-projects. E.g. let’s assume you have a project named “myproj”, which depends on other libraries. You can set "project myproj" docker:publish in the Jenkins build step so that SBT switches to your myproj project before building the docker image, so that it won’t try to run docker:publish on your subprojects. If you’re using SBT’s aggregation to compile or run the tests of these sub-projects when doing the same for myproj, you’re probably going to want to disable this for publishing the Docker image. You can do this by adding the setting aggregate in Docker := false to your build.sbt:

lazy val myproj = project
    .enablePlugins(DockerPlugin, GitVersioning, PlayJava, SbtWeb)
      aggregate in Docker := false  // when building Docker image, don't build images for sub-projects

Note that you’ll have to handle garbage collection of old Docker images. Docker has this on their roadmap. Until then, I recommend Spotify’s Docker GC.

MongoDB data migration


Here is some benchmarking data regarding transferring data from one machine to another. These benchmarks were run on the AWS i2 instance class.

  • mongodump – 15min / 100GB
  • gzip using pigz – 15min/100GB
  • network transfer – 20min/100GB
  • extract archive – 30min/100GB
  • mongorestore -j 12 – 2hr/100GB

Injecting JUnit tests with Guice using a Rule


GuiceBerry is a pretty helpful library for injecting JUnit tests with Guice. However, it’s not super actively maintained and many of it’s methods and members are private making it difficult to change it’s behavior. Here’s a class, which essentially does what GuiceBerry does in one single class that you can edit yourself.

import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

import com.connectifier.data.mongodb.MongoConnection;
import com.connectifier.data.mongodb.MongoDBConfig;
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.Module;
import com.mongodb.DB;
import com.mongodb.MongoClientOptions;

public class DbRule implements MethodRule  {

  private final Injector injector;
  public DbRule(Class envClass) {
    try {
      this.injector = Guice.createInjector(envClass.newInstance());
    } catch (InstantiationException | IllegalAccessException e) {
      throw new IllegalStateException(e);
  public Statement apply(Statement base, FrameworkMethod method, Object target) {
    return new Statement() {
      public void evaluate() throws Throwable {
        try {
        } finally {

  protected void runAfterTest() {
    DB db = MongoConnectionFactory.createDatabaseConnection(


To use:

  public final DbRule env = new DbRule(DataEnv.class);

IntelliJ Setup


The font rendering on IntelliJ is horrendous and makes you want to gouge your eyes out. This is because is uses Swing. In order to make this not completely horrible, you’ll need to install tuxjdk, which contains series of patches to OpenJDK to enhance user experience with Java-based and Swing-based tools. I also recommend installing the Monokai Sublime Text 3 theme.

If you install the Lombok plugin, then you’ll also need to set: Settings > Build …. > Compiler > Annotation Processing > Enable Annotation Processors

Formatting a Disk on Amazon EC2


The following commands will format and mount your disk on a newly created EC2 machine:

sudo mkfs -t ext4 /dev/xvdb 
sudo mkdir /storage
sudo sed -i '\|^/dev/xvdb| d' /etc/fstab # delete existing entry if it exists
sudo sh -c 'echo "/dev/xvdb /storage ext4 defaults,nobootwait,noatime,nodiratime 0 2" >> /etc/fstab'
sudo mount -a



Here are some things I consider when designing a web API.

Consider using the following response code:

  • 200 – OK
  • 400 – Bad Request
  • 500 – Internal Server Error
  • 401 – Unauthorized (i.e. authentication error)
  • 403 – Forbidden (i.e. not authorized)
  • 404 – Not Found

Version your API
Use limit and offset for pagination
Return JSON responses by default with camel case property names
Append extension to URL to indicate other types (e.g. /person/123.xml)
Host APIs off a subdomain like api.yelp.com
Use OAuth 2.0 for authentication
Pretty print the results by default

Running Marathon and Mesos with Panamax


Technology Overview

Panamax is a new tool that allows you to manage multiple Docker containers and to link them together. In this post, I’ll talk about creating a Panamax template which will allow you to run Marathon and Mesos in Docker containers. Mesos is a cluster manager, which allows you to run many jobs in a fault-tolerant manner. It can scale to thousands of machines and is well suited for running large jobs like Hadoop or running many different services in a microservice architecture. Marathon is a Mesos framework which provides a UI for scheduling jobs on Mesos. Marathon and Mesos both rely on a distributed application called Zookeeper to store configuration information. Panamax is very helpful in wiring together Marathon, Mesos masters, Mesos slaves, and Zookeeper instances.

Running Panamax

Panamax has some great installation instructions. Locally it depends on Vagrant and VirtualBox to create a CoreOS instance on which to run the Docker containers. I got a bit hung up on running it for the time and the VM wouldn’t start. I debugged this problem by opening the VirtualBox UI and running the VM manually. It turns out that I didn’t have virtualization extensions turned on in my BIOS on this computer yet, so I got the error message “VT-x is disabled in the BIOS.” Most computers have VT-x disabled by default as a security precaution, so if you’ve never turned VT-x on, you’ll have to do so.

Creating a Panamax application

The first step of creating a Panamax application is to find Docker containers to use. This part was trickier than I imagined given that this was my first time using Docker. I first tried to use thefactory/marathon Docker image. However, it turned out that the version they published did not match what was in the Docker description because the DockerHub automated build didn’t build one of their commits, and so Marathon wouldn’t actually run. I filed a bug on this issue and it has since been fixed, so it would be a great image to try again. It’s always good to review the docker images you use. E.g. I ended up using the redjack/mesos-master and saw that it was doing some of its software installation over insecure HTTP, so I sent them a commit that they merged to change it to HTTPS. I also saw that it was using Ubuntu 14.04, but using the Mesos install for 12.04, so I also sent a pull request to have it use the correct Mesos install and upgrade it to Mesos 0.20 at the same time.

One problem with the way I set things up was that the initial download of all the images takes a long time. I used images from a few different sources and they all used slightly different base OS images. They’re quite big nearing 1GB each and need to be downloaded. If they used the same base file then it’d only have to be downloaded once. Now that the issue the thefactory images is fixed, it’d probably be nice to try to give those images another shot in order to speed up usage of the Panamax template.

One of the things you’ll have to figure out is how to pass configuration information to your docker containers. I passed some command line flags directly in the Docker run command. Another great strategy is to run services with wrapper script that reads config from environment variables as is done in this script in the CenturyLink MySQL Docker image.

Running the template

You can find my template by searching for “Marathon on Mesos and Zookeeper” from the Panamax Contest Templates. It has some great instructions for getting started, so I won’t rehash them here. After the various images are up and running and you’ve set the required settings, you should be able to see a Marathon screen like the following:


Things to watch out for

Panamax seems to struggle with being disconnected from the internet while downloading an image, so be sure you have time to wait for your downloads to complete. As long as you’re plugged in and not going anywhere you shouldn’t have any problems. The other issue I had was a hard time saving my Panamax template because it wasn’t dealing well with GitHub accounts with lots of repos. That issue has already been fixed, which is evidence of how quickly this project is moving. I also wasn’t sure if it was possible to test local Docker images as part of a Panamax application, so it seems like you’ll want to publish any images you plan to use.

You’ll also have to be careful to create good documentation for your Panamax templates and to use templates with good docs. I saw that someone else posted a Mesos template, so I tried it out to see how it would compare to mine, but was unable to run it. I thought for awhile that it was broken and wouldn’t work, but I think now that it’s probably a case of missing documentation instead. However, those missing docs could cause hours of debugging. Panamax is really easy to use and has a nice UI, but there’s still technology under the covers that has to be configured correctly when using it.

Future improvements

The thing I’d most like to see change is for Marathon to offer better authentication and authorization support. I’ve submitted a pull request to the Chaos Web Framework, which was created for use by Marathon and Chronos, to make this possible.

Marathon on Panamax template interest

This blog post was mentioned in the CenturyLink Labs newsletter. I tweeted about this template and it was favorited or retweeted by several folks including Marc Averitt (Managing Director of Okapi Venture Capital) and AllThingsMesos. Ross Jimenez (Director of Software at CenturyLink Labs) tweeted as well and was retweeted and favorited by several folks including Florian Leibert (Founder of Mesosphere), the Panamax Project, and Lucas Carlson (CIO of CenturyLink Labs and CEO of AppFog). Grégory Horion said this was his favorite Panamax template (besides the Locomotive CMS template he created 🙂 and this Tweet was favorited by the engineering team at Twitter. Seen this template mentioned other places? Let me know!

What’s next for Panamax

Panamax is a very cool project. One of the biggest things that the Panamax team is working on is support for multiple hosts. Things will really start to get fun then. It will be very cool to see this deployed in production. I can see web hosts really loving something like this since it’s great for running software like WordPress where there are multiple components that need to linked together such as PHP, Apache, and MySQL.

Older Posts