Ben McCann

Co-founder at Connectifier.
ex-Googler. CMU alum.

Ben McCann on LinkedIn Ben McCann on AngelList Ben McCann on Twitter

How SBT does dependency resolution


SBT uses its own fork of Ivy to do dependency resolution. The code to do this resolution is split between a few classes such as ConvertResolver.

I’ve posted below an excerpt of what SBT does. It uses a chained resolver, which I’ve simplified here to use a single resolver for demonstration purposes.

IvySettings settings = new IvySettings();
ResolveEngine engine = new ResolveEngine(settings, new EventManager(), new SortEngine(settings));
ResolveData data = new ResolveData(engine, new ResolveOptions());
IBiblioResolver resolver = new IBiblioResolver();


ModuleRevisionId mrid = ModuleRevisionId.newInstance("commons-cli", "commons-cli", "1.3.1");
ResolvedModuleRevision resolved = resolver.getDependency(new DefaultDependencyDescriptor(mrid, true), data);

System.out.println("Resolved: " + resolved);

Setting up Mac OSX


Install Homebrew. It will fail to install packages by default due to issues writing to /usr/local. To fix this:

sudo chmod -R g+w /usr/local

Make hidden files visible in the Finder:

defaults write com.apple.finder AppleShowAllFiles TRUE
killall Finder

Change the following settings using the Mac system preferences to make the trackpad usable:

  • Key Repeat – all the way long
  • Delay Until Repeat – all the way short
  • Tracking speed – faster

Install Karabiner, so that you can remap keys.

Go to “Misc & Uninstall” then click “Open private.xml”. Paste the code below. Switch back to the main tab “Change Key” and hit “Reload XML”. Now check “Swap Command and Control unless tabbing through windows”. Also check “Disable all settings while you are using Remote Desktop or VNC”.

 The autogen format is:
   new keys
   original keys

<?xml version="1.0"?>
    <name>Swap Command and Control unless tabbing through windows</name>
      KeyCode::TAB, ModifierFlag::CONTROL_L,
      KeyCode::TAB, ModifierFlag::COMMAND_L
      KeyCode::TAB, ModifierFlag::COMMAND_L,
      KeyCode::TAB, ModifierFlag::CONTROL_L
      KeyCode::BACKQUOTE, ModifierFlag::CONTROL_L,
      KeyCode::BACKQUOTE, ModifierFlag::COMMAND_L
      KeyCode::BACKQUOTE, ModifierFlag::COMMAND_L,
      KeyCode::BACKQUOTE, ModifierFlag::CONTROL_L

OAuth in a command line script


Many APIs today use OAuth. If you want to use an OAuth API from the command line, then what I recommend is starting a web server locally to handle the OAuth callback. Here’s a quick and dirty example of doing that in Python.

#!/usr/bin/env python

from flask import Flask,redirect, request
import json
import logging
import threading
import time
from urlparse import urlparse
import urllib
import urllib2
import webbrowser

CLIENT_ID = 'xxxx'
CLIENT_SECRET = 'yyyyyyyy'

SCOPE = 'repo:read'
AUTH_URL = 'https://quay.io/oauth/authorize'
IMAGES_URL = 'https://quay.io/api/v1/repository/myorg/myrepo/image/'

oauth_access_token = None

app = Flask(__name__)

def oauth_request_token():
  url = 'https://quay.io/oauth/authorize?response_type=token&redirect_uri=' + urllib.quote('http://localhost:7777/oauth_callback') + '&realm=realm&client_id=' + urllib.quote(CLIENT_ID) + '&scope=' + urllib.quote(SCOPE)
  print 'Redirecting to ' + url
  return redirect(url)

def oauth_callback():
  result = """
    getHashParams = function() {
      var hashParams = {};
      var e,
        a = /\+/g,  // Regex for replacing addition symbol with a space
        r = /([^&;=]+)=?([^&;]*)/g,
        d = function (s) { return decodeURIComponent(s.replace(a, " ")); },
        q = window.location.hash.substring(1);

      while (e = r.exec(q))
        hashParams[d(e[1])] = d(e[2]);
      return hashParams;
    ajax = function(url, callback, data) {
      try {
        var x = new(this.XMLHttpRequest || ActiveXObject)('MSXML2.XMLHTTP.3.0');
        x.open(data ? 'POST' : 'GET', url, 1);
        x.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
        x.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
        x.onreadystatechange = function () {
            x.readyState > 3 && callback && callback(x.responseText, x);
      } catch (e) {
        window.console && console.log(e);

    hashParams = getHashParams();
    ajax('/receive_token', function() { window.close(); }, 'access_token=' + hashParams['access_token']);
  return result

@app.route('/receive_token', methods=['POST'])
def receive_token():
  global oauth_access_token
  oauth_access_token = request.form['access_token']
  return '{}'

class ServerThread(threading.Thread):

  def __init__(self):

  def run(self):

if '__main__'==__name__:

  thread = ServerThread()
  thread.daemon = True


  while oauth_access_token is None:

  print 'Retreived auth code ' + oauth_access_token

  opener = urllib2.build_opener()
  opener.addheaders = [('Authorization', 'Bearer ' + oauth_access_token)]
  images = opener.open(IMAGES_URL)
  print images.read()

Building Docker images with SBT


A typical way to setup Jenkins is to connect it to your source repository (e.g. with the Git Plugin), run your tests after each commit, and then build a package for deployment when the tests pass. We’ll use SBT’s sbt-native-packager for this last step, which allows you to package your applications in numerous different formats including zip, deb, rpm, dmg, msi, and docker.

To setup sbt-native-packager to publish you Docker images you need to add sbt-native-packager to your project and specify your Docker repo in your build.sbt. E.g. dockerRepository := Some("quay.io/myorganization"). You now need to setup the credentials to publish to your Docker repository. This typically goes in ~/.dockercfg. You can place the .dockercfg in the Jenkins home directory, which on Ubuntu will by default be located at /var/lib/jenkins/.

The next thing you need to setup is the build step to build the Docker image. This can be a bit confusing because Jenkins has build steps and post-build actions and it’s not completely clear what the difference is. I’ve found that the build step does what we want. You can use the Jenkins SBT Plugin to run your sbt tests with each commit. Now, to build a Docker image you can click “Add build step” followed by “Build using sbt” and in the Actions field enter “docker:publish”

Another thing you may need to deal with is having SBT sub-projects. E.g. let’s assume you have a project named “myproj”, which depends on other libraries. You can set "project myproj" docker:publish in the Jenkins build step so that SBT switches to your myproj project before building the docker image, so that it won’t try to run docker:publish on your subprojects. If you’re using SBT’s aggregation to compile or run the tests of these sub-projects when doing the same for myproj, you’re probably going to want to disable this for publishing the Docker image. You can do this by adding the setting aggregate in Docker := false to your build.sbt:

lazy val myproj = project
    .enablePlugins(DockerPlugin, GitVersioning, PlayJava, SbtWeb)
      aggregate in Docker := false  // when building Docker image, don't build images for sub-projects

Note that you’ll have to handle garbage collection of old Docker images. Docker has this on their roadmap. Until then, I recommend Spotify’s Docker GC.

MongoDB data migration


Here is some benchmarking data regarding transferring data from one machine to another. These benchmarks were run on the AWS i2 instance class.

  • mongodump – 15min / 100GB
  • gzip using pigz – 15min/100GB
  • network transfer – 20min/100GB
  • extract archive – 30min/100GB
  • mongorestore -j 12 – 2hr/100GB

Vision and Culture at Connectifier


There are an infinite number of things to focus on when building a company – building a product, marketing it, selling it, keeping the servers running, finding office space, recruiting a team, fundraising, accounting, payroll, benefits, legal, training. The list goes on forever. With so many things to work on, it’s a necessity to delegate. At the same time, all oars must rowing in the same direction. One of the ways we’ve accomplished that at Connectifier is by having a strong vision and culture.

Connectifier makes talent search engine technology that helps recruiters discover and connect with exceptional job candidates with roughly twice the efficiency of existing methods. The vision for Connectifier is to be able to instantly connect companies and job candidates with perfect matches given just a job description or resume. The few companies that have tried to put people into the right jobs have fallen fall short of what’s possible today for numerous reasons. E.g. there are tons of latent variables which are not taken into consideration. It’s so difficult to tell if a position is a good match for someone’s professional abilities, skills, and interests. Recruiters aren’t equipped with tools that help them understand how the keywords on resumes and job postings relate. The culture of a company is often not easy to discern for a candidate until at least well into the interview process. Our vision for Connectifier is an exciting one and is one we can all relate to having conducted our own job searches. It’s so important to each of us to be professionally fulfilled and to help others find professional fulfillment.

The other half of the equation for us in aligning the organization is Connectifier’s company culture. Culture is more than just a nice office with a ping pong table and cool t-shirts. When folks first visit the Connectifier office and meet everyone, they say that they’re struck by the intellectually curiosity and high caliber of the team. We place a lot of emphasis on hiring the very best. Not every company requires that and many perform better focusing on other combinations of traits. But we’re solving a very difficult technical problem where the team with the highest intellectual horsepower will outperform. We’ve seen this already as we’ve out-performed much larger competitors like Monster and Dice, which have tried to do some of the things that we’re doing with larger teams. But it’s highly unlikely that those companies are modifying the source code of the databases and web browsers used by their products in the same way that we are. Having the team that we do has enabled us to be bold in ways others can’t.

Building the highest quality team isn’t limited to our employees. We have great investors and advisers. Connectifier joined Launchpad LA early on. We thought it was important that Connectifier join the So Cal startup scene, so that folks interested in joining high-growth companies would know who we are. We also have great members of our team at our accounting firm and law firm. And we’ve consulted with numerous firms for projects such as penetration testing (we were recently nominated for an award due to our security efforts). There are so many ways to organize a company – build around functional roles, divisions, geographies, create a cross-functional matrix, a no-hierarchy holocracy. But without the right vision and culture, no organizational structure can be successful.

Learn more about working at Connectifier.

Injecting JUnit tests with Guice using a Rule


GuiceBerry is a pretty helpful library for injecting JUnit tests with Guice. However, it’s not super actively maintained and many of it’s methods and members are private making it difficult to change it’s behavior. Here’s a class, which essentially does what GuiceBerry does in one single class that you can edit yourself.

import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

import com.connectifier.data.mongodb.MongoConnection;
import com.connectifier.data.mongodb.MongoDBConfig;
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.Module;
import com.mongodb.DB;
import com.mongodb.MongoClientOptions;

public class DbRule implements MethodRule  {

  private final Injector injector;
  public DbRule(Class envClass) {
    try {
      this.injector = Guice.createInjector(envClass.newInstance());
    } catch (InstantiationException | IllegalAccessException e) {
      throw new IllegalStateException(e);
  public Statement apply(Statement base, FrameworkMethod method, Object target) {
    return new Statement() {
      public void evaluate() throws Throwable {
        try {
        } finally {

  protected void runAfterTest() {
    DB db = MongoConnectionFactory.createDatabaseConnection(


To use:

  public final DbRule env = new DbRule(DataEnv.class);

IntelliJ Setup


The font rendering on IntelliJ is horrendous and makes you want to gouge your eyes out. This is because is uses Swing. In order to make this not completely horrible, you’ll need to install tuxjdk, which contains series of patches to OpenJDK to enhance user experience with Java-based and Swing-based tools. I also recommend installing the Monokai Sublime Text 3 theme.

If you install the Lombok plugin, then you’ll also need to set: Settings > Build …. > Compiler > Annotation Processing > Enable Annotation Processors

Formatting a Disk on Amazon EC2


The following commands will format and mount your disk on a newly created EC2 machine:

sudo mkfs -t ext4 /dev/xvdb 
sudo mkdir /storage
sudo sed -i '\|^/dev/xvdb| d' /etc/fstab # delete existing entry if it exists
sudo sh -c 'echo "/dev/xvdb /storage ext4 defaults,nobootwait,noatime,nodiratime 0 2" >> /etc/fstab'
sudo mount -a



Here are some things I consider when designing a web API.

Consider using the following response code:

  • 200 – OK
  • 400 – Bad Request
  • 500 – Internal Server Error
  • 401 – Unauthorized (i.e. authentication error)
  • 403 – Forbidden (i.e. not authorized)
  • 404 – Not Found

Version your API
Use limit and offset for pagination
Return JSON responses by default with camel case property names
Append extension to URL to indicate other types (e.g. /person/123.xml)
Host APIs off a subdomain like api.yelp.com
Use OAuth 2.0 for authentication
Pretty print the results by default

Older Posts