Quantcast

Ben McCann

Co-founder at Connectifier.
ex-Googler. CMU alum.

AngelList Twitter LinkedIn Google+

Migrating from MongoDB to TokuMX

05/13/2014

First be sure to install the latest version of TokuMX on the target machines, which is currently 1.4.2.

Also, for all long-running commands, you’ll want to run them in a tmux session. You can create a new tmux session with tmux new, attach to the default session with tmux attach -d, and quit a tmux session with exit after you’re in it.

Run the following commands on the MongoDB secondary with credentials and paths updated to match your environment:

sudo service mongodb stop
sudo mongodump -u adminuser -p 'password' --dbpath /var/lib/mongodb --journal

Connect to the Mongo primary admin DB and run rs.status(). Get last timestamp from secondary and use it in the mongo2toku command below. You can now restart the MongoDB secondary with sudo service mongodb start.

If you want to copy a file from one machine to another with scp, you’ll want to ssh to the first machine using the -A option to enable forwarding of the authentication agent connection. Note that if this is a long running copy command, you’ll want to use tmux, but the -A option will only work with tmux new and not tmux attach -d without jumping through a bunch of extra hoops. So, using ssh -A and tmux new copy the files to the new machine:

scp -r dump remoteip:/media/ephemeral0/mongodump

Now run the following on the Toku primary being sure to use your credentials, data paths, and oplog time:

sudo mongorestore --dbpath /media/ephemeral0/tokumx dump
mongo2toku --from rs/primary:27017,secondary:27017 --ruser adminuser --rpass 'password' --host localhost:27017 --authenticationDatabase admin -u adminuser -p 'password' --ts=9999999999:9

Finding the size of all MongoDB collections

04/28/2014

Here’s a helpful script for finding the size of every table in MongoDB in MB:

var collNames = db.getCollectionNames();
for (var i = 0; i < collNames.length; i++) {   
  var coll = db.getCollection(collNames[i]); 
  var stats = coll.stats(1024 * 1024); 
  print(stats.ns, stats.storageSize);
}

Sound Insulation for Noisy Offices

03/06/2014

I’m a founder at Connectifier, a fast growing tech startup in Newport Beach, CA. We have a great office with many amenities to make it more comfortable such as a kitchen, ping pong table, and sofas. We also have an open floor plan, which is great for keeping everyone in the loop, but less awesome for quiet concentration. As we grow, a space that originally held two people now holds closer to a dozen. At some point we’ll need to find larger office space for a larger team, but there will probably be some point in between now and then where the office is getting uncomfortably full.

In order to plan ahead for our growth, I investigated several office noise solutions. Here’s an idea of what I found.

Ikea Risor Room Divider – $99
ikea-risor

Framery Phone Booths – $7500.00 for a Framery-C and $8500 for a Framery-O unit
framery

Clearsonic MiniMega Isolation Booth – ~$2,750 + $200 shipping
clearsonic-isolation-booth

Vocalbooth.com – ~$6000 depending on model. Shipping included
vocal-booth

Buzzispace – ~$8,000 for Buzzibooth, $2,371 for Buzzicockpit
buzzibooth

Airea Phonebooth with door – ~$6,500 + shipping
airea-phone-booth

Other options include a really good pair of Shure SRH440 headphones and Alpine Earplugs

Debugging tools for java.lang.OutOfMemoryError: Java heap space

02/22/2014

If you have a Java app that’s crashing due to out-of-memory errors then you can create a heap dump by utilizing the following flags:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mydump.hprof

To read the head dump, you’ll need to:

  • Install eclipse memory analyzer
  • Open eclipse with lots of memory: eclipse -vmargs -Xmx6G
  • Open memory analysis perspective: Window > Open Perspective > Other > Memory Analysis

TodoMVC: An Angular vs React Comparison

01/21/2014

Two of the more talked about frameworks today are Google’s AngularJS and Facebook/Instagram’s React, but there are limited comparisons between them. TodoMVC is a project which aims to provide a comparison of JavaScript frameworks by implementing a todo list in the most popular frameworks. I have a little experience with Angular and none with React. I looked at both the Angular TodoMVC app and the React TodoMVC app to try to compare them and was very intimidated that the React one took twice as many lines. In this blog post, I’ll aim to break down the code differences between the AngularJS and React versions and try to decide whether React really is much more verbose and cumbersome to write or if there was some difference in implementation and coding style between the two which had a larger affect.

One thing TodoMVC does is allow the user to type onto the list, and if the user hits enter, it moves the current item down and creates a new empty space for a new item.

Here’s the React version for creating a new item:

handleNewTodoKeyDown: function (event) {
  if (event.which !== ENTER_KEY) {
    return;
  }

  var val = this.refs.newField.getDOMNode().value.trim();

  if (val) {
    var newTodo = {
      id: Utils.uuid(),
      title: val,
      completed: false
    };
    this.setState({todos: this.state.todos.concat([newTodo])});
    this.refs.newField.getDOMNode().value = '';
  }

  return false;
},

Here’s the Angular version:

$scope.addTodo = function () {
  var newTodo = $scope.newTodo.trim();
  if (!newTodo.length) {
    return;
  }

  todos.push({
    title: newTodo,
    completed: false
  });

  $scope.newTodo = '';
};

Much of the extra code in React is because it is listening for a key and then deciding if it was the enter key whereas Angular was simply listening for a submit. This seems likely to be not related to the framework, but merely a difference in implementation in this case.

Let’s look at removing an item from the list. This also takes additional lines in React:

destroy: function (todo) {
  var newTodos = this.state.todos.filter(function (candidate) {
    return candidate.id !== todo.id;
  });

  this.setState({todos: newTodos});
},

And here’s the Angular version:

$scope.removeTodo = function (todo) {
  todos.splice(todos.indexOf(todo), 1);
};

The big difference here is that React creates a new array whereas Angular alters the existing array. I’m not familiar enough with React at this point to know if there’s a requirement to avoid mutating the data structures. However, it’s important to note that the implementation here in React does not work in IE8 because of the use of Array.filter. Throughout the code base, it is a very common theme that much of the extra code results from the React implementation using immutable data structures.

The React also has some extra code for performance improvements. It has included a shouldComponentUpdate method as an example of how performance improvements can be made with React. This is method is not necessary and is used to demonstrate how you could make such an improvement.

/**
 * This is a completely optional performance enhancement that you can implement
 * on any React component. If you were to delete this method the app would still
 * work correctly (and still be very performant!), we just use it as an example
 * of how little code it takes to get an order of magnitude performance improvement.
 */
shouldComponentUpdate: function (nextProps, nextState) {
  return (
    nextProps.todo.id !== this.props.todo.id ||
    nextProps.todo !== this.props.todo ||
    nextProps.editing !== this.props.editing ||
    nextState.editText !== this.state.editText
  );
},

However, this creates extra code besides just this method. The React version also tracks whether the user is in an “editing” state, which is something Angular does not have any code devoted to and which is only ever used in the shouldComponentUpdate function. This means we need a cancel function and about half a dozen other places to track state that are not present in the Angular version.

cancel: function () {
  this.setState({editing: null});
},

Some extra code in React is needed in order to show optional components because it requires making an extra variable which sometimes has its value set:

var footer = null;
if (activeTodoCount || completedCount) {
  footer =
    <TodoFooter
      count={activeTodoCount}
      completedCount={completedCount}
      nowShowing={this.state.nowShowing}
      onClearCompleted={this.clearCompleted}
      />;
}

In Angular, no extra extra lines are required to show an optional attribute and instead you simply use ng-show or ng-if:

<footer id="footer" ng-show="todos.length" ng-cloak>

Similarly, switching between all, completed, and active todos is quite cumbersome in React:

var shownTodos = this.state.todos.filter(function (todo) {
  switch (this.state.nowShowing) {
    case ACTIVE_TODOS:
      return !todo.completed;
    case COMPLETED_TODOS:
      return todo.completed;
    default:
      return true;
  }
}, this);

That took 10 extra lines for something that takes no extra lines in Angular:

<li ng-repeat="todo in todos | filter:statusFilter track by $index" ng-class="{completed: todo.completed, editing: todo == editedTodo}">

A big portion of the extra lines in the React example are also due to coding style. HTML elements which would more typically be placed on a single line have been split between several in the React example:

<input
    id="toggle-all"
    type="checkbox"
    onChange={this.toggleAll}
    checked={activeTodoCount === 0}
    />

Here’s that same code in the Angular example:

<input id="toggle-all" type="checkbox" ng-model="allChecked" ng-click="markAll(allChecked)">

Most of these difference are just implementation or style differences. I think it’s nice to show each example using the idioms of that framework. It may also be nice for TodoMVC to make some basic guidelines so that apps get implemented analogously (e.g. whether to use a submit listener or keypress listener to determine item completion). The one thing that I think would be really annoying is the manner in which React handles if statements in templates vs. the way that this is done in Angular which is much less verbose. I’m also curious about the React implementation’s aversion to mutating state, which seems quite unique to that framework.

Getting started with the Go Language

01/04/2014

Here are a few tricks I picked up when trying to get started with go.

Firstly, don’t install Ubuntu’s version of go as you’ll get an old one. Instead, use gvm to install go1.2 or whatever the latest is.

Secondly, be careful to structure your code in the right directory format. If you do not have the correct directory structure it will cause certain errors such as a “cannot find package” error when running “go test”. E.g. if you create a directory named gocode and want to check code out from github into that directory, you must have it in the src/github.com/organization directory such as:

gocode/src/github.com/fsouza/go-dockerclient

You’ll need to add the directory containing all of your Go code to the GOPATH environment variable. E.g. I did this with:

export GOPATH=$GOPATH:/home/${USER}/src/gocode

You’ll also probably want godep installed since many packages use it for managing dependencies:

go get github.com/tools/godep

Automated Play Framework Testing with Jenkins

11/26/2013

Jenkins automates builds and tests. This post describes setting up Jenkins for the Play 2 Framework.

First off, you need a machine with a good amount of resources. I tried first on a small cloud machine with 2GB of RAM and it was not sufficient, so get a machine with 4GB of RAM.

Next you need to install Java and SBT. Also, git if you use it for source control.

sudo apt-get install openjdk-7-jdk git
wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.0/sbt.deb
sudo dpkg -i sbt

Now that you have Java installed, you can install Jenkins:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

To be able to have Jenkins check the source code out of GitHub or BitBucket, you need to setup an SSH deploy key. First create an SSH key:

sudo su jenkins
ssh-keygen -t rsa
$ cat /var/lib/jenkins/.ssh/id_rsa.pub

Copy the key into GitHub/BitBucket as a deploy key.

Accept the remote host as a machine to trust connections to:

ls-remote -h git@host:org/repo.git HEAD

Now go to the Jenkins web UI. First, Install Jenkins GIT plugin and Jenkins sbt plugin.

Then, setup the system configuration under “Manage” > “Configure System”. Point the sbt plugin to the launcher jar we installed earlier at /usr/share/sbt/bin/sbt-launch.jar.

You’ll also want to set some type of notification of failed builds. Set the “System Admin e-mail address” under “Jenkins Location”, which will be the email address you will receive alerts from and set the SMTP host, username, and password under “E-mail Notification”. I recommend installing the Email Extension Plugin in order to be able to customize the emails that you’ll receive. You can then set project to use Editable Email Notification as a Post-build Action. With the Email Extension Plugin, you’ll need to choose Advanced Settings… and then select Recipients or else your emails won’t go to the recipients you’ve specified. This frustrating option should not exist let alone be unselected by default. To include the build log in the email you can add ${BUILD_LOG,maxLines=10000}. I also suggest adding a trigger so that you get notified both on failed builds and also when the build is fixed.

At this point, you can create a “New Job” selecting “Build a free-style software project”. Enter your git repo location, how often to build, and set it to build with sbt. Enjoy!

Installing Mesos and Marathon on Ubuntu

11/23/2013

Mesos is a distributed task framework. Marathon runs long-running tasks on Mesos. Here’s how you can install the latest versions.

#### Dependencies: Java & Zookeeper
sudo apt-get install -y default-jdk zookeeper-bin zookeeperd
#### Install Mesos
ubuntu_version="12.04"
curl -fL "http://downloads.mesosphere.io/master/ubuntu/${ubuntu_version}/mesos_0.14.2_amd64.deb" --output mesos.deb
sudo dpkg -i mesos.deb
rm mesos.deb
#### Install Marathon
sudo mkdir -p /opt/marathon
sudo curl -fL "http://downloads.mesosphere.io/maven/mesosphere/marathon/0.2.1/marathon-0.2.1-jar-with-dependencies.jar" --output /opt/marathon/marathon.jar
sudo chmod ug+rx /opt/marathon/marathon.jar
sudo curl -fL "http://downloads.mesosphere.io/marathon/marathon.conf" --output /etc/init/marathon.conf

You should be able to start all services at this point either by rebooting or manually:

sudo initctl reload-configuration
sudo start zookeeper
sudo start mesos-master
sudo start mesos-slave
sudo start marathon

If it worked you’ll be able to load the marathon administration page at http://localhost:8080/.

I ran into a problem where I was getting the error messages mesos-master: error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory and java.lang.UnsatisfiedLinkError: no mesos in java.library.path. I had installed OpenJDK via the package openjdk-7-jdk. Installing default-jdk fixes this by symlinking libjvm.so in /usr/lib.

Running ElasticSearch in Production

08/20/2013

ElasticSearch is a very awesome tool with very awful documentation.

One thing you’ll need to be sure to do is update the ES_HEAP_SIZE in your /etc/init.d/elasticsearch to use half your system’s memory in production. The default will give you awful performance.

TOTAL_MEM_K=`grep MemTotal /proc/meminfo | awk '{print $2}' `
HALF_MEM_K=$((TOTAL_MEM_K / 2))
ES_HEAP_SIZE="${HALF_MEM_K}k"
Older Posts