Popular Posts

Sorry. No data so far.

Ben McCann

Co-founder of Connectifier.
Investor at C3 Ventures.
Google and CMU alum.

Ben McCann on LinkedIn Ben McCann on AngelList Ben McCann on Twitter

Resyncing a very stale MongoDB Replica

08/22/2012

I logged into the primary member of the replica set and ran rs.status() which showed me that the replica was too stale (“error RS102 too stale to catch up”):

{
	"_id" : 4,
	"name" : "55.55.55.55:27017",
	"health" : 1,
	"state" : 3,
	"stateStr" : "RECOVERING",
	"uptime" : 502511,
	"optime" : {
		"t" : 1340841938000,
		"i" : 5028
	},
	"optimeDate" : ISODate("2012-06-28T00:05:38Z"),
	"lastHeartbeat" : ISODate("2012-08-22T22:47:00Z"),
	"pingMs" : 0,
	"errmsg" : "error RS102 too stale to catch up"
},

The MongoDB wiki has some instructions on resyncing a very stale replica. I chose to go the simplest route of doing a full resync. To do this I had to figure out where the data was stored, so I looked in /etc/mongodb.conf to see that the dbpath was set to /var/lib/mongodb. Stopping the node, deleting the data directory, and then restarting the node solved the problem. You’ll need the key file (if you’re using auth) and the data directory to both exist with the proper ownership and permissions to bring the node back up.

Installing Windows 7

07/29/2012

If you need to re-install Windows 7 because you got a new hard drive, you can download a copy of Windows from Microsoft here and then enter your existing product key.  ABR provides one way to get your existing product key.  You can also get a product key for Windows 7 Ultimate and Microsoft Office from Microsoft Bizspark if you run a startup.  If you want to upgrade your existing copy of Windows to a higher edition you can do that by searching for Windows Anytime Upgrade in the start menu.

Google video chat volume on Windows

07/22/2012

I frequently use Google video chat. It was common for the person on the other end to have difficulty hearing me, which I eventually realized was because the microphone level kept being auto-adjusted to very low levels. It turns out that it was Google video chat constantly auto-adjusting the volume level and that the behavior can be turned off by editing a registry setting.

  • Open regedit
  • Navigate to: HKEY_CURRENT_USER\Software\Google\Google Talk Plugin
  • Change the audio-flags value data to 1

The audio-flags registry key is not there on a clean install of Windows, but will show up once you change the Google video chat settings in GMail.  Note that if you ever change the video chat settings in GMail then it will override the registry settings and you will need to set this flag again.

Backing Up MySQL with Percona Xtrabackup

05/25/2012

You can create a MySQL backup with Percona Xtrabackup by running:
$ innobackupex --user=DBUSER --password=DBUSERPASS /path/to/BACKUP-DIR/

You can then restore the backup with:

$ xtrabackup --prepare --datadir=/var/lib/mysql --target-dir=/path/to/BACKUP-DIR/
$ sudo service mysql stop
$ sudo mv /var/lib/mysql ~/dbbackup
$ sudo mv /path/to/BACKUP-DIR /var/lib/mysql
$ sudo chown -R mysql:mysql /var/lib/mysql
$ sudo service mysql start
$ mysql -u root -p
mysql> GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY 'password from /etc/mysql/debian.cnf' WITH GRANT OPTION;

Setting up the RockMongo GUI on Ubuntu

04/17/2012

The easiest way to get started is to install Apache and PHP:

$ sudo apt-get install apache2 php5 php-pear

If you need to edit the Apache ports because you already have another server running on port 80 then edit /etc/apache2/ports.conf.

You’ll need to install the PHP Mongo connector:

sudo pecl install php_mongo

Add “extension=mongo.so” to the “Dynamic Extensions” section of /etc/php5/apache2/php.ini and restart Apache with sudo service apache2 restart.

Download the latest RockMongo and unzip it under /var/www. You should now be able to login with the default username and password of admin/admin.

Brewer’s CAP Theorem Explained

03/24/2012

When dealing with distributed systems, Brewer’s CAP theorem is often brought up when discussing how a system will behave in certain error conditions. The CAP theorem means that you can only have two of: consistency, availability, and partition tolerance.

Here’s what you’ll be giving up for each of the three that you may sacrifice:

  • C: Consistency means that two different machines will return the same responses for the same query.
  • A: Availability means that requests will be answered even if a machine goes down.
  • P: Partition tolerance means that the system continues to function even if there’s a network outage that stops communication.

Web developers rarely want to give up P since that means you could get split brain syndome where the data is out-of-sync between machines. As a web developer, CAP means you must make the choice between having a site that never goes down, but regularly return stale data or a site that never returns stale data, but goes down if there’s a problem.  Thus the real choice is between C and A in this context. A bank website would choose consistency over availability. Getting the balance in someone’s account wrong is worse than having the site be down.  Google chose availability, which is why you never see it go down. The tradeoff is that it may be looking at a slightly stale version of the index when ranking some queries.

Installing Oracle Java JDK on Ubuntu

03/23/2012

Due to licensing restrictions, Ubuntu no longer comes with Oracle’s Java JDK.

The first trick to installing on a headless server is being able to wget the file. I recommend downloading manually and then placing in on a host you control, your Dropbox directory, etc.

You can install it by running:

sudo mkdir -p /opt/java/64
cd /opt/java/64
sudo wget https://www.dropbox.com/s/xcrs0ok7no8tv7x/jdk-7u45-linux-x64.gz
sudo tar -zxvf jdk-7u45-linux-x64.gz
sudo rm jdk-7u45-linux-x64.gz
sudo update-alternatives --install /usr/bin/java java /opt/java/64/jdk1.7.0_45/bin/java 2000 \
    --slave /usr/bin/javac javac /opt/java/64/jdk1.7.0_45/bin/javac \
    --slave /usr/bin/javadoc javadoc /opt/java/64/jdk1.7.0_45/bin/javadoc \
    --slave /usr/bin/javah javah /opt/java/64/jdk1.7.0_45/bin/javah \
    --slave /usr/bin/javap javap /opt/java/64/jdk1.7.0_45/bin/javap

To verify:

$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

Using Python’s Pandas inside IPython Notebook

02/14/2012

IPython is a cool shell to run Python from and Pandas is a Python library for holding tabular data similar to R’s data frame.

To install the software run:

sudo apt-get install python-pip libzmq-dev python-dev g++ libfreetype6-dev libpng12-dev libblas-dev liblapack-dev gfortran cython libhdf5-serial-dev
sudo pip install ipython
sudo pip install tornado
sudo pip install pyzmq
sudo pip install pygments
sudo pip install numpy
sudo pip install matplotlib
sudo pip install scipy
sudo pip install patsy
sudo pip install statsmodels
sudo pip install pandas
sudo pip install pytz
sudo pip install numexpr
sudo pip install tables
sudo pip install jinja2
sudo pip install jsonschema

To run IPython notebook run:

ipython notebook

As an example, you can run the following code in the IPython web notebook to draw a chart of the S&P 500:

%matplotlib inline

import datetime
import matplotlib.pyplot as plt
from pandas.io.data import DataReader

sp500 = DataReader("^GSPC", "yahoo", start=datetime.datetime(2000, 1, 1)) # returns a DataFrame
top = plt.subplot2grid((3,1), (0, 0), rowspan=2)
top.plot(sp500.index, sp500["Adj Close"])
bottom = plt.subplot2grid((3,1), (2,0))
bottom.bar(sp500.index, sp500.Volume)
plt.gcf().set_size_inches(18,8)

Migrating from MySQL to Percona Server

12/11/2011

Percona Server is just MySQL with a few extra options added in by Percona. It’s backwards compatible and based off the same code base. If you’re not familiar with Percona, they are the world’s leading MySQL consultants. The main reason I switched is because Ubuntu uses an old version of MySQL. Ubuntu is about a year behind in packaging MySQL. Something to do with checking the copyright after Oracle got ahold of it. This seemed to be the easiest way to update. A few other reasons follow.

Everyone and their mom says xtraBackup is the way to go for MySQL backups. Even Facebook uses it. xtraBackup is an open source project made by Percona. mysqldump is fine for small projects, but it’s not real scalable when you have any real amount of data. It’s available in the Percona apt repositories.

By default, older version of MySQL use the MyISAM storage engine, which has fallen out of favor. The default in newer MySQL installs is InnoDB. Percona also makes a storage engine called XtraDB, which is backwards compatible with InnoDB and supposedly a bit more performant. MariaDB (MySQL fork maintained by the MySQL creator) uses it as their default as well. Sounds like most people don’t notice a huge difference between XtraDB and InnoDB, but both are much favored over MyISAM which caused lots of problems for people.

Finally, there’s also HandlerSocket, which is a plugin for MySQL. It allows you to do primary key lookups directly to the storage engine bypassing MySQL’s SQL layer. It’s supposed to be 5-10x faster because it doesn’t have to parse the SQL and do table locking. It turns MySQL into a key/value as good as any of the NoSQL solutions. It’s actually much better because you can still run SQL queries on your data, which you can’t do with most of the NoSQL solutions and you get MySQL’s replication etc. which is all very well documented. As long as your DB can fit in RAM on a single machine it makes MySQL much faster. Perhaps even faster and easier to use than even memcached.

To migrate, first create a backup:

mysqldump -uroot -p --all-databases > dump.sql

Then do the upgrade:

gpg --keyserver  hkp://keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
gpg -a --export CD2EFD2A | sudo apt-key add -
sudo emacs /etc/apt/sources.list
Add:
    ## Percona repository
    deb http://repo.percona.com/apt oneiric main
    deb-src http://repo.percona.com/apt oneiric main
sudo apt-get update
sudo apt-get install percona-server-server-5.5
sudo apt-get autoremove

Running Ubuntu on VirtualBox

11/30/2011

I had to figure out a few things to get Ubuntu installed and working well on VirtualBox.

I had to enable virtualization technologies in my BIOS. I have a Lenovo T520 and did this by pressing F1 during startup and then going to Security > Virtualization. If I did not do this then I would receive the error “VT-x features locked or unavailable in MSR” when trying to run with more than 1 CPU or 3584 MB of RAM. Don’t forget to increase the VirtualBox settings to use more RAM and CPUs after updating this.

The default disk is an 8GB dynamically expanding VDI. You may want to consider changing the default from 8GB to something more like 100GB. This is the max size only and will not be used unless needed.

I had to check “Enable 3D Acceleration” under the “Display” settings in order to get Ubuntu Unity to work. If your screen looks super messed up like a bunch of vertical lines then check out this Stackoverflow post.

I had to run “sudo apt-get install dkms” before installing the VirtualBox Guest Additions to get them to work.

To get USB 2.0 devices to pass through (necessary for Android development), you’ll need to download and install the extension pack. Make sure you’re on the latest version of VirtualBox, then right click the icon and choose “Run as administrator”, followed by “Preferences” -> “Extensions”, and choose and install the downloaded extension pack.

Finally, I remapped the host key. By default all kinds of weird things happen when you use the right Ctrl button. This can be fixed by going to File > Preferences… > Input and then setting Host Key to something you never use like Pause.

Newer Posts
Older Posts