Our email transfer service YippieMove is essentially software as a service. The customer pays us to run some custom software on fast machines with a lot of bandwidth. We initially picked VMware virtualization technology for our back-end deployment because we desired to isolate individual runs, to simplify maintenance and to make scaling dead easy. VMware was ultimately proven to be the wrong choice for these requirements.

Ever since the launch over a year ago we used VMware Server 1 for instantiating the YippieMove back-end software. For that year performance was not a huge concern because there were many other things we were prioritizing on for YippieMove ’09. Then, towards the end of development we began doing performance work. We switched from a data storage model best described as “a huge pile of files” to a much cleaner sqlite3 design. The reason for this was technical: the email mover process opened so many files at the same time that we’d hit various limits on simultaneously open file descriptors. While running sqlite over NFS posed its own set of challenges, they were not as insurmountable as juggling hundreds of thousands of files in a single folder.

The new sqlite3 system worked great in testing – and then promptly bogged down on the production virtual machines.

Tough CPU week on a server running VMWare

We had heard before that I/O performance and disk performance are the weaknesses of virtualization but we thought we could work around that by putting the job databases on an NFS export from a non virtualized server. Instead the slowness we saw blew our minds. The core servers spent a constant 70% of CPU time with system tasks and despite an uninterrupted 100% CPU usage we could not transfer more than 400KBit/s worth of IMAP traffic per physical machine. This was off by a magnitude from our expected throughput.

Obviously something was wrong. We doubled the amount of memory per server, we quadrupled sqlite’s internal buffers, we turned off sqlite auto-vacuuming, we turned off synchronization, we added more database indexes. These things helped but not enough. We twiddled endlessly with NFS block sizes but that gave nothing. We were confused. Certainly we had expected a performance difference between running our software in a VM compared to running on the metal, but that it could be as much as 10X was a wake-up call.

At this point we realized that no amount of tweaking was likely to get  our new sqlite3 version out of its performance hole. The raw performance just wasn’t there. We suspected at least part of the problem was that we were running FreeBSD guests in VMware. We checked that we were using the right network card driver (yes we were). We checked the OS version – 7.1, yep that one was supposedly the best you could get for VMware. We tuned various sysctl values according to guides we found online. Nothing helped.

We had the ability to switch to a more VM friendly client OS such as Ubuntu and hope it would improve performance. But what if that wouldn’t resolve the situation? That’s when FreeBSD jails came up.

Jails are a sort of lightweight virtualization technique available on the FreeBSD platform. They are like a chroot environment on steroids where not only the file system is isolated out but individual processes are confined to a virtual environment – like a virtual machine without the machine part. The host and the jails use the same hardware but the operating system puts a clever disguise on the hardware resources to make the jail seem like its own isolated system.

Since nobody could think of an argument against using jails we gave them a shot. Jails feature all the things we wanted to get out of VMware virtualization:

• Ease of management: you can pack up a whole jail and duplicate it easily
• Isolation: you can reboot a jail if you have to without affecting the rest of the machine
• Simple scaling: it’s easy to give a new instance an IP and get it going

At the same time jails don’t come with half the memory overhead. And theoretically IO performance should be a lot better since there was no emulated harddrive.

And sure enough, system CPU usage dropped by half. That CPU time was immediately put to good use by our software. And so even that we still ran at 100% CPU usage overall throughput was much higher – up to 2.5MBit/s. Sure there was still space for us to get closer to the theoretical maximum performance but now we were in the right ballpark at least.

More expensive versions of VMware offer process migration and better resource pooling, something we’ll be keen to look into when we grow. It’s very likely our VMware setup had some problems, and perhaps they could have been resolved by using fancier VMware software or porting our software to run in Ubuntu (which would be fairly easy). But why cross the river for water? For our needs today the answer was right in front of us in FreeBSD: jails offer a much more lightweight virtualization solution and in this particular case it was a smash hit performance win.

If you’ve followed us for a while, you might remember our article ‘Selecting an Accounting System’. Not too many things have changed since then. The developers over at Quickbooks are still not competent enough to write an OS independent web-app, and no other major events have occurred. The most interesting thing that have happened since we wrote the article was that SQL-Ledger silently changed license from GPLv2 to SQL-Ledger Open Source License. Although we would rather have seen SQL-Ledger staying GPLv2, we don’t really blame them. The company needs to make some money in order to survive. So, despite the change of license, we still decided to use SQL-Ledger as our accounting system. At least for now.

Enough about that, let’s get started. First time I installed SQL-Ledger I ran into a couple of problems. Even though I spent a fair amount of time doing research on how to set everything up, I still hit a couple of speed bumps. Since I’m quite new to PostgreSQL, some of my problems were related to this. Some other problems were related to the fact that the manuals did not really match what I saw on the screen.

Now, let’s get started. First we start with updating the ports to make sure we’re getting the latest version.

# portsnap fetch; portsnap update


After updating our ports-tree, we need to install PostgrsSQL. When writing this post, the 8.2-series is the latest stable series.

# cd /usr/ports/database/postgres82-server/
# make config


Select the flags you prefer. The only flag I changed from the default was the optimization flag. Not that I know if it makes much of a different, but if you’re compiling it you might as well try to build more optimized binaries.

# make install


Now Postgres is installed. However, in the normal FreeBSDish manner, we need to enable it in rc.conf before we can fire it up.
Edit /etc/rc.conf and add postgresql_enable=”YES”

When you’ve activated Postgres, we first need to initiate the database, and then start the service.

# /usr/local/etc/rc.d/postgresql.sh initdb
# /usr/local/etc/rc.d/postgresql.sh start


Voila, now we have Postgres initiated and up running.

Let’s take a look at the package we’re actually interested in installing: SQL-Ledger.

We’re just going to use a simple ports-installation of SQL-Ledger.

# cd /usr/ports/finance/sql-ledger
# make install


After installing the package, we need to make changes to Apache in order to enable SQL-Ledger. Simply add the following line at the appropriate location in your httpd.conf or ssl.conf:

Include /usr/local/etc/sql-ledger-httpd.conf


To make sure our Apache-file is intact, we run:

# apachectl -t


And if that went well, we run:

# apachectl restart


The only step remaining now is some database-related stuff.
Since I’m not an expert at Postgres, I might not be explaining this in the best way. Anyways, I’m just trying to share my experience, since this was the step where I ran into some speed bumps when first setting up SQL-Ledger. The problems I was facing was that the prompts I received differed quite a bit from the ones found in the various manuals I read before installing.

First, log into the Postgres user

# su - pgsql


When you’re logged in, we need to create a user for SQL-Ledger:

# createuser -d -P sql-ledger
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) y


Note that if you remove the -P, you won’t be prompted for password. However, I personally prefer setting a password here.

Lastly, we need to copy the template.

#createlang plpgsql template1


Hopefully that went without any problems. Now it’s time to surf into SQL-Ledger to make the final configurations. Open your browser and surf into http:///sql-ledger/admin.pl. Log in without any password.

Select the “Database Administration” link.

Use the user ‘sql-ledger’ and the password you assigned when creating it. For ‘connect to,’ use ‘template1.’ When you’re done filling that out, hit ‘create dataset.’

The next screen that will pop up is the Create Dataset-screen. Here you need to set the name of your dataset. Use only lower-case letters. I’d suggest the name ‘sql-ledger’ to keep things from being complicated later on. For ‘encoding,’ select UTF8. As for the ‘chart of accounts,’ it really depends on what business you’re setting up SQL-Ledger for.

You’re done! All you need to do now is to set up the users. Since this is quite straight forward, I’m not going to cover that.

For more information, please visit sql-ledger.org. You might also want to take a look at this ‘unofficial’ manual (the official one costs $190).

Edit 1: Dieter Simader, the founder of SQL-Ledger, e-mailed me to point out that the only non-GPL’d version of SQL-Ledger is the 2.8-series. The 2.6-series used by FreeBSD ports is still under GPL.

           

If you get the following error when you’re trying to ‘make world’ for a jail in FreeBSD,

can’t cd to /usr/src/tools/build/make_check

the problem is simply that you’ve forgotten to run cvsup to download the sources.

I’m posting it here so that in case someone else has this problem they won’t waste 20 minutes wondering what’s wrong and Googling fruitlessly like I did. :)