We have a python application that polls directories using threads and inotify watchers. We have always run this application in a cloud server provided by Voxel.

We are currently testing a different cloud server provider StormOnDemand and when we ran our application, our load averages were a lot higher then they were when they were running on the Voxel cloud server despite the specs being about the same (Refer below for more details on setup). We have also ensured then when testing the server was not handling any other loads.

I have written a simple test application (test_threads.py) that simulates the issues we are seeing by starting up  threads that loops, sleeping for a user defined time on each loop. It takes 2 parameters, the amount of threads to start and the interval period.

When I run, “python test_threads.py 50 0.1″ for about 10 minutes

Load average results:

StormOnDemand - $ uptime 18:46:22 up  7:29,  6 users,  load average: 4.43, 4.16, 2.93

Voxel - $ uptime 18:48:14 up 9 days, 15:09,  9 users,  load average: 0.51, 0.47, 0.43

The load average on the StormOnDemand server is a lot higher.

Python version:

StormOnDemand – 2.6.5

Voxel – 2.6.5

Server spec:

StormOnDemand – 8 x Intel(R) Xeon(R) CPU E5506 @ 2.13GHz; 16GB RAM; 230GB HDD (Storm Bare Metal servers)

Voxel – 7 x Intel(R) Xeon(R) CPU L5640 @ 2.27GHz; 14GB RAM; 200GB HDD (VoxCloud servers)

OS:

StormOnDemand – Ubuntu 10.04 – 2.6.36-rc8101910 #1 SMP Tue Oct 19 19:18:34 UTC 2010 x86_64 GNU/Linux

Voxel – Ubuntu 10.04 –  2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux

Virtualisation method:

StormOnDemand – Not 100% sure, but I think they use Xen

Voxel – Not sure

It is still unclear as to why there is this difference in the load averages between the two providers. I can only guess that it might have something to with the way the virtualisation is being done.  The StormOnDemand servers come with custom compiled kernels. We have also done some testing on GoGrid and do not seem to have this issue.

One of the reason we were considering the StormOnDemand servers were because we were having I/O bottlenecks during certain times of the days on Voxel. StormOnDemand Bare Metal Servers don’t have those issues because the server is dedicated to you and true enough we were very happy with the I/O performance. Too bad we discovered this other issue and might probably need to drop them off the list.