Inappropriate Benchmarks
I was pointed at this performance comparison by Joyent regarding their SmartMachines compared to Amazon EC2 images and asked for a comment and how it compares to our machines.
| Server Size | Geekbench Score | Cost per month | Price per Geekbench point per month | EC2 compute units |
|---|---|---|---|---|
| EC2 m1.small (1 core, 1.7GB) | 1615 | $70.68 | $0.043 | 1 |
| EC2 m1.large (2 cores, 7.8GB) | 1887 | $282.72 | $0.150 | 4 |
| EC2 c1.xlarge (8 cores, 7GB) | 2375 | $565.44 | $0.238 | 8 |
| Joyent 1GB Smart Machine | 8461 | $125 | $0.014 | ? |
| Mythic Beasts Servers | ||||
| Apple TV (1 core 1Ghz) | 886 | n/a | n/a | 1 |
| Mac Mini (Core Duo 1.66Ghz) | 1920 | £20.83 | £0.011 | 2 |
| Mac Mini (Core 2 Duo 1.83Ghz) | 2304 | £29.17 | £0.013 | 3 |
| Mac Mini (Core 2 Duo 2.5Ghz) | 3331 | £50 | £0.015 | 4 |
| Mac Mini (Core i5 2.3Ghz) | 5353(estimate from 2.5Ghz model) | £41.67 | £0.008 | 6.5 |
| Mac Mini (Core i5 2.5Ghz) | 5928 | n/a | n/a | 7 |
| Mac Mini (Quad Core i7 2.0Ghz) | 8743 | £80 | £0.009 | 11 |
| Old RAID 1 server (Dual AMD 2.5Ghz) | 2788 | £100 | £0.035 | 4 |
| New RAID 1 server (Quad Intel 1.83Ghz) | 6430 | £135 | £0.021 | 10 |
| VDS256 (KVM virtual machine) | 3005 | £12.50 | £0.0042 | 2 |
The first obvious thing is the cheapest way to buy geekbench points from us is with a KVM virtual machine which offers much better price performance than any hardware platform. This means either we’re very stupid and we’re selling our virtual machines at a loss, or just like Joyent, our virtual machines will use as much CPU as is available on the host at the time – performance isn’t guaranteed and will slow down when other customers use it. On a fully loaded server with every VM being 256MB and running benchmarks at the same time I’d expect that performance figure to divide by about a factor of six for the worst case.
The second thing to notice is the performance comparison with Amazon doesn’t show a substantial increase with number of CPUs unlike the tests on Mythic Beasts servers. That suggests to me that Joyent have managed to run the benchmark in single threaded mode when testing their competitors but multi-threaded mode on their own hardware. I find it exceedingly hard to believe that an 8 core amazon virtual machine has the same CPU performance as a 4 year old introductory level mac mini and is blown away by my desktop mac with only two cores.