Think about the white paper’s title for a second: it sounds like it’s going to test how well the storage performs with SQL Server. Virtual SAN – so we’re testing a storage-intensive workload, right? Wrong, because the very first paragraph of the executive summary says:
“Experiments show the Virtual SAN storage sub-system is never the bottleneck and the workload saturates the host CPU while the I/O latency remains constant.”
Oooo, must be a lot of throughput if the CPU goes to 100%, right? Surely they didn’t handcuff the guests with small amounts of CPU power to keep the storage from being a problem. Let’s check out page 4:
“The entire DVD Store benchmark tools, including the query generator and the database backend, were encapsulated in a single virtual machine, which ran the Microsoft Windows Server 2008 R2 operating system and Microsoft SQL Server 2008. The virtual machine was configured with 4 virtual CPUs (vCPUs), and 4GB of memory.”
They ran the front end app and the database on a single VM with 4GB RAM. This isn’t a load test, it’s a grudge match! And it’s utterly pointless because each host has 128GB memory. They’re just leaving the memory sitting idle. Alright, so what kind of throughput were they able to get with these little toys?
“An aggregated “orders per minute” of 77,206 across 12 DVD Store instances”
Wait – what? I’ve never seen a SQL Server benchmark that just added up all of the transactions that a bunch of completely isolated SQL Servers ran, and called that a single score. You wouldn’t run a real store that way – that’s 12 different databases that can’t see each other.
But let’s play along anyway – 77,206 orders per minute divided by 12 databases equals a whopping 107 transactions per second. If you haven’t done benchmarking before, that number is what we in the performance industry call “low.” Even when combined, 1,287 transactions per second isn’t all that impressive – especially when I bet a single properly configured VM with that exact same hardware could eclipse all 12 of these misconfigured guests. (Not to mention the licensing overhead of the setup in this whitepaper.) To put this in perspective, Anandtech got 1,940 orders per second on a single server back in 2009.
I don’t have anything against VMware’s Virtual SAN product – I bet it’s awesome – but this white paper doesn’t do it justice.