SoftLayer, Google Top Cloud Benchmark
An expanded cloud-benchmarking test again found public cloud competitors outperforming industry leader Amazon Web Services in terms of performance and price, with the bare metal version of IBM's SoftLayer platform leading the way followed by Google Cloud.
In-memory database vendor VoltDB, an IBM (NYSE: IBM) partner, sponsored the cloud performance benchmark, again using the Yahoo Cloud Serving Benchmark (YCSB). The benchmark recorded similar results last year, but this year's test was expanded to include Google Cloud Platform (NASDAQ: GOOG) and Microsoft Azure (NASDAQ: MSFT).
VoltDB, Bedford, Mass., paid Acme Benchmarking to conduct the expanded cloud performance tests using the "community edition" of VoltDB's in-memory database. The test ran YCSB Workload B (95 percent read, 5 percent update) on the database using six "beefy" servers and clients. The benchmark tested SoftLayer bare metal and virtual instances along with Google, Azure and AWS (NASDAQ:AMZN).
The benchmarking service reported this week that SoftLayer bare metal outperformed its platform-as-a-service competitors, registering more than 1.5 million operations per second and a YCSB price/performance rating for 4.6 billion operations/dollar.
Google Cloud ranked second for both metrics with just over 1 million ops/second and price/performance rating of 3.3 billion ops/dollar.
Source: Acme Benchmarking via VoltDB
AWS faired worse based on the Yahoo benchmark, finishing ahead of only a SoftLayer instance in terms of performance and Azure on price.
Acme noted that in-memory database technology like VoltDB's often created network bottlenecks. While most public cloud vendors claim 10 Gb/s network throughput, the benchmarking service noted that AWS provided the highest throughput (9.455 Gb/s) followed closely by SoftLayer bare metal (9.413 Gb/s). Azure lagged far behind at 4.141 Gb/s.
The benchmarker also noted that comparing cloud services "is no easy task when it comes to provisioning the same 'size' of server." Acme said each cloud provider has it own parameters for server resources. "Some, like Google, keep it extremely simple. Others, like Amazon and IBM, have so many options that you’ll suffer from 'analysis paralysis' when provisioning your first servers."
In a blog post, VoltDB's John Hugg acknowledged "it’s not surprising that bare metal does well." Still, he added, "the difference for this benchmark was significant: 50 percent better absolute performance and 40 percent more operations per dollar."
The other clear winner in the benchmark testing is Google Cloud, which according to the Yahoo benchmark was about 1.8 times faster than AWS in terms of absolute performance and more than two times faster as measured in operations per dollar.
Meanwhile, Amazon, Google and Microsoft have steadily dropped prices for their public cloud services as competition heats up. For example, Google dropped prices on its Cloud Platform virtual machines by 30 percent in May. Google claimed its cloud service is 40 percent cheaper than its competitors "for many workloads."
VoltDB noted that Google offers a simpler pricing model than AWS: Amazon charges by the hour while offering discounts for long-term commitments and pre-payments; Google charges hourly but provides discounts the longer an instance is used.
"The most meaningful take-away [from the cloud performance benchmark] is the understanding that both total performance and price/performance can vary dramatically between clouds, and also between virtualized platforms and bare metal platforms," Hugg noted.
