If you are buying cloud service from any of the Big Three, you may not be getting what you think you’re paying for. For example, performance is variable, with smaller machines sometimes beating larger ones. And no vendor is always best, but one tends to be fastest and another most expensive.

Those are some of the findings from an InfoWorld Test Center comparison of cloud speeds, summarized by the headline, “A diverse set of real-world Java benchmarks shows Google is fastest, Azure is slowest, and Amazon is priciest.”

The talented Peter Wayner performed the testing using the open source DaCapo benchmarks, a suite of 14 common Java programs conveniently bundled into a single JAR for easy use. The programs offer varying processing and memory load and no machine configuration will be best for all of them.

InfoWorld found some surprises. For example: “Adding more CPUs often isn’t worth the cost. While Windows Azure’s eight-CPU machine was often dramatically faster than its one-CPU machine, it was rarely ever eight times faster — disappointing given that it costs eight times as much,” InfoWorld reported.

The report, which prints out as 12 pages, includes a huge amount of interesting details, mostly about cost vs. performance and apps running slower than expected.

The bottom line

 .

For performance: “Google was the fastest overall. A Google machine had the fastest time in 13 of the 14 tests. A Windows Azure machine had the fastest time in only one of the benchmarks. Amazon was never the fastest.”

For cost: “Google was also the cheapest overall, though Windows Azure was close behind. Executing the DaCapo suite on the trio of machines cost 3.78 cents on Google, 3.8 cents on Windows Azure, and 5 cents on Amazon. A Google machine was the cheapest option in eight of the 14 tests. A Windows Azure instance was cheapest in five tests. An Amazon machine was the cheapest in only one of the tests.”

• Do your own testing?

Reading InfoWorld’s test results may make you want to go out and run your own tests. For some companies, this may be fairly easy but others may not find a way to create accurate comparisons using their apps on multiple configurations.

That’s why benchmarks exist, so you can easily run the same software in different places and get comparable results. Those results, however, are specific only to the benchmarks used. That’s another way of saying “your mileage will vary.”

Having spent some years around the InfoWorld Test Center during its glory days in the 1990’s, I can tell you that good testing is not easy to do yourself. Yes, you can do the same benchmarks Peter Wayner did and you’d probably get something like the results he got. Provided you worked to create apples-to-apples comparisons as he did.

• Using the results

Based on InfoWorld’s findings, I’d skip benchmarking and do a simple head-to-head comparison over a period of time. If easily possible, I’d run the same code on a similar configuration from all three services and see which saved me the most money, made me happiest or (ideally) both.

Depending on your total cloud spending, it may make sense to run something with each vendor all the time – making sure each knew that you constantly test the other two.

This is a bit like during the 1970’s, when what are now known as CIO’s, used to have an Amdahl coffee mug visible when an IBM rep showed up on a sales call. Amdahl was the closest thing to competition Big Blue had at the time. Supposedly, presence of “the mug” won lower prices from IBM.

• Pick up and move?

Nothing in the results would make me want to move from one vendor’s cloud to another’s. Though the supposed ease of cloud migration would make this threateningly possible if I were unhappy with my incumbent vendor. But, if you spend enough the price differences might make a move worthwhile.

What I hope will happen is Microsoft will raise its game, necessary to get Azure out of second/slowest place in the testing and Amazon will match Google’s pricing, which seems possible. Amazon matching Google’s performance could prove much more difficult given the technology involved. We’ll see.

photo: yanyanyanyanyan via photopin cc