When it comes to features, security and support from public cloud service providers (CSPs), there’s often a direct correlation between price and what is delivered. That isn’t the case with performance, according to research from Cloud Spectator.

Cloud Spectator’s new study, “2016 Top 10 Cloud Vendor Benchmark, European Report,” found the best-value CSPs offer virtualized resources at the lowest prices.

When looking at the monthly costs and the performance of public infrastructure-as-a-service (IaaS) providers in Europe, 1&1 Internet Inc.’s Cloud Server has the best price-performance value, according to Cloud Spectator’s CloudSpecs Score. 1&1 had the least-expensive packaged pricing compared with other providers at the time of the study, and it had high virtual memory (VM) performance. 1&1 was also top on the price-performance list in Cloud Spectator’s North American cloud vendor benchmark report.

Close behind 1&1 is UpCloud Ltd., which had the second least-expensive pricing and the highest block storage performance.

Amazon Web Services (AWS), which scored higher in VM (defined as vCPU and memory) performance than CloudSigma Holding AG, Cloudwatt, Google Compute Engine and ProfitBricks Inc., does not rank within the top 10 due to lower relative disk performance and higher monthly costs.

Higher costs also prevented Microsoft Azure and IBM SoftLayer from making it onto the list.

“Azure’s new VM series, the D2, shows very high performance among the hyperscale providers. AWS demonstrated average performance on VMs, and SoftLayer ranked last among the three,” said Kenny Li, CEO of Cloud Spectator. “Factoring in price did lower their ranking, however, because the smaller providers in the IaaS space are more price-competitive.”

Li said he was not surprised to see the smaller IaaS providers ranked higher.

“We see many smaller players find an advantage by offering high-performance infrastructure at a very competitive price,” he said. “The hyperscale providers offer the size and ​additional ​service​s​ for handling massive customer volume, but the ​volume also comes with additional performance considerations, such as throttling to provide a standard user experience across the entire infrastructure​, which may result in lower performance on cloud servers​.”

Top performers: vCPU and memory

When evaluating just performance, Cloud Spectator found that in certain scenarios, the top 10-ranked CSPs exhibited a difference of more than 3.4x in CPU and memory performance, emphasizing the need for performance testing to understand value.

The differences in performance across the top 10 providers demonstrate a strong mismatch of VM sizing and standardization in public cloud IaaS, according to Cloud Spectator. For example, UpCloud’s small VM outperformed CloudSigma’s large VM on vCPU and memory performance, allowing a user the possibility to use fewer resources and lower expenses for certain applications.

Of the providers tested, 1&1, Aruba Cloud and UpCloud “achieved the best combination of low variability to high performance over the 24-hour test period, contributing to high value and predictable performance,” according to the study.

The study also found the five providers with lowest variability (UpCloud, 1&1, Aruba Cloud, Google and ProfitBricks) have a higher average CPU/memory score of 73 compared to the remaining five providers, which average 67 in CPU/memory score.

Top performers: block storage

After testing the providers’ block storage, Cloud Spectator found Google had the most stable performance, with variability of less than 3 percent. Other providers that had low variability: UpCloud, CloudWatt, ProfitBricks and CenturyLink.

UpCloud block storage sustained higher overall Input/Output Operations Per Second (IOPS) than all of the other CSPs. However, CloudSigma had higher IOPS on Type 2 block storage testing, indicating better performance for applications running larger block sizes. UpCloud and CloudSigma also offer the highest overall disk performance.

The below table lists the performance scores and the variability percentages by CSP:

Pricing

Cloud Spectator measured monthly pricing for small, medium, large and extra-large VMs. In the first three categories, 1&1 charged the least of the providers on the top-10 price-performance list. For the extra-large VM, the provider had the second least-expensive rate.

Other CSPs that had low monthly costs: CloudSigma, City Cloud, ProfitBricks and Google.

Looking at traffic costs, bandwidth pricing for egress traffic varies by CSP and by the amount of traffic. At the time of the study, 1&1 and Aruba Cloud, for example, charged nothing regardless of the amount, CenturyLink charged $0.04 regardless of the amount, and Google charged $0.12 for the first GB, the first 5 GB and up to 1 TB and then lowered its pricing to $0.11 for 1 to 5 TB and 5 to 10 TB.

Readers can download “2016 Top 10 Cloud Vendor Benchmark, European Report” for free using coupon code REPORT2016EU.