When it comes to the fees the big cloud platform providers charge for customers to retrieve their data, Vince Kellen is pretty blunt. “It’s a stupid pill,” he said. “I believe they do this because they believe it will keep me from leaving them. I call that ‘value subtract competition.’”

Kellen, who is the chief information officer at the University of California at San Diego, isn’t alone. Information technology leaders have long grumbled about having to pay to download, back up or transfer data they store in the cloud, or even to move it from one physical region to another. The charges, called egress fees, have been part of the cost of using services from the big “hyperscaler” cloud providers since the very beginning. A similar charge called application programming interface gateway fees is levied for API calls that access customer data.

The actual charges look modest — list prices range from nine to 12 cents per gigabyte for the first 10 terabytes customers download. But they can easily add up to thousands of dollars for even a midsized business with data-intensive workloads such as analytics or machine learning model training.

UCSD’s Kellen calls data egress fees “value subtract competition” that dissuades him from giving them more business. Photo: UCSD

Kellen said the fees have held him back from doing more business with the big cloud service providers. “In university land, especially in the research domain, we often move very large sets of data around,” he said. “The costs might be such that it throws the issue over the top and we opt to do the work on-premises rather than in the cloud.”

The fees are coming under new scrutiny because of customers’ growing use of multiple clouds and spiking interest in artificial intelligence. The first use case requires that data be shuffled back and forth between infrastructure platforms, incurring charges with each byte transferred. Training machine learning models requires vast amounts of data. And all this is happening amid growing concerns about the opacity of cloud costs.

Cost invisibility

Pretty much everyone agrees that containing cloud costs is a herculean task. Gartner Inc. estimates that 60% of public cloud customers encounter cost overruns. An International Data Corp. survey found that 99% of cloud storage users said they had incurred egress fees averaging 6% of their cloud storage costs. And a study last year by Standard & Poor’s LLC’s Global Market Intelligence subsidiary found that 34% of enterprises said egress charges have affected their use of cloud storage, causing them to repatriate data on-premises or shift to a service provider that doesn’t charge for egress.

Yellowbrick’s Cusack: “It’s harder and harder to go with just one cloud.” Photo: Yellowbrick

Meanwhile, enterprises are trying to diversify their cloud footprints. A 2021 VMware Inc. survey found that 73% of enterprises use two or more public clouds today and 81% plan to do so by 2024. Some regulatory agencies are now recommending that the firms they audit use at least two cloud services for resiliency purposes, said Mark Cusack, chief technology officer at data warehousing vendor Yellowbrick Data Inc. “Cloud concentration risk is a real concern,” he said. “It’s harder and harder to go with just one cloud.”

Kellen thinks cloud providers are shooting themselves in the foot by charging customers to access their data. “I think they wouldn’t lose one jot of profitability and would get more attention from education if they said they wouldn’t charge data egress fees,” he said. “If they would drop them, they’d get more of my business tomorrow.”

But the broad unpopularity of the charges hasn’t convinced any of the major cloud providers to abandon them. The industry’s continued torrid growth outstrips the modest defection rate to such an extent that there’s little incentive to change. While fees have declined slowly over time, the reductions haven’t come close to matching the fall in cloud instance pricing. “I will dance in the streets if this happens, but I just don’t see it happening anytime soon,” said Corey Quinn, chief cloud economist of cost management advisory firm Duckbill Group.

Cloud providers had little to say about the issue. Microsoft Corp. didn’t respond to inquiries. Google LLC declined to comment, citing preparation for an upcoming conference. It provided links to two recent announcements regarding cross-cloud capabilities for its BigQuery analytics service. Amazon Web Services Inc. pointed to last November’s announcement of an expansion of its Free Tier, an offer Quinn dismissed as “$90 worth of free nonsense.”

Transparency defense

There is an argument to be made that the fees benefit customers by providing transparency about what they pay for and thus encourage greater operational discipline.

“The criticism that egress charges are somehow an ‘extra charge’ that hyperscalers have come up with to make more money, and could potentially do away with, isn’t really the case,” said Gartner analyst Craig Lowery. “They’re just being much more detailed in how they break out their cost structure and exposing it in their pricing model.”

Gartner’s Lowery: Egress fees help customers better manage cloud costs. Photo: Twitter

The thinking is that transferring data incurs bandwidth charges that are simply passed along to customers. Documenting those costs helps them run their cloud instances more efficiently. And hyperscalers have already given customers a break by not charging them to upload their data in the first place. “Egress charges can be a good thing in that they help customers to know where there may be inefficiencies in their use of the network,” Lowery said.

Jake Reichert, vice president of engineering at Yotascale Inc., a company that sells products and services to manage cloud costs, sees merit in that argument. “If I leave all the lights on in my house all day I’m going to spend more than I need to,” he said. Disclosing bandwidth costs to customers “forces you to be more aware of building software that has good network patterns,” Reichert said. “It enforces discipline.”

But some critics say the fees aren’t always transparent or predictable. “What makes egress seem so unfair is that they are often incurred for backup or disaster recovery, which are extremely hard to predict,” said Andrew Smith, senior manager of strategy and market intelligence at cloud storage provider Wasabi Technologies Inc.

Duckbill’s Quinn: “AWS’s data transfer pricing is a sad joke.” Photo: SiliconANGLE

Critics also charge that the fees are far higher than they should be. In a blog post last summer, content delivery provider Cloudflare Inc. estimated that AWS users in the U.S., Canada and Europe pay nearly 80 times what it costs AWS to deliver egress bandwidth. “For the bandwidth that they exchange with a network like Cloudflare, where they are directly connected over a private network interface, there are no meaningful incremental costs and their effective margins are nearly infinite,” wrote Cloudflare Chief Executive Matthew Prince and Nitin Rao, the company’s senior vice president of global infrastructure.

“AWS’s data transfer pricing is a sad joke that should give folks serious pause when evaluating a cloud provider,” said Duckbill’s Quinn. “These are very large, very complex environments. Which service is sending data to the outside world? What is driving the bulk of that?  Is it spread evenly across all workloads? Could one thing that’s misconfigured send the same thing over and over again? The transparency argument radically falls down.”

Hidden agenda?

There’s also an emotional dimension to the argument that’s harder to dismiss. Moving large amounts of data is a hassle, so “where your data lives is invariably where the compute moves to,” Quinn said. “It’s easier to move compute than it is to move a crap ton of data.”

Critics contend that egress and API fees are a way to discourage customers from shifting workloads to other suppliers. “I think it’s disingenuous to say this is about transparency,” said Yellowbrick’s Cusack. “It’s done to lock in customers and prevent them from taking a multi-cloud stance. Egress fees are a tax on what customers want to do.”

“It almost feels like a fairness issue,” said John Graham-Cumming, Cloudflare’s chief technology officer. “Whether the dollar amount is huge or not, there’s something odd about being charged to look at my data.”

No one accuses the cloud providers of acting in bad faith. Rather, they are charging what the market will bear. “I have to say I have never seen AWS act in ways that are eyebrow-raising, unethically suppressing competition,” Quinn said. “It is just such a nuanced and arcane area.”

Host of alternatives

Cloudflare’s Graham-Cumming: “There’s something odd about being charged to look at my data.” Photo: Cloudflare

There’s no reason that egress fees should be an immutable fact of life for companies wanting to move workloads to the cloud. Several vendors offer AWS S3-compatible storage services that are free of egress or API fees and priced well below offerings from the major hyperscalers. They include Wasabi, Seagate Technology Holdings PLC’s Lyve Cloud, Cloudflare’s R2 Storage and Iron Mountain Inc.’s Iron Cloud.

These companies shift data back and forth between cloud providers using low-cost direct connections or peered interconnects and absorb the costs into their storage fees. “We are in a beautiful multicloud ecosystem where customers are leveraging CSPs and Lyve Cloud is a good storage backplane for their multicloud setup,” said Manjusha Gangadharan, vice president for customer engagement at Lyve Cloud. Seagate charges nine dollars per month per terabyte with no additional fees.

Wasabi claims its pricing of $5.99 per month per terabyte is 80% less than Amazon’s. The company has raised more than $530 million, including a huge $250 million round last month. “We can be half the price of what you’d consider the cheapest AWS tier and we can still make money,” said CEO David Friend.

Cloudflare says it signed up more than 11,000 developers in the weeks following R2’s beta test launch. The Bandwidth Alliance, a consortium of service providers it launched four years ago, has enlisted 19 partners — including Google and Microsoft — that have pledged to reduce data transfer fees by using Cloudflare’s content delivery network.

Application-layer protection

Managed service providers have also invented tricks for minimizing data transfer costs for their customers. Alluxio Inc., the maker of a virtual distributed file system for data science and analytics workloads that spans multiple clouds, uses a data abstraction layer and caching to store frequently accessed data close to the user’s application and reduce round-trips to the cloud. “Users can access data from different data siloes and we can tell if data is hot and store it closer to the computation,” said Bin Fan, vice president of open source and the company’s founding engineer. “We can cache it so you don’t have to extract the data again and again.”

Walmart’s Evans. The retailer all but eliminated egress fees by using direct connections. Photo: LinkedIn

MariaDB Corp.’s SkySQL managed database service “supports multicloud and provides a single pane allowing users to see their databases running in different clouds,” said Kevin Farley, the company’s director of strategic alliances.

And the reality is that few large enterprises pay full price for data access, anyway. When negotiating reserved instances and multiyear agreements, cloud providers are often only too happy to use egress fee discounts as a bargaining chip. In building a nationwide multicloud platform, Walmart Corp. was able to all but eliminate the fees by using direct connections and Microsoft Azure Express Routes. “Cost is not the major constraint,” said Kevin Evans, vice president of infrastructure services at the retail giant.

AWS “has very significantly waived their fees,” for some UCSD contracts, Kellen said.

Here to stay?

Will all these forces sway any of the big three hyperscalers to reduce or do away with egress fees entirely? Observers say you shouldn’t hold your breath. “I don’t think there’s enough pressure at the moment to eliminate them,” said Wasabi’s Smith. In his previous role as a cloud analyst for IDC, “I asked AWS if they’d consider it and the answer was basically no.”

UCSD’s Kellen says he raises the issue to platform providers “all the time. I strongly encourage the vendors to reconsider their policy. The response I get is inertia. It’s the way they work,” he said. That’s even though “cloud sales reps and sales leaders say they know they’re losing business because of this.” But the amount of business they’re bringing in is a disincentive for change.

Experts also say customers shouldn’t get too hung up on egress fees when there are so many other areas of potential cost containment. “We are seeing more zero-egress cloud object storage services emerging,” said Alluxio’s Fan. “These kinds of new storage services will become more prevalent for enterprise buyers.”

Photo: Foto-Rabe/Pixabay