Applications and software will undergo a radical transformation in the coming years as developers move to embrace all-flash environments in the data center. This transformation will be hastened by the continuing evolution of all-flash array architectures, which research firm Wikibon believes will supersede spinning disk within the next five to 10 years. Cost will be the chief factor driving this evolution, with flash storage set to become cheaper than traditional disk storage in most use scenarios by 2016.

Wikibon CTO David Floyer believes the combination of continuing declines in the cost of raw flash, new flash array architectures that allow data sharing across many applications without performance impacts and new data center deployment models will trigger a major shift in the market away from high-speed disk to all-flash arrays.

These all-flash environments will lead to benefits such as savings in power and cooling, greater operational dependability and the elimination of unpredictable catastrophic failures caused by disk crashes. Organizations will also be able to reorganize IT infrastructure management, slim down staffing, automate and orchestrate deployment and move rapidly toward a shared architecture for almost all enterprise data.

But industry experts say an even bigger impact will be felt by developers, who’ll be able to take advantage of the shift to all-flash to  rethink the way they build applications. The shift to all-flash “will change everything, particularly the way in which apps are written,” said Dave Vellante, CEO and Co-Founder of Wikibon.

The flash effect

The single biggest advantage of flash is that it’s faster and more connectable, explained Wikibon’s Floyer in an interview. Latency is massively reduced, and that can lead to big opportunities for application developers. “One of the problems with disks is that they’re very slow, and that restricts how you can write applications because of this variance, particularly database applications which are constrained by how much data you can move and how many database calls you can make,” Floyer said.

In an all-flash environment, those problems are significantly reduced. Flash’s low latency and low jitter allow for hundreds or even thousands of times more database calls to be made in the same period of time, Floyer explained. As a result, applications that are designed to run on flash can process more data faster than those built for traditional disk.

“For example, if you wanted to use an application to check whether something is in stock, look up the a possibility of getting it to somebody or ensure there’s nothing wrong with the shipment, you can do all of that in real-time,” the analyst said. “You can create much more efficient applications. Organizations will need to rethink how they design applications and how they design their workflows to take advantage of all-flash.”

For Reynold Xin, Spark Project Management Committee member and co-founder of Databricks, Inc. the effects of the coming flash evolution are already being felt. “Applications are now able to run much faster than before,” Xin said in an interview. “Today you can process data relatively fast on flash storage assuming that it fits on flash, which you are likely to have more of than RAM.” Databricks already uses flash as an intermediate cache in its software.

Further changes will be needed within the data center, too. Josh Goldstein, vice president of marketing and product management for XtremIO at EMC, said flash will enable new approaches to data reduction and copy management, among other things. These will combine to deliver “more speed, simplicity, and agility – at lower overall costs.”

Rethinking application design

But the most notable change will be the way in which developers build applications, the experts agreed. Wikibon’s Vellante believes all-flash environments will lead to a massive boost in app response times and performance. “Users will see much better response times for existing applications and new applications that were previously impossible will be developed,” he said. “The emphasis will be on near real-time and with data in-memory – in other words, real, real-time.”

This will necessitate fundamental changes in the way applications are designed, said XtremeIO’s Goldstein. He explained that classic application design tries to utilize small working sets and cache as much data as possible to avoid having to access slow disk storage. Developers have also had to gang together and reorder writes to squeeze the most performance out of disk. But in flash environments all of this can be side-stepped, meaning application design will be much more streamlined. “All of that code can go away now,” Goldstein said. “We’re entering an era where applications and OSs no longer need to be designed that way.”

With a more streamlined design process, application developers will be able to take advantage of the faster data processing abilities of flash to build apps that are far more powerful and offer a much richer experience, said Brian Bulkowski, CTO and co-founder of Aerospike, Inc.. By unlocking the storage layer, future apps will be able to direct more resources toward creating a faster and more responsive user experience. “In application terms, that means firing off more requests to different API components and bringing them together,” Bulkowski said. “To do so efficiently requires asynchrony, and programmers competent in thinking asynchronously with Node, message queues, HTML5, and similar technologies will consistently outperform teams using block-and-wait infrastructures.”

In other words, all-flash architectures will enable the “real, real-time” apps that Wikibon’s Vellante speaks of, apps that can crunch more data and spit out insights as much as twenty times faster than today’s breed.

“Imagine the benefit to a retailer of being able to real-time process and correlate 1,000 factors instead of 50 – it can be game-changing,” said Goldstein. “One of our customers told us that if they could forecast inventory needs one day further into the future it could save them $1 billion a year. That’s an amazing business justification for what some people consider ‘expensive’ storage.”

Quick as a flash

These changes are likely to happen sooner than we think. Ali Ghodsi, head of engineering at Databricks, told SiliconANGLE that his company is already working to accommodate the all-flash future. The Databricks Cloud service now provides a solid state device (SSD) cache developed specifically for flash storage that stripes data across multiple drives for better throughput. RAM caching might be faster, but SSD caching has its own advantages, like being able to survive machine restarts and system failures.

“We have invested heavily in tuning and optimizations for SSDs,” said Ghodsi, adding that by tuning systems and saturating the throughput of 8 SSDs on Amazon EC2, the company set a new world record for sorting 100TB of data.

Databricks isn’t alone in trying to take advantage of the cheaper cost of flash. Aerospike’s Bulkowski said numerous app vendors are already investing in tooling around machine learning and higher context applications. “It’s quite clear that high context, personally addictive applications are getting easier and easier to build,” Bulkowski said. He cited Facebook’s timeline management, which incorporates sophisticated algorithms that go far beyond “likes” to provide rapid response and a customized user experience.

Unintended consequences

The shift to all-flash data centers will also result in a number of unintended consequences, including a bottleneck on the networking side.

“The bottleneck moves,” said XtremIO’s Goldstein. “What used to be a storage bottleneck becomes a server, network or application code bottleneck.”

Aerospike’s Bulkowski made a similar point, noting that when you remove one bottleneck (storage), the next one comes to the fore. “With the rise of flash, network throughputs will be under great strain,” he said. “Ten gigabit Ethernet is now looking puny. We have customers asking for the next generation of 40G and 100G Ethernet.”

But the network bottleneck will be relatively shortlived, Bulkowski believes. Advances in areas like software-defined networking will quickly improve performance. When combined with the power of all-flash storage, this will give rise to something he calls “deep structure machine learning,” or artificial intelligence.

Achieving deep machine learning requires “terabytes upon terabytes” of random-access memory, Bulkowski. “The software architecture requires fast networks, lots of CPU cores, and lots of random access storage,” he explained. “With flash and the increase in networking, multilayer associative machine learning will move quickly from research to reality in not just self-driving cars and self-adjusting thermostats, but self-composing stories and songs.”

Rob Enderle of the Enderle Group agreed that flash is superior to disk storage in almost every way. “While flash is more expensive than magnetic media, the value is high enough that buyers prefer it anyway, and the gap is clearly closing to the point where most magnetic media market opportunity should collapse in the next 12 to 36 months,” he said.

Just a flash in the pan?

But flash may not have time for a victory lap before something better comes along.

Paul Teich, CTO and senior analyst at Moor Insights & Strategy, believes that “fast, non-volatile, commodity-priced DRAM replacements are also probable within the next five years.”

Teich is referring to much-hyped but long-delayed technologies like Memristor, which is currently being developed by Hewlett-Packard Co. and others. Memristor, which is the fourth circuit element after the capacitor, resistor, and inductor, is a type of resistor that stores information even after its power source is cut off. It offers the potential for incredibly fast storage with minimal power consumption.

The problem with memristor is that it’s devilishly difficult to manufacture, as evidenced by the delays HP has experienced. The PC maker made its first memristor prototype back in 2008, promising to launch “a replacement for flash in the summer of 2013”. HP’s plans have suffered a string of setbacks since then, and its current stance is that 100TB memristor drives won’t be available until 2018 at the earliest.

Teich believes that the arrival of non-volatile memory is inevitable sooner, and that its impact will be even more far-reaching than flash.

“Systems today are all designed around booting a system by loading OS and apps from comparatively inexpensive and high density non-volative mass storage into more expensive, less dense volatile RAM-based system memory,” he explained. “Think about a completely non-volatile memory and storage system. When you shut it off, it simply stops where it is. When you turn it on, it starts immediately where you last were, because everything is exactly the same as when you left it. You’ll only need to reboot to install hypervisors and OS patches. Windows will have to be completely redesigned.”

In short, flash storage is just a stepping stone to something better. “The big step will come when we eliminate the difference between main memory and storage,” Teich predicted. “SSDs are not a disruption. They are a last performance bump for mass storage before we get non-volatile system memory.” At that point, he said, “Things really do change a lot.”

photo credits: scismgenie via photopin cc; Yuu Nakamichi via photopin; Lou Tamposi via photopin cc; ccbl0ndeeo2 via photopin cc