This is a sponsored post commissioned by Hitachi Vantara Corp. Sponsored posts are paid content that appear on all pages of SiliconANGLE.com, supporting editorial efforts. For pricing and inquiries, email sponsor@siliconangle.com.

There’s a quiet revolution going on in the personal computer gaming industry, and it’s being driven by big-data analytics.

The $18.4 billion market has traditionally been an over-the-counter business, with DVDs boxed and sold at prices of up to $60 each. Although many games are still packaged that way, the industry is beginning a massive move to the internet. This is reshaping nearly everything about the way gaming companies are doing business.

Think of it as the difference between scripting a feature-length film and a TV show. PC games used to be self-contained, with a clear beginning and end. Once released, there was no chance to modify the experience or rescue a floundering title. With the shift to online, games can now go on for years.

ZeniMax Media Inc. is one of the companies leading this trend. The maker of such hits as The Elder Scrolls, Fallout (pictured), Doom, Quake and Wolfenstein is transitioning not only in its delivery model but the way it does business.

PC games used to be a “one and done” business. Customers might receive updates and bug fixes online, but the game experience was fixed and limited. “If you’re selling a box, once customers finish playing, they’re done,” said Rob Walsh, technical director of enterprise business intelligence at ZeniMax.

The business ran the same way. Development teams worked independently of each other and chose their own story lines and business metrics. “Every studio was self-contained; they developed, shipped and moved on,” Walsh said.

Shaking up the model

The shift to online delivery changes everything. Instead of charging at the checkout counter, game makers have more revenue options, including subscription fees and micro-payments. At ZeniMax, in-game transactions are increasingly the focus of its business.

“The whole industry is moving toward a free-to-play with micro-payments model,” Walsh said. “A $60 price is a barrier to entry. So instead, we give them the game for free, and charge them once they’re in.”

Players pay for items such as costumes, badges and special avatars. In some games, they can buy boosters, extra lives or stronger powers. The challenge for developers is to extend the story line to keep players engaged. “You can play a game, go away, come back a half year later and it’s completely different,” Walsh said. “That shift in customer engagement is a whole different world. We need to keep the game afloat so we can encourage people to spend more money.”

That requires rethinking the way games are made. The video game industry has a reputation as one of the toughest ways to make money. The majority of games fail to recoup their investments, which can run north of $100 million, and fewer than 5 percent are bona fide hits. Game makers have a short window of success. “You get a spike of interest in first 30 days and then interest declines as people move on to the next shiny object,” Walsh said.

Longer life

Online games have a much longer lifecycle because the experience can constantly change. If developers can keep players engaged with new adventures, challenges and rewards, play can go on for years. ZeniMax’s Elder Scrolls Online launched in 2013 and is still going strong. Developers also have a better chance of rescuing a game that starts off badly.

But there are risks to the free model as well. The cost of entry is low, but so is the cost of exit. If game developers don’t keep their audience’s attention, players quickly go elsewhere. “If the game isn’t fair, people won’t play it,” Walsh said. “That has direct monetary impact.”

This is where big data becomes the game changer. The new business model demands that developers closely and constantly monitor how the games are being played. Every sword that’s swung, spell that’s cast and dragon that’s slain is recorded and analyzed as developers seek the perfect balance of challenge and triumph.

Games generate a huge amount of data, as much as 5 terabytes per day for some of ZeniMax’s most popular titles. All of that is loaded into its data lake for trend-spotting. Are there particular points in the game when players tend to abandon ship? Are certain battles too difficult or too easy? How quickly do players return after they log off? What are the characteristics of players who sign up for multiple ZeniMax titles or who play the longest?

Unifying silos

Getting to an integrated data model was a challenge. ZeniMax’s historically siloed organizational structure gave developers little reason or incentive to share data. Each team selected its own metrics and made its own decisions. The result was a mishmash of formats. “We had data in MySQL, Postgres, JSON, XML and plain text,” Walsh said. “The data was there, but there was no way to consolidate raw information into one database and present it in a form that could be digested by the business.”

The integrated, online model demanded a unified approach. Player accounts may span multiple titles and they may spend money differently in each. The company needed to track activity of individual players across different games and forums. Most importantly, it needed to know where and what players buy.

Step one of the consolidation process was to unify the games into a single service in a data warehouse. Step two was to create a single view of disparate data. It started with transforming data from a jumble of formats into a single stream that could be loaded into the warehouse.

Transformation initially was done with a set of Python scripts, but the process didn’t scale well, and finding programmers with the right skills was a problem. ZeniMax went looking for a commercial solution that would serve as a long-term connector between the data lake and the warehouse.

“There are big-data solutions that can do the ingestion, and there are data warehousing solutions that a provide a good mechanism for delivering a view that’s good for the company,” Walsh said. “The problem was how to glue together the data lake and data warehouse into a single pipeline.”

Pentaho is the glue

After evaluating data integration products from Talend SA, Informatica Corp. and Pentaho Corp. (now Hitachi Vantara), ZeniMax selected Pentaho’s Data Integration and Analytics Platform. Part of the appeal of Pentaho was that it provided both data transformation and analytical modeling features.

“We had a warehouse and a lake that were very disparate,” Walsh said. “How do we take that noise and transform it into something useful? Pentaho was the glue.”

Raw data is loaded and transformed in an Amazon Web Services Inc. Redshift data warehouse, with all modeling taking place in Pentaho. The integration solution has also unified two existing data lakes into one. The entire data integration process is now visible on a single screen. “It brought us performance and a mechanism to create a single view of the business without having to see what we’re doing behind the scenes,” Walsh said.

Pentaho automated much of the work that had previously been done manually or with Python scripts. “We’ve been able to reduce the overall level of effort,” Walsh said. “We have a system that’s designed for our needs instead of something we cobbled together. And it’s much more scalable.”

Analysts can analyze

Analysts are now able to spend their time digging into gamer behavior instead of loading data. The time needed to launch a new game has been reduced from months to weeks. Equally important, developers can now fine-tune players’ experiences more quickly and keep gamers playing longer.

“We have much more visibility into people’s behavior, so we can analyze on day one instead of day 20,” Walsh said. For example, within a few days after launch, analysts can determine that players who lose a certain number of consecutive matches are likely to quit. “The developers can now react and prepare for that,” Walsh said.

Big data also improves a gamer’s chance of success. Instead of guessing which games and features will interest players, the studio can test ideas and get behavioral feedback in near real-time. It can also double down on games that show promise. “We’ve got a healthy base of people who have been playing some of our game for years,” Walsh said. “A five-year lifecycle would have been unthinkable a decade ago.”

The bottom line for ZeniMax is faster launches and improved likelihood of success longer life of successful games. “We’ve always known how many copies we’ve sold,” Walsh said. “Now we know why.”

Image: ZeniMax