Data should do more than help companies save money — it should make them money, said Peter Fishman, Director of Analytics at Yammer, Inc.. Since July of 2012, Yammer has been a part of Microsoft, Corp., providing private social networks within the company and contributing to Microsoft’s suite of products.

Peter Fishman discussed Yammer’s integration with Microsoft, how companies can get more out of their data, and why tinkering with testing is so essential while on theCUBE at Hewlett-Packard, Co.’s HP Vertica Big Data conference.

Yammer’s role within Microsoft

Fishman says Yammer has held steady to its mission — “to define social within the enterprise” — while integrating with Microsoft. “We view ourselves as part of Office […] I’d say that we sort find ourselves really in the o365 world and the cloud component of Microsoft’s enterprise offering.”

When asked about upcoming trends in social within the enterprise, Fishman answered, “social has a ways to go. Nobody has entirely figured out where social is going to be in the enterprise.”

Within Microsoft, though, Yammer seems to be catching on; Fishman proudly cites the talk Microsoft CEO Satya Nadella gave to the entire Microsoft.org from the Yammer headquarters in San Francisco: “What I really like is that he gets up in front of the entire team, fields questions, nothing’s rigged, all the questions come in on Yammer and he’s able to really address it with a great vision. I think we’re really alighted to Satya’s vision of the world. And it also plays nicely for me because he’s such a big believer in data.”

The Economics of ROI

Touching on how important analytics will be for the ecosystem going forward, Furrier asked for a quick preview of Fishman’s “Data to Dollars” talk. Fishman obliged, explaining how companies have historically been using data the wrong way.

Companies have been pursing data strategies, Fishman theorized, that pursue ways data can “provide cost reductions to counterbalance all of the other costs.”  Data is an expense in terms of the technology it requires and the manpower it occupies.  This attitude misses the point, says Fishman. He believes data should “accelerate the revenue streams for the product such that we don’t just get a return in terms of cost reductions but also in terms of revenue enhancements.”

In fact, Fishman believes that “the entire economics of the ROI to data needs to be revisited.” As storage and compute costs drop, collecting data becomes much less expensive, resulting in much more massive amounts of information. “Now that we’re collecting things,” Fishman said, “we sort of stumble into ways that data can be useful.”

Tinkering with experimentation

Furrier recalled a comment that Fishman made in a previous theCUBE interview a year ago: “Focus on what’s not working.” Furrier wondered whether there have been any innovations over the past year that changed Fishman’s opinion about A/B testing. Fishman replied that running experiments solves, “In some sense, of correlation versus causation.” He stresses, though that there are several factors that can affect the results and the value of tests:

– Tests aren’t’ always cleanly implemented
– Small sample sizes
– Unfair random assignments

When engaged in testing, it is paramount that the experiment is causing the difference between two groups, says Fishman, not other factors. Some of the innovations that have emerged within the past year around A/B testing have involved figuring out ways to, as Fishman said, “Make the race a fair race.” That way, testers can be sure that the factor they manipulated is “driving the difference between [the] two groups.”

Analytics at Yammer

Vellante asked how Fishman’s A/B testing policies apply to Yammer and how Fishman uses analytics to improve Yammer services. With over “10 million users in the enterprise and over a million companies using the product,” Fishman noted, “if you think about those numbers, you [are] able to detect really small UI changes as having a benefit or a cost to the product.”

While larger test sizes have a smaller standard deviation, allowing testers can make a better statistical inference, Fishman cautioned that sometimes, using smaller sample sizes enables testers to target specific groups in order to make an inference about a sub-population. This enables testers to figure out if there are certain groups, whether they’re segmented by geography, user type, or network type, more affected by product changes. For Fishman, it’s key for testers to keep in mind that while the number of losses in an experiment is “always more dramatic that you expect,” testers still learn “when something doesn’t win.”

Can product guys be CEOs?

Shifting gears, Furrier wondered, “Do you think that CEOs should come from a product background?” Fishman recalled that David Sacks, Yammer’s previous CEO, used to argue that “a CEO has to be a product person who’s really a dictator. And ideally you have a benevolent dictator.” Fishman also mentioned one executive’s assertion that the “product person has to say no.”

Product people have to be able to maintain their vision for the product and refuse to compromise — but this can also kill a product person’s “social capital in the workplace.” What Fishman said he hopes analytics can mitigate that balancing act, helping to figure out what actually improves user experience, as opposed to negotiating what Fishman calls “the cacophony of opinions.”

photo credit: Marius B via photopin cc