Skip to content

UPDATED 14:10 EDT / JUNE 06 2012

Buffer Using MongoDB for Real-Time Analytics Without MapReduce

In a blog post, Tom Moor – co-founder and chief hacker at social media post scheduling app Buffer – describes how his company uses the NoSQL database MongoDB to gain real-time insight into internal analytics without noticably affecting front-end performance…and without MapReduce.

While Buffer uses common solutions like Kissmetrics and Google Analytics for many metrics, it needs to keep track of custom benchmarks, too, for things like Firefox extension usage or API call endpoints.

Using atomic ‘inc’ methods and so-called unsafe save states, Moor writes that he’s seen lightning quick access to real-time analytics. The trick is that by Buffer’s method, the driver doesn’t wait for a save success before continuing. By limiting data growth via storing statistics in minutely, weekly or hourly batches, read speeds in the admin role are increased.

For code snippets and implementation advice, I highly recommend checking out Moor’s blog. But for my money, the real news here is that using MongoDB for analytics is keeping “168 documents created per collection, per week for data recorded hourly.” That’s a reasonably small enough sample size that Buffer can actually do the analytics in code, without requiring MapReduce or any of the solutions that implement the same. The whole idea was inspired by a two-year-old official MongoDB blog post.

The flip-side to this bare-bones method is that data visualization (by way of Flot, Moor confirms in the comments) is lacking. Right now, Moor writes that most of their charts are simple line graphs, even in situations where pie or bar charts would be more called for.

Of course, MongoDB isn’t the only option for this kind of real-time analytics implementation. We saw plenty of developers doing something similar with HBase at HBase Con 2012 in May. Others use Redis bitmaps to achieve the same.


A message from John Furrier, co-founder of SiliconANGLE:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

  • 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
  • 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network

Are you an AWS customer?  Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/

 

About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.

Send us a news tip

Send us a News Tip

  • This field is for validation purposes and should be left unchanged.

Sign in or create an account

SIGN IN

OR

New User? SIGN UP

Join us

SIGN UP

Bio

Ethics statement

Extract the signal from the noise

Get SiliconANGLE updates and analysis.

Contact us

Partner with us

Contact us

Guest inquiry