Netflix’s Jeff Magnusson on Big Data Visualization

Netflix is not only the world’s largest provider of streaming media, but also one of the most prominent contributors to the Hadoop ecosystem. Jeff Magnusson, Netflix’s head of Data Platform Architecture, discussed the open-source technologies his company uses to keep track of consumer trends in an interview with SiliconANGLE NewsDesk Head Writer Winston Edmondson.

Winston starts the session by asking about Netflix’s famed corporate culture. Magnusson says that the firm is constantly on the prowl for intelligent people, including analysts and other professionals who don’t necessarily possess advanced technical skills. Netflix’s engineers are tasked with developing tools that can help their less tech-savvy colleagues gain insights from data without concerning themselves with the underlying technologies.

Magnusson highlights that a lot has changed since the days of DVD deliveries: back then, most consumers didn’t interact with Netflix after ordering titles. Today, viewers generate so much data that members of his team have to use custom visualization software just to keep up.

Netflix utilizes Apache Pig for recommendations, video optimization and other important workflows. Pig syntax is highly abstracted, Magnusson explains, but raw code is often difficult to understand due to the complexity of certain scripts. He tells Winston that Netflix solved this problem with a visualization tool called Lipstick. The homegrown program transforms code into directed acyclic graphs, or DAGs, that make it easier to spot bugs in large projects. These charts also enable developers to view MapReduce jobs as they execute.

Mangusson says that Netflix’s appetite for data will continue to grow as it expands to new markets. He predicts that real-time technologies such as Storm will shape the future of the Big Data ecosystem and revolutionize user experience in the process.

Check out the video below for the full interview.

 

About Maria Deutscher

Maria Deutscher is a staff writer for SiliconANGLE covering all things enterprise and fresh. Her work takes her from the bowels of the corporate network up to the great free ranges of the open-source ecosystem and back on a daily basis, with the occasional pit stop in the world of end-users. She is especially passionate about cloud computing and data analytics, although she also has a soft spot for stories that diverge from the beaten track to provide a more unique perspective on the complexities of the industry.