Demystifying the complexities of Spark and Hadoop | #SparkSummit

Demystifying the complexities of Spark and Hadoop | #SparkSummit

There is a lot of confusion around about Spark and Hadoop, but Matthew Hunt, head of Big Data at Bloomberg LP, thinks this is only due to a lack of understanding.

In an interview at Spark Summit East 2016 at the New York Hilton Midtown in NYC, Hunt talked with Jeff Frick and George Gilbert, cohosts of theCUBE from the SiliconANGLE Media team, to demystify the complexities of working with Spark and Hadoop, as well as provide an analysis of how Spark and Hadoop fit within the market.

Hadoop developed to answer a practical problem

In a conversation that delved deep into the mechanics of Big Data frameworks, Hunt explained the differences between Spark and Hadoop, answering where they come from, where they are going, what they do today and how they fit together.

For example, Hadoop was created to resolve a practical problem — how to download and index the web economically — by engineers rolling up their sleeves to solve real issues. As the platform grew, layers were added on top and the complexity and number of tools grew, yet the instruction set remained basic. Spark was developed with a more complicated instruction set that increased its speed yet simplified the many tools of Hadoop into one.

Spark takes language constructs and makes them performant

Hunt gave a practical example to help viewers create a mental model of Spark: When you compile a program, you write code, hit a button and the computer turns it into machine-level instructions. The same thing happens in Spark — it has an instruction set under the hood where whatever you are writing in is transformed.

Gilbert summarized this as Spark “taking language constructs and making it performant.”

Mental model shift

People assume what will be in a fast computation engine, but they are often not accurate and this causes confusion, said Hunt. “There is a mental model shift, and there are pieces that haven’t come together yet to make that happen.”

RELATED:  OpenStack: From survive to thrive | #OpenStackSV

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. You can also join in on the conversation by CrowdChatting with theCUBE hosts.

Photo by SiliconANGLE
Betsy Amy-Vogt

Betsy Amy-Vogt

Betsy is a beat writer for SiliconANGLE's roving news desk, theCUBE. She covers live events worldwide, spanning expert commentary on the hottest trends in IT. Got news? Tweet it to @siliconangle.
Betsy Amy-Vogt

SIGN UP FOR THE SiliconANGLE NEWSLETTER!

Join our mailing list to receive the latest news and updates from our team.

SIGN UP FOR THE SiliconANGLE NEWSLETTER!

Join our mailing list to receive the latest news and updates from our team.

Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Share This

Share This

Share this post with your friends!