LinkedIn reflects on its open-source successes in 2015

LinkedIn reflects on its open-source successes in 2015

With 2015 coming to an end, LinkedIn Corp. has taken a look back at its year of using, developing and contributing to open-source software.

Throughout the last year, LinkedIn made some of its biggest ever contributions to the open-source community by releasing ten new original projects, including Burrow, Goblin and Pinot, while pushing major updates to existing projects such as Apache Samza, Apache Kafka, Rest.li and Voldemort.

“We’ve worked to scale our infrastructure as we reached 400 million LinkedIn members, so it’s no surprise many of our open-source projects this year focus on building out our data pipelines and tools to help make sense of our data,” wrote LinkedIn’s Igor Perisic in a blog post. “The infrastructure improvements we’ve made in Kafka have allowed us to handle 1.3 trillion messages per day, and Espresso now serves 2.2 million rows per second.”

LinkedIn open-sourced its Pinot real-time analytics infrastructure last June. The technology allows LinkedIn to sort through and analyze enormous amounts of data in real-time for a wide variety of its products.

“At LinkedIn, we have a large deployment of Pinot storing hundreds of billions of records and ingesting over a billion records every day,” said Kishore Gopalakrishna, a senior software engineer at LinkedIn, in a blog post describing how it works. “Pinot serves as the backend for more than 25 analytics products for our customers and members. This includes products such as Who Viewed My Profile, Who Viewed My Posts and the analytics we offer on job postings and ads to help our customers be as effective as possible and get a better return on their investment. In addition, more than 30 internal products are powered by Pinot…”

LinkedIn also open-sourced its lightweight PalDB technology for storing side data last October. As Linkedin engineer Matthieu Monsch explains, side data is the extra read-only data needed by a process to do its job, such as a list of stop words used by a natural language processing algorithm, or machine learning models used in machine translation, content classification or spam detection are also side data. The problem is that when this side data becomes too large, it creates bottlenecks for applications that depend on it. PalDB was built to provide a read-only embeddable database that makes it easier to scale side data.

RELATED:  Confluent retools Kafka for traditional enterprises

In his blog post, Perisic said he believes that LinkedIn’s engineers benefit from open-sourcing their projects because it means their work is exposed to the entire developer community.

“It seems paradoxical to think that developers write better software for others than they do for themselves, but it actually makes sense,” Perisic wrote. “When software is written ‘internally,’ developers have a tendency to cut some corners—and I’m as guilty as anyone—especially around documenting, making code easily readable and reusable and having all the right tests in order.”

“With open source, developers’ names are attached to the software they create and the entire community can look at it,” he continued. “This puts a human face on code and reputations on the line. Once a developer open sources some software, their names will be forever associated with it. This is a huge incentive to cross their T’s and dot their I’s. A developer wants to be associated with good stuff that is well written.”

Image credit: Minachan via pixabay.com

Mike Wheatley

Mike Wheatley is a senior staff writer at SiliconANGLE. He loves to write about Big Data and the Internet of Things, and explore how these technologies are evolving and helping businesses to become more agile.

Before joining SiliconANGLE, Mike was an editor at Argophilia Travel News, an occassional contributer to The Epoch Times, and has also dabbled in SEO and social media marketing. He usually bases himself in Bangkok, Thailand, though he can often be found roaming through the jungles or chilling on a beach.

Got a news story or tip? Email Mike@SiliconANGLE.com.

SIGN UP FOR THE SiliconANGLE NEWSLETTER!

Join our mailing list to receive the latest news and updates from our team.

SIGN UP FOR THE SiliconANGLE NEWSLETTER!

Join our mailing list to receive the latest news and updates from our team.

Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Share This

Share This

Share this post with your friends!