Yahoo just made deep learning easier with CaffeOnSpark

2673033658_8a2f2ebc88_b

Yahoo! Inc., is getting into the artificial intelligence (AI) game with the release of new internally-built software under an open-source license. Called CaffeOnSpark, the software is able to perform ‘deep learning’ on the vast ocean of data kept in Yahoo’s Hadoop file system. Now, the company has made it available on GitHub for everyone to use.

Deep learning is a machine learning method that’s particularly useful in helping computers come to sort through and recognize user-generated data, and one of its most exciting use cases is where images are concerned. As such, Yahoo built CaffeOnSpark to help identify the billions of images posted onto its Flickr photo sharing website.

Yahoo explained that the idea behind CaffeOnSpark was to make the search function on Flickr more useful, so that it could pick out suitable images based on more than just the descriptions and tags users write when uploading their pics. CaffeOnSpark is able to learn and recognize the specific common characteristics of certain kinds of objects, animals and landscapes, for example.

Perhaps the most surprising thing is that Yahoo came up with this technology all by itself. Yahoo isn’t exactly considered to be at the cutting edge of tech anymore, but the release of CaffeOnSpark suggests it’s not so far behind bigger names like Google and Microsoft in the AI stakes at least. Yahoo’s move comes just a few months after Google outsourced its TensorFlow machine learning framework, a move that was quickly followed by Microsoft, which open-sourced its own CNTK machine learning framework just a few weeks later. Other companies, including Facebook and China’s Baidu, have also open-sourced their own machine learning technologies in recent weeks.

Yahoo’s take on machine learning is interesting though. One of the main disadvantages of deep learning is you need to move around massive amounts of data. For example, if you want to teach a search engine how to recognize dogs in photographs, you need to feed it a huge number of canine images so it knows what to look for. This usually involves transferring all of those images to new servers, which is costly and time consuming.

However, Yahoo did things differently. As the name suggests, CaffeOnSpark is a combination of two technologies – the Caffe framework, and the Apache Spark data processing engine, which sits atop of Hadoop. In other words, Yahoo has created a way to perform deep learning on existing databases without any need to move that data around.

It seems unlikely that Yahoo is going to emerge as a leader in the field of machine learning, but its innovation will certainly benefit those who are at the forefront of this exciting new technology.

Photo Credit: aftab. via Compfight cc