Big Data’s reputation has taken a bit of a battering lately thanks to allegations that the NSA is collecting and storing people’s web and phone records, leading to a wider debate about the appropriateness of such extensive data-gathering operations. But this negative publicity detracts from the reality of Big Data today, which for the most part will only benefit society as a whole. There’s more to these massive data sets than simply catching terrorists (or spying on law abiding citizens), as you’ll see with the following recent developments.

One story that caught my eye recently relates how ‘spying’ on people’s public data can actually help medical professionals to save lives. Data can be gathered from many different sources, but few are as superior as Twitter, and not just because of its widespread user-base that’s spread across the globe. More important, tools such as TwitterHose facilitate this data calection, allowing anyone to download 1% of tweets made during a specified hour at random, giving researchers a nice cross-section of the Twitterverse.

See the entire Future Big Data Series on Pinterest and Springpad!

.

Researchers at Johns Hopkins University have been taking advantage of this tool, downloading tweets at random and sifting through this data to flag any and all mentions of flu or cold-like symptoms. Because the tweets are geo-tagged, the researchers can then figure out where the sickness reports are coming from, cross-referencing this with flu data from the Center for Disease Control to build up a picture of how the virus spreads, and more importantly predict where it might spread to next.

Of course, there are many countries where Twitter isn’t all that popular and so researchers are forced to use more creative methods to track disease. One way of doing so, pioneered in Kenya, is through using the so-called metadata from cell phone calls to try and predict the spread of malaria.

Now you might think that’s an impossible task if you’re not actually listening into people’s calls, but you’d be wrong. Caroline Buckee, a professor at Harvard University’s Center for Communicable Disease, hit upon the idea of assigning specific cell phone users to an ‘area’ based on the location their calls and SMS messages originated from. These areas where then rated according to the level of malaria risk, which was calculated according to reported cases of the disease. Using a mathematical model, the researchers were able to accurately predict someone’s probability of becoming infected in each ‘area’.

Their system hasn’t yet been put in place by any local authority, but the potential for saving lives is readily apparent – if the government can know in advance how many cases of malaria are likely to occur in a given area each year, it’ll be able to prepare for any epidemics well in advance.

Saving lives is a worthy cause, but the motivation to save money can be just as powerful (if not more so), and once again Big Data is helping us to do so. What with hurricane season already upon us and Tropical Storm Chantal fast approaching Florida, government and energy industry officials alike are starting to get a bit twitchy.

We can’t stop the forces of nature, but by preparing for the coming onslaught we can certainly minimize the damage it’ll cause. With that in mind, the US Energy Information Administration (EIA) has just published a new interactive map that’s aimed to help the energy sector to better see and understand the potential impact of any coming storms. The map details the country’s most vital energy infrastructure, overlaid with data from the NOAA’s National Hurricane Center. Updated in real-time, the maps allow industry chiefs to study the predicted paths of storms and understand exactly what kind of threat they might be facing from Mother Nature.

All very interesting, but the EIA’s maps are only as good as the data they’re based on, which is often far from perfect. Lucky for us, a new startup is hoping to improve the reliability of weather predictions. GeoMetWatch recently closed on $6 million of funding, and with that cash it’s planning to buy ‘space’ on six commercial satellites that it can outfit with atmospheric, data-gathering sensors.

The company believes that this data will help it to predict severe weather incidents and improve climate modeling. It will then sell this information at a fraction of the cost of traditional methods. GeoMetWatch has already secured the confidence of one high-profile customer – no less than NASA, which recently agreed a four-year deal to buy weather data from the company.