Smart cities are no longer just science fiction elements, as “internet of things: devices and self-driving cars and trucks are already a reality and produce a lot of data. The ability to capture, store and analyze the immense volumes of log data generated across these networks is no small feat.

This is a challenge for companies such as BAI Communications Pty. Ltd., which provides state-of-the-art communications infrastructure (cellular, Wi-Fi, broadcast, radio and IP networks) in major commuter cities, including New York, Toronto, Tokyo, Hong Kong, Sydney and London. In other words, it keeps commuters around the globe connected while they’re in transit, driving the evolution of smart cities.

“There was an explosion of data, and a lot of our insights aren’t coming from one system anymore; it’s coming from collecting data from all of the different pieces, the different infrastructure, whether that’s your fiber infrastructure or your wireless infrastructure,” said Jeremy Foran (pictured, right), head of data analytics at BAI Communications. “And then, to solve problems, you need to correlate data across those systems.”

Foran and Thomas Hazel (pictured, left), founder and chief technology and science officer of ChaosSearch Inc., spoke with John Furrier, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the AWS Startup Showcase: The Next Big Things in AI, Security & Life Sciences. They discussed the challenge surrounding the capture, storage and analysis of increasing amounts of data, ChaosSearch’s solutions to solve these problems and how it enables BAI Communications to generate value for its customers. (* Disclosure below.)

Data compression means less cost

With different equipment to provide public Wi-Fi, its 5G networks, and the need to be able to adhere to rules like PCI compliance and ISO 27000, BAI holds a tremendous amount of data. To store and handle this in a cost-effective way, the company took on the solutions of the log data analytics platform company ChaosSearch.

“We wanted to use [Amazon] S3, but when we were doing some projections, we don’t really have the budget for all of these places,” Foran said. “Working with ChaosSearch, using their compression, brought those costs down drastically.”

The ChaosSearch Data Platform, which is available in the AWS Marketplace, enables search and analysis of multi-model cloud data on S3.

BAI streams its devices and services into one centralized data lake on its cloud outer storage. ChaosSearch then connects to that storage and turns it into an analytic database for log analysis. All new workloads also flow into this lake.

Although an essential aspect of the ChaosSearch solution is data retention, there is no chance that the data lake will become a “data swamp,” according to Hazel. The reason is that the data is discovered, organized and automatically indexed, so businesses can get insights and therefore value from them. The solution also provides real-time notifications, alerting, monitoring and will soon offer machine learning services.

“The idea is that you use this lake offering to store all your data in a cost-effective way, but our service allows you to analyze it both in a long retention perspective as well as real-time perspective,” Hazel explained. “Bringing those two worlds together is so key, because typically you have silo solutions, whether it’s real-time at scale or retention scale.”

Another fundamental feature of ChaosSearch is that data remains fully controlled by customers.

“Unlike a lot of solutions where you move the data into them and now they are responsible, actually BIA owns everything,” Hazel explained. “They provide access so that we could provide an analysis that they could turn off at any point in time.”

Getting unpredictable insights

The benefit of having a large amount of data stored is getting more and more insights, even unpredictable ones, from it, according to Foran. For example, by providing Wi-Fi in a subway infrastructure, BAI was able to get that Wi-Fi data to begin to understand the flow of people entering and leaving the subway network. This resulted in insights for rail operators to get passengers from one place to another quicker.

“When we built the Wi-Fi, it wasn’t with the intention of getting Torontonians across the city faster, but that was one of the values that we were able to get from the data,” Foran stated.

Relying on the ChaosSearch compression solution, BAI is storing all of its data. In addition, the company believes that the services offered by the platform will make it easier to use technologies such as machine learning.

“Starting from scratch and trying to build out models that have value, that takes a fair amount of work, and that landscape keeps changing,” Foran said. “Being able to push our data into an S3 bucket and then retain that data and then get anomaly detection on top of it, that’s something special and that unlocks a lot of ability for our teams.”

Watch the complete video interview below, part of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase: The Next Big Things in AI, Security & Life Sciences. (* Disclosure: ChaosSearch sponsored this segment of theCUBE. Neither ChaosSearch nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE