At four million log lines a second, CloudFlare, Inc. sees a great deal of traffic. So much traffic, in fact, that it’s impossible to digest without some sort of Big Data solution and filtering to break it down into its informative components. To explain how CloudFlare does this heroic job, John Graham-Cumming spoke at the European tech conference on stability dotScale 2015.

John Graham-Cumming, while only titled as a programmer at CloudFlare, is best known to the world for his successful effort to petition the British government to apologize for its atrocious persecution of World War II genius cryptographer Alan Turing.

CloudFlare is the world’s preeminent anti-distributed denial of service (DDoS) attack service for web hosts—and for everyday customers it’s still free. For enterprise and business level customers, CloudFlare offers advanced DDoS mitigation plans. As a result of its free before priced model, CloudFlare’s service has an extensive and broad established base leading to that four million log lines a second figure.

The CloudFlare service sits between web pages and intercepts requests in order to determine if they are attackers and also caches web pages reduce load times—which is especially useful when thwarting potential attackers because it hides the origin of the web page served making it harder to hit.

To get actionable information out of all those logs, CloudFlare uses NGINX, LuaJIT, Cap’n Proto, Redis, Go, and Apache Kafka. Graham-Cumming describes the process in the same way that John Furrier describes TheCube, SiliconANGLE’s technology video series: “Extracting signal from the noise.”

CloudFlare uses NGINX as a reverse-proxy and a cache system and LuaJIT to script code to scan headers. Cap’n Proto is used to batch, compress, and move data around (to get it where it needs to go.) Apache Kafka ingests streaming data and queues it up for the analytics algorithms to bite into, Graham-Cumming explains that it’s used for clustering and redundancy to increase resilience. Finally, after passing through the analysis (done with Go) the analysis and consolidated logs are passed into Postgres via CitusDB, which is a sharded version of Postgres.

CloudFlare sees 400TB of data a day and that’s after compression—this is all from over 10 trillion log lines a month (at four million a second.) This means the service relies heavily on algorithms that can do analysis in-stream because there’s so much data flowing past that it cannot be stored for any reasonable length of time.

The code that does the analysis has been optimized as much as possible to reduce additional latency to web requests. Currently, CloudFlare’s technology only adds one millisecond of latency to requests in order to do threat analysis.

CloudFlare: battle hardened and weathered on Internet DDoS storms

The company launched five years ago in June 2009 and since then has become a hardened veteran of Internet attacks and DDoS. This has given CloudFlare a considerable amount of insight into the ever-evolving shape of how hackers engage cyberweaponry.

In 2010 and 2011, CloudFlare became a target for hackers and DDoS attacks because the company had not denied protection to infamous Internet mayhem crew LulzSec. Prince spoke to critics and experts about the need for services such as CloudFlare and the importance of not arbitrarily censoring customers. Prince described hosting LulzSec on the free service as “actually kind of a fun experience” that added information to CloudFlare’s arsenal as the company became the target of numerous attacks ranging from “harmless” to “clever.”

In 2014, CloudFlare experienced one of the biggest DDoS attacks in Internet history: an attack smashed against the company’s European servers that measured 400Gbps, reported Matthew Prince, CEO of CloudFlare. It measured 100Gbps higher than the previous largest DDoS attack.

Image credit: John Graham-Cumming at dotScale 2015, via YouTube