Last year, the RSA Conference was dominated by the conversation around artificial intelligence and machine learning. Just months after the initial release of ChatGPT, virtually every vendor took the opportunity to tell visitors about how its platform or tool is powered by AI/ML engines, but few of those visitors were able to verify those claims with their own cyber or AI knowledge.

This year, conference attendees will be just as hooked on AI/ML solutions as they were 12 months ago. But I believe there’s an opportunity to educate this year’s attendees beforehand on the questions they need to be asking vendors to determine whether the next AI/ML presentation they listen to is legit or just buzzword soup.

There’s a reason the Securities and Exchange Commission has issued protections around public companies claiming to use artificial intelligence and why they’re following through against those making false claims. In March, the SEC fined two investment advisers for misleading the public about their use of AI, a practice that the industry has dubbed “AI washing.”

It can be a huge benefit for a company to coast off the buzz provided by using AI and machine learning in their marketing materials, and at last year’s RSAC, these vendors likely did not have to prove to too many attendees that they knew what they were talking about. But as AI and regulations around it have matured, there’s no more room for free and false advertising. Vendors should not be masking data scientists as AI experts, even if they know more than the average attendee.

For example, if a vendor says it’s offering an AI-based threat detection tool, it should also be quizzed on where the training data is coming from, whether its model has exhibited any drift, what is the decay rate, how it performs against the receiver operator characteristic curve, and what approach it took to build the model.

These questions represent the bare minimum of knowledge that a vendor should have when it’s advertising its AI/ML tool, and it’s critical for attendees to do their due diligence by asking it. Otherwise, they may be making a bad decision to purchase a tool or service not based on reality but on vaporware.

That’s not to say that there won’t be effective AI- and ML-based tools on display this year, or that the technology isn’t well on its way to revolutionizing cybersecurity operations. Forward-thinking vendors are already using AI-based models to identify patterns and anomalies while monitoring for threats. Even more are creating test code to train their threat detection solutions or using an AI-powered model as a natural language processor, all of which saves security practitioners valuable time and energy that they can put toward solving more complex security problems.

For now, though, the best security operations will come from a mix of artificial intelligence and human intelligence, and there is no such thing as a fully automated and AI-powered solution that can replace a human.

Of course, having a B.S. detector at industry conferences isn’t a skill that’s only valuable with AI and ML. No matter what the cybersecurity technology is, a responsible organization will have somebody on staff who is knowledgeable enough to call out a vendor on a dubious claim.

And to some degree, as long as business is providing positive security outcomes to its customers, it’s not so important whether it’s using AI or not. I’ve said in the past that AI is not always the answer, but as with many topics hyped at RSA, you must separate what’s real from what is envisioned.

Dan Schiappa is chief product officer at Arctic Wolf. He wrote this article for SiliconANGLE.

Image: SiliconANGLE/Google Gemini