A few years ago, deepfakes were a novelty — something relegated to Reddit threads and face-swap memes. Now, they’re tools in the arsenal of cybercriminals, paired seamlessly with large language models to deliver personalized attacks that are as scalable as they are convincing. 

The sophistication lies in the handoff: Large language models generate nuanced, context-aware dialogue for use in emails, scripts and Slack messages while deepfake tech gives them a face and voice. Alone, each of these tools is dangerous. Together, they’re capable of emulating an executive in tone, content and physical presence, enough to bypass the most skeptical gatekeepers.

The issue came to the fore this week with reports that a bad actor posing as U.S. Secretary of State Marco Rubio contacted several government officials using deepfakes to imitate Rubio’s voice and writing style. It’s believed that the imposter was attempting to gain access to privileged accounts.

A Hong Kong-based employee at a multinational finance firm fell victim to a deepfake impersonation scam in 2023 that utilized video and voice synthesis to mimic the CFO during a video conference. The attackers had gathered sufficient audio-visual data from public meetings and social media posts to create a convincing replica. Backed by LLM-generated scripts referencing internal operations, they orchestrated a high-trust environment — one that concluded with a $25 million transfer. There was no malicious payload or malware — just deception executed flawlessly.

Traditional phishing markers, such as broken grammar, awkward phrasing, and vague calls-to-action, are being erased by LLMs that write grammatically, mimic tone, adapt to jargon and mirror sentence cadence. With a little prompt engineering, attackers can train LLMs to imitate the voice of a CTO, use insider references and even anticipate likely responses from the recipient.

What makes this trend deeply alarming is its potential for widespread impact. Unlike manual spear-phishing operations, which require reconnaissance and custom content for each target, LLMs allow for rapid generation of thousands of individualized messages. Voice and video synthesizers can create convincing multimedia avatars in minutes from just a few seconds of sampled content. These are no longer boutique operations. They’re becoming automated phishing factories, and they’re accessible to anyone with modest technical skills and a graphics processing unit.

Security infrastructure plays catch-up

Modern phishing attacks exploit trust. Our current security posture and tools aren’t built for that. Most phishing defenses rely on identifying suspicious patterns, such as malformed URLs, unusual IP addresses and inconsistent metadata. Deepfake-driven phishing skips all of that. It arrives via Slack, Google Meet or even a phone call.

Worse still is that AI-generated content often eludes screening by traditional email filters and behavioral detection tools. A GPT-generated message is unlikely to trigger grammar heuristics. A cloned voice doesn’t sound like a robot but like your boss. Even platforms like LinkedIn and GitHub, which are intended to foster transparency, have become sources for model training. Anything shared publicly is potential fuel for synthetic deception.

Security awareness training is falling behind, too. The traditional red flags no longer exist. Tell-tale signs, such as unusual greetings or odd phrasing, have been replaced with corporate slang and authentic details. Verification protocols — “Call your manager if something seems suspicious” — are useless when the call itself is faked.

Even newer solutions such as deepfake detection AI are only partially effective. Most require clean, uncompressed video and audio feeds to analyze facial micro-expressions or vocal modulations. Real-world conditions like poor lighting and dropped frames obscure those signals. Plus, there’s the latency problem. By the time a detection system flags a potential fake, the wire transfer may already be complete.

What’s needed is a shift toward contextual and behavioral baselining. Security systems must learn what normal communication patterns, linguistic fingerprints, and working hours look like for every user and flag deviations not just in metadata, but also in tone, semantics and emotional affect. LLMs can be trained on internal communication logs to detect when an incoming message doesn’t quite match a sender’s established patterns.

Static multifactor authentication must also evolve into a continuous process that encompasses biometrics, device location, behavioral rhythm and other factors that add friction to the impersonation process.

Stakes have been raised

The rise of high-quality deepfakes marks a fundamental shift in how we perceive digital identity. Until recently, humans were the best validators of other humans. But deepfakes and LLMs now produce outputs that meet or exceed human standards of authenticity. The human sensor is no longer reliable. That has staggering implications not just for enterprise security, but for law, governance, journalism, geopolitics and any domain that relies on trust.

Attackers are already several moves ahead in the early days of this synthetic reality arms race. Closing the gap will take more than better software; it will require architectural and philosophical shifts. Even processing healthcare documents and sharing confidential information has to be reimagined from the ground up.

Prevention and response strategies should proceed along several fronts.

• Adversarial testing — a technique for evaluating the robustness of AI models by intentionally trying to fool them with specially crafted inputs — needs to go mainstream. Red teams must start incorporating AI-driven phishing simulations into their playbooks. Security teams should build synthetic personas internally, testing how well their defenses hold up when bombarded by believable but fake executives. Think of it as chaos engineering for trust.
• Vendors must embed resilience into their tools. Collaboration platforms such as Zoom, Slack and Teams need native verification protocols, not just third-party integrations. Watermarking AI-generated content is one approach, though not foolproof. Real-time provenance verification — or tracking when, how, and by whom content was created — is a better long-term approach.
• Policies need more teeth. Regulatory bodies should require disclosure of synthetic media in corporate communications. Financial institutions should flag anomalous behavior with more rigor. Governments need to standardize definitions and response protocols for synthetic impersonation threats, especially when they cross borders.

Perhaps most urgently, we must start treating trust as a scarce resource. Synthetic content will only grow in volume, realism and accessibility. Organizations must build layers of verification around every interaction, assuming, by default, that what they see and hear can be faked.

Isla Sibanda is an ethical hacker and cybersecurity specialist based in Pretoria, South Africa. For more than 12 years, she has worked as a cybersecurity analyst and penetration testing specialist for several companies, including Standard Bank Group, CipherWave and Axxess.

Image: SiliconANGLE/Reve