Backup and recovery has evolved into data protection, security and cyber recovery. Today, the focus is on business resilience. The scope continues to expand, with growing attention to how attackers use artificial intelligence and how defenders can use it to better protect their assets.

In new field research with our partner Qualitate, we explored these issues, including a less frequently discussed risk: What happens when AI deployed inside an enterprise makes a mistake?

Our research indicates that confidence in recovery is running well ahead of capabilities. The study underscores the immaturity of cyber resilience and AI security. Agentic AI is putting more pressure on practitioners to secure their data, reconstruct an agent’s actions, understand what it was trying to accomplish and confidently restore the business to a trusted state. The risks created by an enterprise’s own AI deployments are becoming a growing part of the resilience challenge.

Welcome to Breaking Analysis No. 333, AI cyber resilience and the trust gap. In this episode, we welcome back Krista Case, who recently conducted in-depth interviews with 25 security and cyber resilience practitioners as part of theCUBE Research’s Cybersecurity & Resilience Summit.

Access the full slide deck and appendix with survey details and practitioner quotes.

Study objectives and scope

Information technology and security practitioners face two growing pressures: 1) Protecting the business from adversaries using AI and 2) Putting guardrails in place as their own organizations move AI projects into production. Our study examined how organizations are approaching security and resilience across both fronts, with particular attention paid to the risks created by their own AI deployments.

We explored how organizations are managing agent identity, permissions, monitoring, etc., alongside their preparations for incident response and recovery. A central question was what happens when an agent takes an incorrect action using legitimate permissions. That scenario raises three questions:

• Reconstruction: Can the organization establish what the agent did?
• Downstream effects: Can it determine the consequences for affected systems and business processes?
• Recovery: Can it recover those business processes?

These questions shaped our study into how organizations are preparing for AI to take a more active role in their operations. As agents gain the authority to act, practitioners need to understand how a mistake could spread and what recovery would require.

Ultimately, the study examined organizations’ readiness to secure their own AI deployments and recover from mistakes made by authorized agents, alongside the continuing need to defend against adversarial AI.

Confidence is running ahead of tested recovery

Our study identified just four of 25 organizations as having exercised an AI or agent incident scenario, including partial exercises. We believe this finding exposes a core trust gap at the center of the research: Confidence in recovery often runs ahead of what organizations have actually tested.

These findings come from a small sample of in-depth interviews conducted with our partner Qualitate. The conversations allowed us to explore what customers have tested, the additional security work their own AI creates and what it takes to resume a business process safely. One healthcare practitioner phrased the disconnect between confidence and readiness as follows:

No, we have not completed any tabletop exercises as it pertains to an AI agent or other similar attacks. We’re highly confident because these are scenarios similar to non-AI incidents we have handled before.

— Senior manager, information security operations, healthcare and biotech

Across the interviews, respondents who did express confidence often drew heavily on existing ransomware and incident response playbooks. Their experience with non-AI incidents understandably provides a degree of trust. But it also raises a concern in that organizations may assume that an agent taking an incorrect action can be handled in the same way as an insider or external attacker doing something similar. We don’t believe existing playbooks cover all scenarios.

Rather, we believe that assumption needs to be tested against the specific demands of an agent incident. The interviews revealed a mixed picture of true validation:

• Partial exercises: The four identified cases include limited exercises, so the count itself is not a signal for mature trust and should not be read as four demonstrations of complete business recovery;
• Individual controls: Other respondents described testing permissions, containment and the like. Those tests provide useful evidence about particular controls while broader recovery questions remain.

Complete business recovery requires an organization to reconstruct a sequence of authorized actions, identify incorrect outcomes, repair downstream effects and safely resume the affected business process. Demonstrating that the full sequence works is a nontrivial exercise.

The speed, scale and interconnected nature of agent activity makes that even more challenging. Actions can affect multiple systems and processes, creating consequences that existing playbooks need to account for. We see a risk of false confidence when organizations assume that their experience with traditional non-AI incidents will translate automatically. Testing needs to establish how well their capabilities hold up under the conditions an agent can create, which may be novel.

Bottom line: Evidence of validated, end-to-end business recovery from an agent incident remains limited in our study. We believe organizations should exercise their existing playbooks against realistic agent scenarios, with safe business resumption as the measure of success.

Validating tools across the full agent workflow

Our interviews raised questions about tooling, maturity and the processes needed to prove that existing capabilities work for agentic AI. Two findings as shown below highlight the limited reach of detection and reconstruction capabilities in the organizations studied:

• Only two of 25 respondents reported broad detection of anomalous or unintended agent behavior;
• Only three of 25 respondents reported broad monitoring and reconstruction of agent activity.

One healthcare practitioner’s assessment of the controls already implemented underscores the trust gap:

No, for the ones that have been implemented, we have not tested or validated that they work.

— Senior Manager, Information Security Operations, Healthcare & Biotech

Enterprises already have identity systems, application logs, data protection tools, etc., along with established incident response processes. They are trying to extend these capabilities to AI. The central challenge is validating that the tools can reconstruct activity across an entire agent workflow.

Agents interact with multiple applications, invoke tools and autonomously trigger downstream actions. Each system may capture its own events without providing a coherent record of the complete workflow. Incident response requires correlating those records to establish which agent acted and under what authority, then tracing the tools it invoked and the changes that resulted.

This reminds us of the early days of disaster recovery, when organizations would test failover but hesitate to test failback because they were nervous about what might happen. We see a similar reluctance to ask questions that could expose gaps in readiness or cause downstream problems. The concern is whether the connected tools and processes will hold up under realistic agent incident conditions.

Bottom line: Organizations need to exercise the full agent workflow under realistic conditions, including a plausible worst-case event, and validate that their tools and processes can reconstruct the activity and support the response.

Restoring a trusted business service

Our assessment of the field interviews highlights both the technical and business dimensions involved in a recovery. An AI agent can use legitimate access to change a record incorrectly or send a message that should never have been sent. A backup may work and the restored systems may appear healthy while the business remains unable to resume safely.

StorageTek veteran Fred Moore, an early mentor to Dave Vellante, coined the phrase: “Backup is one thing, recovery is everything.” The scope of recovery continues to expand as organizations consider the consequences of agents acting across business systems.

A director of information security at a media and entertainment firm described the problem this way:

We could restore our systems, we could restore within our recovery objectives, but that doesn’t necessarily guarantee that it’s safe. The bottleneck here would be trust.

— Director of Information Security, Media & Entertainment

Recovery point objectives (RPO) and recovery time objectives (RTO) are the so-called business metrics. But business users need more meaningful KPIs that resonate. Specifically, what a trusted, viable service looks like and how that condition can be verified. That includes establishing who needs to participate in recovery and who signs off that operations can safely resume.

This direction needs to come from the top down; and be verified bottom up. Business leaders must define the requirements, with IT and security acting as partners to provide the underlying technology and help achieve the required outcome. The responsibility extends beyond the traditional data protection team’s work of recovering data and systems.

Incorrect transactions, human error and failures that ripple across systems have existed for a long time. Agentic AI adds two complications:

• Speed and scale: Incorrect actions can propagate rapidly across the environment, increasing the scope of the incident;
• Tracing the activity: Teams need to understand the agent’s access and intended task, establish what actions it took and follow the downstream effects.

That requires a comprehensive view of what went wrong and what needs to happen to recover. The organization must be able to trace the agent’s activity and evaluate the resulting business state against clear criteria for safe operation.

Bottom line: Recovery requires explicit criteria for verifying that a business service can be trusted and clear responsibility for approving its return to operation. The business must set that direction, with IT and security providing the capabilities to carry it out.

Human oversight can mask a recovery gap

Humans in the loop help organizations restrict what agents can do. Resilience also has to account for what happens when a human misses an incorrect action or cannot keep pace with agents operating at machine speed. Two statements from the same CIO illustrates the issue:

So the given scenario will be detected by human beings before the AI agents can take the actions.

If it really happens, currently we do not have the capability to roll back the agent’s action.

— Chief Information Officer, Real Estate & Construction

The first statement shows confidence that human oversight will catch the problem. The second acknowledges a missing “undo command” if the human oversight fails. This captures why approvals and recovery testing need to advance together. Reliance on human intervention can bring a false sense of security and defer attention to the recovery details needed when a mistake gets through.

Our field study also revealed how customers are weighing the tradeoffs around autonomy, with organizations generally still containing what their agents can do. Those decisions become particularly important when agents receive authority to take remedial actions during an incident or recovery.

The level of autonomy granted should reflect the risk of the action. Before allowing an agent to approve or execute a remedial action, organizations need to understand the consequences of an incorrect decision. That calls for a tiered approach built around two considerations:

• Action risk: What could happen if the agent makes a mistake, and what level of risk is the organization willing to tolerate?
• Guardrails: What authority should the agent have, and what limits need to surround its ability to act?

We expect human approval to become an increasingly significant bottleneck over the next six to 12 months. Organizations need to establish these boundaries as they begin trusting agents to take more remedial actions.

Bottom line: Human oversight and recovery testing need to develop together. As organizations delegate more authority to agents, they must account for mistakes, the limits of human review and the risk associated with each action.

Rehearsing agent recovery in a replica environment

A retail practitioner described a viable approach to rehearsing an agent incident and we see this as a best practice. Specifically, this firm replays the agent’s actions in a replica environment, investigating the root cause and using granular snapshots to understand the transactional effects. The respondent explicitly confirmed that the approach had been tested in the sandbox with the following statement:

Yes, if the agent went across multiple systems, my approach is to replay this within my sandbox environment, which is a full replica of all of the systems.

— Director of Information Technology, Retail

The value of this approach is in reproducing the sequence of events and investigating their effects across systems. Our assessment of what such an exercise should confirm is anchored on three outcomes:

• A trace of the incident: Reconstruct what happened across the affected systems;
• A defined scope for remediation: Identify the root cause and affected transactions;
• A path back to a trusted business state: Establish how the consequences can be addressed and recovery can proceed.

We see these as minimum viable expectations for the evidence an exercise should produce.

We use the term digital twin here as a broad analogy for this approach. A fuller digital twin would involve a richer representation of business processes, dependencies and state. We believe the industry still has work to do in that direction, and that work will become more important as enterprises use AI to run critical functions. In addition, it will further expand the notion of recovery as the state of the business will become increasingly more complicated as agents perform more of the work.

The respondent also acknowledged a limitation saying: larger transaction volumes could require more time for reconciliation. Replaying actions and identifying affected transactions provides useful information, while the time needed to resolve their consequences remains an open item. Richer simulations of business environments could help organizations test agent behavior and prepare for what a failure might look like in production. In other words, testing the recovery sequence in a variety of scenarios inside a digital twin of the business prior to pushing the process into production. Further, this “red teaming” exercise must be ongoing – expanding the concept of continuous data protection into the state of a firm’s operations in more granular snapshots.

Bottom line: This example offers a practical approach to reproducing agent actions, investigating their consequences and rehearsing recovery. Its value also points toward the need for richer representations of the business processes those agents affect.

More autonomy requires demonstrated recovery

If a CISO takes one action from this study, we recommend exercising an agent incident affecting one consequential workflow already in production. Choose a workflow in which an agent has permission to act across multiple systems, and design the exercise to include incorrect actions taken with valid credentials and permissions.

The exercise should also include a scenario requiring human approval. This helps establish the boundaries around where autonomy is appropriate and where human oversight remains necessary.

The practical steps are as follows:

• Establish what the agent can reach: Identify the systems it can access and the actions it is authorized to take;
• Demonstrate containment: Test whether the agent can be stopped, its access revoked and further tool use prevented. These “kill switch” capabilities are important to limiting the scope of an incident;
  • Ensure the conditions that flip a kill switch are determined and controlled by you, the customer.;
• Reconstruct and correct: Establish what the agent did, whether the consequences can be repaired and what unresolved exceptions or residual effects remain;
• Demonstrate safe resumption: Produce evidence that the affected business process is ready to resume and establish who approves that decision;

Validation by the business owner is essential. Security teams can establish that an incident has been contained and provide evidence of what happened. The business owner must determine whether the resulting business state is acceptable. That decision needs to account for the consequences of the agent’s actions and any remediation work still required.

The exercise should document what was proved, what remains untested and which teams are responsible for closing the remaining gaps. That evidence should also inform decisions about how much authority an agent receives. The organization needs to understand its ability to reverse or correct actions and recover their consequences as it considers granting greater autonomy.

Build resilience into agent deployments from the start

The discussions at the AI Cybersecurity and Resilience Summit reinforced the themes emerging from the field study. Conversations with vendors, practitioners and researchers underscored the need to build resilience and recovery into agent deployments from the outset. Evidence that these capabilities work becomes essential as agentic workloads move into production.

The challenge includes preserving and recovering a trusted business state when an agent produces an incorrect outcome despite having legitimate credentials. Organizations need to understand how those outcomes affect their operations and what it will take to address them.

Existing investments in data protection infrastructure, security and recovery remain critical. The work ahead is to identify the gaps that emerge under the new requirements and pressures created by enterprise AI. That requires collaboration across internal teams and with vendors to understand where capabilities need to improve.

Bottom line: We believe trust in enterprise AI requires tested evidence that organizations can understand what agents have done, stop them, reconstruct their actions and recover to a healthy business state. Our research indicates that organizations still have work to do to demonstrate that their existing security and recovery capabilities can meet that requirement.

Here’s the full video analysis:

Disclaimer: All statements made regarding companies or securities are strictly beliefs, points of view and opinions held by SiliconANGLE Media, Enterprise Technology Research, other guests on theCUBE and guest writers. Such statements are not recommendations by these individuals to buy, sell or hold any security. The content presented does not constitute investment advice and should not be used as the basis for any investment decision. You and only you are responsible for your investment decisions.

Image: theCUBE Research