Security ratings company SecurityScorecard Inc. today released an open benchmark for testing whether artificial intelligence models can handle third-party risk management work without being turned against the companies using them.

The TPRM Benchmark is built for vendor risk programs, where a model may need to interpret security telemetry and decide which exposures matter. A lot of what the model reads comes from questionnaires and documents that vendors filled out themselves, and parts of that may be incomplete or manipulated on purpose. SecurityScorecard says generic benchmarks built around static or academic tasks were not designed with that in mind.

On the capability side, tasks cover disciplines such as exposure triage and fourth-party technology risk, and each one starts as a snapshot of SecurityScorecard’s own vendor-risk telemetry that domain experts check before any model is scored. The answer keys are protected. Inputs rotate too, so a model that memorizes the test has less to gain.

Scores are measured against a baseline with confidence intervals attached, and each model gets an autonomy readiness level from L0 to L4 based on its performance across repeated trials. SecurityScorecard says the rating is meant to show how far a model can be trusted to work without human oversight.

Attack resistance is tested separately in the Model Security suite, which hits each model with prompt injection, jailbreaks and other techniques. Grades run from A to F. Some of the capability tasks are also seeded with deceptive vendor self-attestations that contradict the underlying telemetry, and scoring on those tasks shows how far the false claims pull down a model’s accuracy.

Performance in the early runs, which covered closed frontier, open-weight and international models, depended heavily on the discipline being tested. Open-weight models are closing the gap with leading frontier models on supply chain security tasks, according to Mike Mylrea, chief information security officer and chief AI officer at SecurityScorecard. Several of the open-weight models are “highly susceptible to adversarial attacks,” he said. The announcement did not identify the models tested or give their scores.

Manoj Nair, chief technology and innovation officer at developer security company Snyk Ltd., said the benchmark gives security teams “a shared, transparent way to evaluate models on capability and attack resistance together.” Snyk’s own research across 3,000 enterprise environments found that 74.5% of those with a detected model reference at least one that scores “high” on an attacker goal under adversarial testing.

Aleksandr Yampolskiy, co-founder and chief executive of SecurityScorecard, likened putting an untested AI agent into a vendor risk program to “giving a new employee privileged access on day one and hoping for the best.” A company that can’t say whether its model can do the work, or whether the model can be turned against it, has “a new attack surface,” he said.

SecurityScorecard has open-sourced the framework. Participants who download it can run their own models or agents and submit the results to a public leaderboard. A limited group of AI labs, security vendors and researchers is also being invited to help decide what the benchmark tests and how.

Version 2026.09 of the public release is available now with documentation, the Model Security suite and registration for the leaderboard. Official expert-verified scores come from SecurityScorecard’s live internal tier. A security team running its own tests on the public tier will get results marked synthetic-calibrated, which the company says are directly comparable for research and self-assessment. For a certified evaluation of a candidate model or agent, enterprises have to go to SecurityScorecard directly.

Image: SecurityScorecard