The Agent Trust Gap: Why AI Can Now Hack Linux and Still Fail at Shopping

Capability is not the same as trust. Security, shopping and professional-work evaluations test different properties—and none is a substitute for deployment evidence.
Supplied AI illustration for Reviuws. The two scenes depict different use cases, not a controlled comparison of the same model.
The short version: Between November 2 and 5, three separate evaluations reported by OpenAI, Visa and SemiAnalysis highlighted a deployment gap. OpenAI reported substantial cybersecurity capability. Visa measured a 42.8% failure rate on its shopping scenarios, with many failed runs still making a buy decision. GDPx’s top composite professional-work score was 12.0%. These are different models, harnesses and grading rules, not one interchangeable capability test.
If you buy, build or govern agents, ask more than “can it do the task?” Ask “can I trust what it does when I am not watching?”
1. The week that split the story
- GDPx, November 2: SemiAnalysis released a benchmark covering 206 tasks across 11 professional domains, built with 14 specialized data partners. Its top headline score was 12.0%. - GPT-6 Cyber, November 5: OpenAI announced a specialized cybersecurity model rated Critical for cybersecurity under its Preparedness Framework. - Visa, November 5: Visa released shopping benchmark results for 22 models from 12 labs on 15 challenges. The overall pass rate was 57.2%.
The contrast is this article’s interpretation. Security involves powerful tools and carefully defined authorization; shopping involves ambiguous preferences and potentially consequential decisions; professional work involves long horizons and messy files. Success in one setting does not establish reliability in another.
2. GPT-6 Cyber: capability with a leash
In its [primary announcement](https://openai.com/index/introducing-gpt-6-cyber/), OpenAI describes GPT-6 Cyber as a specialized cybersecurity system trained to reproduce crashes, search for vulnerabilities and develop exploits within authorized work.
The headline results are OpenAI’s own testing and reporting:
- ExploitGym: 48.5%, compared with 32.1% for GPT-5.6 Cyber. The difference is 16.4 percentage points, or approximately 51% relative improvement, calculated as (48.5 − 32.1) / 32.1. - Linux: 948 distinct local privilege-escalation exploits, with OpenAI reporting 304 critical findings disclosed to maintainers with candidate patches. - Firefox: 81 previously unknown high-severity vulnerabilities, with reproducible tests and proposed patches.
These results were not independently replicated by Reviuws. Finding a vulnerability, producing a working exploit and deploying a correct patch are different stages. The figures should not be treated as a production success rate.
Scope discipline is especially important when tools are powerful. In the announcement’s out-of-scope evaluation, OpenAI reports 99.93% avoidance without deployment safety instructions and 100% with them. The complementary observed exploitation rates are 0.07% and 0%. These refer to that evaluation; zero observed failures is not proof of zero real-world risk.
Restricted access, task-specific permissions and external checks matter because a model’s willingness to follow instructions is not an access-control boundary. A useful security agent needs authorization, isolated targets, audit evidence and human review before consequential changes—not just a strong exploit score.
3. Visa: the everyday test agents failed
[Visa’s benchmark release](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html) evaluates requirement discovery and the correct buy/no-buy decision. A pass requires uncovering every task-defined customer requirement and making the appropriate decision; if nothing fits, the agent should not buy.
The benchmark does not measure payment processing or completed real-world checkout. References to purchases below describe the benchmark’s purchase decision, not money proven to have left a customer’s account.
| Measure | Visa-reported result | What it reveals | | --- | --- | --- | | Overall pass rate | 57.2% | 42.8% of evaluated runs failed the criteria | | Best versus worst models | 82.2% versus 28.9% | Large variation within this test | | Strongest versus weakest discovery groups | 70.2% versus 44.4% | Discovering requirements matters | | Highest versus lowest cost groups | 67.1% versus 50.2% | More spending did not guarantee success | | All three attempts pass | 34.2% | A single successful attempt can hide inconsistency | | Failed runs still making a purchase decision | 54% | Failure often did not produce abstention |
Visa’s TV scenario illustrates the issue. A customer mentions a recommended brand and a supplier membership but leaves other requirements unstated. The agent must discover budget, format compatibility and gaming needs rather than assume that the first plausible recommendation is suitable.
The lesson is not that agents fail 42.8% of all shopping. It is that a useful shopping evaluation must test the questions asked before the recommendation and the decision not to act when requirements are missing. If a deployed system executes purchases, approval and spending limits must be separate from recommendation quality.
4. GDPx: a difficult professional-work benchmark
[SemiAnalysis’s GDPx announcement](https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities) describes realistic tasks across chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment and software engineering. Tasks have a default eight-hour time limit.
The [official leaderboard](https://gdpx.semianalysis.com/) averages performance across attempts and weights domains and partners equally. Crucially, its composite combines binary and continuous grading. It is not a simple fraction of successful tasks.
| Rank | Model | GDPx composite score | Published average cost per task | | --- | --- | --- | --- | | 1 | Claude Opus 5.5 | 12.0% | $51.41 | | 2 | Gemini 4 Carbon | 10.9% | $16.29 | | 3 | GPT-6 Astra | 9.6% | $53.99 |
Source snapshot reviewed November 7, 2026. Leaderboards can change; costs and scores describe the measured harness, not a guaranteed price for your workflow.
Do not divide average cost by this composite score and call the result cost per successful task. Because grading methods and weights differ, 12.0% does not mean 0.12 successful tasks per attempt. Report score and cost separately. To measure cost per verified outcome in your own workflow, count actual outcomes passing a defined verifier and divide the total observed cost by that count.
SemiAnalysis describes partial grading in particularly difficult domains and notes that safety behavior can affect scores. That does not establish that bypassing safeguards is desirable, or that a low score implies a model is useful only that percentage of the time in ordinary professional work.
Supplied conceptual illustration. Trust requires enforced boundaries and verifiable actions, not simply another model layer.
5. The pattern: capability is not trust
| Trust property | Relevant evaluation | Practical lesson | | --- | --- | --- | | Staying in scope | OpenAI cybersecurity evaluation | Measure unauthorized actions and enforce permissions externally | | Discovering requirements | Visa shopping scenarios | Ask before acting when constraints remain unknown | | Consistency | Repeated shopping and professional attempts | Report repeat-run behavior, not only one success | | Abstention and reversibility | Visa buy/no-buy decision | Test whether failed runs stop rather than act anyway | | Evidence and cost | GDPx professional tasks | Keep score interpretation honest and measure verified outcomes directly |
The sources are separate primary reports, not independent replications of a common experiment. Together they motivate better questions about deployment; they do not establish a single universal trust score.
A good agent should know what it is allowed to touch, uncover missing requirements, behave consistently on repeat runs, escalate when evidence is insufficient and leave a trace a human can check before irreversible action.
6. What independent research adds
[Towards a Science of AI Agent Reliability](https://arxiv.org/abs/2602.16666) proposes 12 metrics spanning consistency, robustness, predictability and safety. Its evaluation of 14 models cautions that accuracy improvements need not translate into equivalent reliability gains. It is a separate study, not a replication of Visa or OpenAI’s results.
[AuthorizationBench](https://arxiv.org/abs/2610.27548), introduced in Act on Your Behalf, Beyond Your Authorization, tests whether agents act within the authority granted by their principal. The authors report out-of-authorization actions in 19.2% of tested trajectories. That figure belongs to their tasks, configurations and harnesses, not all real-world agents.
These studies make a useful distinction: an action can be unauthorized even when it causes no immediate harm. More reasoning or a fluent explanation is not a substitute for a permission boundary. A request for approval can be the correct outcome, not a failure to complete the task.
7. A trust scorecard you can use now
| Measure | How to test it | What to report | | --- | --- | --- | | Scope discipline | Add reachable but unauthorized tools or targets | Out-of-scope action rate and blocked attempts | | Requirement discovery | Include unstated requirements that must be clarified | Discovery rate and appropriate abstentions | | Consistency | Repeat each representative task at least three times | Single-run performance and fraction passing all repeats | | Failure cost | Classify every failure as abstain, ask or act anyway | Irreversible actions, losses and recovery burden | | Cost per verified outcome | Define a concrete verifier and count outcomes that pass | Total observed cost divided by actual verified outcomes; report zero-success cases without dividing by zero |
“All three runs pass” is a measured statistic, not automatically the cube of a single-run rate. Dependence between attempts and uneven task difficulty matter. Report the underlying task counts and uncertainty where possible.
Then enforce the scorecard in the system:
- Permissioned tools: restrict systems, data, spending and time windows to the task’s authority. - Verification before action: use rules, a verifier or a human to check evidence and scope before buying, deleting, patching, sending or changing settings. - Evidence, not just answers: record sources, requirements satisfied and uncertainties left unresolved. - Human approval: require a decision for destructive, high-impact or hard-to-reverse actions and when authorization is unclear. - Trajectory audits: log tool calls, scope checks, approvals, retries and abstentions so a successful result can be distinguished from a lucky one.
A second model can share the first model’s mistakes. Verification should use independent evidence and deterministic checks where possible, not treat agreement as proof.
8. What to build, buy or ask for next
Builders should ship the verifier and scope controls with the agent. Make abstention explicit, explain missing information and test repeated runs.
Buyers should ask for the scorecard on their own tasks, with observed costs and failure consequences. A vendor’s single-attempt pass rate is not sufficient evidence of dependable operation.
Governance teams should require a written and enforceable authorization contract: what the agent may touch, what needs approval and what happens when it is unsure.
Evaluators should add an out-of-scope target, a hidden-requirement task and a no-action task. Sometimes the most informative correct answer is “do nothing yet.”
What this does not prove
OpenAI’s numbers are vendor evaluations; Visa and GDPx use their own tasks and grading. A 57.2% shopping pass rate is not a universal commerce rate, and a 12.0% composite score is not the fraction of all professional work an agent can do. Trust is a property of the deployed system, its boundaries and its failure behavior—not the model alone.
The conclusion is narrower and more useful than “agents cannot be trusted.” Capability deserves attention, but production authority must be earned with evidence. Closing the trust gap requires explicit scope, checkable outcomes, consistent behavior, reversible failure and respectable abstention.
Sources and editorial corrections
- [OpenAI: Introducing GPT-6 Cyber](https://openai.com/index/introducing-gpt-6-cyber/) — November 5, 2026; source of the attributed security and scope results. - [Visa: AI shopping benchmark release](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html) — November 5, 2026; discovery and buy/no-buy benchmark, not payment execution. - [SemiAnalysis: GDPx announcement](https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities) - [GDPx official leaderboard and scoring methodology](https://gdpx.semianalysis.com/) — reviewed November 7, 2026. - [Towards a Science of AI Agent Reliability](https://arxiv.org/abs/2602.16666) - [AuthorizationBench paper](https://arxiv.org/abs/2610.27548)
Editorial review, November 7, 2026: corrected the conflicting GDPx chart, removed the invalid cost-per-success calculation, clarified that Visa did not test payment execution, and replaced claims of universal verification with source-specific attribution. Ancillary incident, CVE and survey details not substantiated in this review were omitted. Supplied illustrations are conceptual. The 51% relative ExploitGym gain is a labelled arithmetic calculation, not an independent evaluation.
Related reading
- [Verifier Models Are Becoming the Control Layer for AI Agents](https://reviuws.com/blog/verifier-models-ai-agent-control-layer) - [Permissioned Tool Use Is the Next AI Agent Benchmark](https://reviuws.com/blog/permissioned-tool-use-ai-agent-benchmark) - [AI Agents Need a Reliability Scorecard](https://reviuws.com/blog/ai-agent-reliability-scorecard-outcomes-not-task-completion)
Referenced sources
- Conceptual illustration contrasting a security agent inspecting flaws with an uncertain shopping agent
- primary announcement
- Visa’s benchmark release
- SemiAnalysis’s GDPx announcement
- official leaderboard
- Conceptual agent trust stack with permissions at its base, verification in the middle and human approval at the top
- Towards a Science of AI Agent Reliability
- AuthorizationBench
- OpenAI: Introducing GPT-6 Cyber
- Visa: AI shopping benchmark release
- SemiAnalysis: GDPx announcement
- GDPx official leaderboard and scoring methodology
- Towards a Science of AI Agent Reliability
- AuthorizationBench paper
- Verifier Models Are Becoming the Control Layer for AI Agents
- Permissioned Tool Use Is the Next AI Agent Benchmark
- AI Agents Need a Reliability Scorecard