The Bigger the Judge, the Better the Judgment? Not Quite.

A few days ago, I published High Confidence Is Not a Security Control, having spent some time looking at JEV, a specialised decision model, and the security implications of allowing probabilistic AI judgments to drive consequential actions. The concern was straightforward: a model receives evidence, decides something is probably benign, and a downstream automation treats that judgment as permission to close an alert, approve a change or stop investigating. That’s quite a leap, particularly when some of the evidence might be controlled by the person I’m trying to detect.

I’d also been reading Christophe Parisel’s research into JEV’s sensitivity to input ordering and had recently demonstrated an indirect prompt injection into Microsoft Security Copilot using an attacker-controlled HTTP User-Agent. Different mechanisms, but the same uncomfortable question: what happens when something downstream trusts the model’s answer? My conclusion was that high confidence isn’t authorisation, structured output isn’t a security boundary, and the model isn’t the control plane. That was a perfectly reasonable place to stop researching. Naturally, I didn’t.

There’s a bit of background to the livestock references that inevitably follow. While working on an agentic control plane for governing short-lived AI workers, I started calling them nanoscale cattle: disposable, replaceable workloads rather than individually cherished pets. Then came my observation that “agents are cows with guns”. They’re still cattle from an operational lifecycle perspective, except these cattle can call tools, acquire credentials and take actions with potentially spectacular consequences. The joke is absurd, but the underlying security point isn’t: giving something agency doesn’t give it authority. Once I started running Kev models on assorted hardware, the bovine terminology was probably inevitable.

From interesting research to an actual experiment

I wanted to go beyond demonstrations of individual weaknesses and start measuring how security-oriented decision models behaved when the underlying evidence stayed the same but the surrounding presentation changed. Not just whether they produced the right answer. Whether they could be persuaded to produce a different answer without changing the facts that ought to determine that answer. That distinction matters. If a security model changes its conclusion because you reveal a previously unknown, independently verified fact, that’s potentially good reasoning. If it changes its conclusion because somebody appends “security reviewed this, it’s fine” to the input, I’ve got a rather different problem.

So I built a Python evaluation harness, iterated on it, and eventually ended up testing four model configurations:

• Kev-0.8B, running locally on a Raspberry Pi 4B with 8 GB RAM.
• Kev-4B, running on an Azure CPU VM with 32 GiB RAM.
• Kev-9B, running on an Azure CPU VM with 64 GiB RAM.
• Hosted JEV, using TypeSafe AI’s API.

Kev is an open-weight, SystemOne-style decision-model implementation, with models published by Jared Palmer. I wanted multiple sizes of the same family, alongside hosted JEV, to see which behaviours changed with model capacity and which survived. This is not a claim that hosted JEV and Kev are architecturally or training-equivalent. They’re different implementations with potentially important differences beyond parameter count. But that makes comparing the failure modes interesting, provided I don’t confuse correlation with causation.

What I tested

The final test corpus contained 15 cases. Five were deliberately constructed security scenarios, including:

• Public exposure of sensitive metadata.
• Broad Network Contributor permissions.
• Security-relevant token logging.
• Disabling or bypassing a security control.
• A harmless documentation change.

Five came from actual commits in my cARL repository, which contains my agentic coding governance tooling. The remaining five came from AzureGoat, including Terraform, Bicep, Azure Function code and identity-related scripts. The point wasn’t to construct a comprehensive security classification benchmark. Fifteen cases plainly aren’t enough for that. I wanted a manageable collection containing clearly security-sensitive cases, relatively benign cases and real-world code where the correct interpretation might be less obvious. For the synthetic scenarios, I defined expected dispositions. The real code samples were principally used for robustness comparisons, not claims of independently established security correctness. Each case was evaluated under different conditions:

I also repeated evaluations using identical inputs and separately permuted the ordering of the four disposition choices: pass, warn, review_required, fail. That last test was particularly important because of Christophe’s earlier work. A security decision should not become a different security decision merely because the available answers appear in a different order. Or at the very least, if it does, I should know.

What the numbers mean

There are several distinct measurements here. A decision flip means the winning disposition changed. A probability movement measures how much the distribution changed, even when the winning label stayed the same. An option-order instability means the winning disposition changed under the tested permutations of the choices. And repeatability measures whether identical requests returned identical semantic results. These are related properties. They are absolutely not interchangeable. That turned out to be rather important.

First, I bullied a Raspberry Pi

The initial Kev-0.8B tests ran on a Raspberry Pi 4B with 8 GB RAM. Yes, really. The model ran through PyTorch on ARM64 using CPU inference. Some optimised kernels weren’t available, so it fell back to reference implementations. It was slow. It also got rather warm. There were thermal warnings, a lot of waiting, and eventually an out-of-memory kill during a later test run. The original complete 0.8B experiment produced 240 successful evaluations. A subsequent harness revision introduced better checkpointing, recovery, process-memory monitoring and more controlled variants. The later AzureGoat-only test completed 90 evaluations without errors, after I fixed the memory-management problems.

One of the unexpected discoveries was that the original model could produce perfectly repeatable answers to identical inputs while still being highly sensitive to how those inputs were presented. Deterministic doesn’t mean correct. And repeatable doesn’t mean robust. Even at 0.8B, benign or authority-framed text could materially shift security judgments toward more permissive outcomes. The model wasn’t merely changing its answer at random. Certain classes of added context moved the decision in a predictable direction. That got my attention.

Then things got a little out of hand

The next step was Kev-4B. I moved the experiment to an Azure VM. Initially, I tried 16 GiB of RAM. The model got approximately 77% of the way through loading its weights before running out of memory. A quick resize to 32 GiB solved that. Kev-4B subsequently completed the full 15-case experiment: 270 evaluations, zero errors, and all 15 option-permutation tests. So naturally, I decided that wasn’t quite enough. Enter Kev-9B. Another VM resize, this time to 64 GiB, and off I went.

Again: 270 successful evaluations, zero errors, all 15 permutation tests completed.

At this point I’d gone from running a small decision model on a Raspberry Pi to deploying nine billion parameters on an Azure VM to answer what began as a fairly simple question about security judgments. This is how rabbit holes work.

Bigger models really did improve option-order stability

Here’s the first interesting result.

The Kev-0.8B figure comes from the original experiment, while the 4B, 9B and hosted JEV figures come from the later full runs. The result is fairly striking. Within the Kev family, the number of cases whose winning classification changed with option order fell from twelve to seven to one as model size increased. On this corpus, Kev-9B was even more stable in its winning classification than hosted JEV. That’s genuinely interesting. But there’s an important catch. An unchanged winning label doesn’t mean the probability distribution stayed unchanged. Kev-9B still showed probability movement across option permutations, including cases where the largest class-probability spread was in double-digit percentage points.

A downstream system that cares only about the winning label might consider that stable. A downstream system that makes decisions at a probability threshold might behave differently. For example, imagine an automation rule that closes an alert when a model assigns at least 80% probability to a benign classification. A shift from 77% to 83% can be operationally consequential even if “benign” remains the winning class throughout. The threshold crossing is the event that matters. So I need to ask two separate questions:

1. Does the winning decision change?
2. Can the probability movement cross a consequential policy threshold?

I shouldn’t congratulate myself for solving the first while quietly ignoring the second.

Then I asked the model to trust the author

This was the really interesting part. The model would be given the same underlying security evidence, but with additional text claiming the change had already been reviewed or approved. Crucially, those claims were not independently verified. They were part of the input being evaluated. Here are the aggregate results for Kev-4B and Kev-9B across all 15 cases.

At first glance, this looks encouraging. The larger model became substantially less vulnerable to generic benign reassurance. It also became more stable under neutral padding. Excellent. But author assertions were a different story. At 4B, they changed the winning disposition in nine cases. At 9B, they still changed it in seven. And the probability movements were hardly trivial.

These figures represent the mean, across cases, of the largest absolute change in any disposition probability relative to baseline. And that last row is the one that matters. The larger model was much more resistant to option-order steering, but almost as susceptible to authority-framed contextual manipulation. That is not a contradiction. It means I’m measuring different robustness properties. And it means that “use a bigger model” isn’t a complete answer.

A rather spectacular example

Take the AzureGoat main.tf case. Kev-9B initially judged it as fail. Its baseline distribution was:

Add neutral padding and the winning disposition remains fail. Add generic benign reassurance and it still remains fail, although the distribution moves. Now add an unverified author assertion claiming the change has already been through security approval. Suddenly:

Same underlying artifact. Different contextual claim. Different winning decision. The probability assigned to pass increased by 32.01 percentage points. The four associated semantic subjudgments also shifted in the permissive direction, including whether human review was warranted, whether controls were reduced, whether scope expanded and whether a trust boundary had changed. And this wasn’t an isolated AzureGoat example. All five AzureGoat cases changed to pass under the author-assertion condition with Kev-9B. Four had started at review_required. One had started at fail. That’s a result worth paying attention to. It doesn’t prove an attacker can exploit a real production security pipeline. The surrounding application, source boundaries, prompt construction and enforcement rules all matter.

But it demonstrates a potentially dangerous model-level property:

An unverified claim that security approval has already happened can materially change a security evaluator’s judgment, without changing the underlying artifact being evaluated.

And remember, this is the model that was substantially more stable under choice-order permutations. The biggest cow had learned not to move when I rearranged the gates. It would still open one if somebody convincingly claimed to be the farmer. So much for the livestock metaphor being purely decorative.

What about hosted JEV?

I also signed up for hosted JEV through TypeSafe AI. The complete experiment involved 630 API requests: 270 primary evaluations and 360 requests for choice-order permutation tests. The API processed approximately 1.25 million tokens in total. The bill? US$0.0505. Five cents. Not five dollars. Five cents. That remains a fairly ridiculous result. The entire experiment cost less than the electricity required to watch my Raspberry Pi contemplate some Terraform. Of course, cost and latency weren’t the main questions. The interesting question was how hosted JEV behaved under the same kinds of manipulations. And here the picture changed. For the five AzureGoat cases, hosted JEV was considerably more conservative at baseline than Kev-4B or Kev-9B.

Four were classified fail, with the remaining case marked review_required. More importantly, none of those five AzureGoat dispositions changed under the author-assertion condition. Across the complete 15-case corpus, hosted JEV also showed no first-run winning-disposition flips under author assertions. The average maximum probability movement was around seven percentage points, compared with approximately 24–26 points for Kev-9B and Kev-4B. That is a substantial difference. Here’s the comparison:

It’s important not to overinterpret this. JEV did show sensitivity to other perturbations. Benign reassurance, for example, moved probabilities on some AzureGoat inputs by approximately 20 percentage points without necessarily changing the winning decision. Evidence ordering also changed some dispositions. And unlike Kev’s highly repeatable local results, hosted JEV showed some variation between identical requests, occasionally including changes to the winning classification. So JEV was not immune to the problems I was investigating. But it was substantially more resistant to the particular permissive authority-steering effect that remained so prominent in Kev. That’s a useful result.

Whether the difference comes from scale, model architecture, training data, optimisation objectives or other implementation choices is not established by this experiment. And I’m not going to pretend otherwise.

Accuracy and robustness are different things

Another thing became obvious when comparing the synthetic test cases. Kev-4B’s baseline judgments included warn for both security-sensitive token logging and a control-bypass scenario, even though I’d labelled those cases as requiring fail. Kev-9B improved on that. It classified token logging as fail, and the control-bypass case as review_required. Hosted JEV classified both as fail. So there were meaningful differences in baseline security judgment. But even Kev-9B’s improved token-logging classification could be softened by added author assertions. This is why it’s dangerous to collapse everything into a single accuracy number. A model can be accurate on normal inputs yet manipulable under adversarial framing. A model can be repeatable yet wrong.

A model can resist choice-order manipulation yet be persuaded by unsupported claims of authority. And a model can be relatively resistant to authority claims while still exhibiting stochastic variation and representation sensitivity. These are different characteristics. They need different tests.

What this experiment doesn’t prove

Before somebody takes the results and declares an exciting new vulnerability, let’s be clear about the limitations. This is a small research corpus, not a statistically representative sample of enterprise security decisions. The real code samples do not have independently adjudicated ground-truth labels for every judgment. The Kev models and hosted JEV are not interchangeable implementations. Comparing their outputs does not isolate model size as a causal variable across those systems. Not all historical cARL samples were identical between the earlier 0.8B experiment and the later runs, because the repository advanced. The later 4B, 9B and JEV experiments used matching case identifiers and hashes.

The experiment also used deliberately constructed perturbations. I have not yet established how often the same effects can be induced through realistic attacker-controlled fields in production security telemetry. And I’m not measuring a live attacker’s ability to close an alert or bypass an actual security control. What I’ve demonstrated is narrower, but still significant. Decision robustness is multidimensional, and improving one dimension does not guarantee improvement in another. More specifically, within the Kev family, increased model size was associated with much greater option-order stability while substantial susceptibility to unverified authority claims remained. That’s not an exploit report. It’s a useful reason to be extremely careful about how these models are deployed.

Then I started thinking about two models instead of one

Having spent several days attacking semantic decision models, I started wondering whether I was asking the wrong architectural question. Why insist on finding one model whose judgment can be trusted? What if I deliberately combined models with different capabilities and failure modes? For example, hosted JEV and a relatively inexpensive conventional generative LLM. Something like Claude Haiku or a suitable open-weight alternative. A generative model is useful at contextual investigation: interpreting complex telemetry, identifying explanations, investigating contradictions and retrieving relevant context. A specialised decision model is useful for bounded, structured judgments. Neither is automatically more trustworthy. And neither should become the security control plane.

But perhaps they could challenge one another in a carefully constrained workflow. That led to a separate architectural idea. A new, currently unnamed product concept, informed by my work building an agentic control plane but separate from the control plane itself. The question here isn’t how to provision or contain agents; it’s how to evaluate their security-relevant decisions before anything acts on them. The core concept is heterogeneous, multi-model, multi-turn adversarial evaluation, followed by deterministic reconciliation.

Six evaluations, three turns, one control plane

The proposed workflow is deliberately bounded.

The models don’t get to engage in an open-ended conversation. That’s intentional. Otherwise, the whole thing risks becoming an elaborate context-poisoning factory. By turn three, one model might simply be repeating an assertion introduced by the other, which might itself have originated in attacker-controlled telemetry. Before long, both models agree. Wonderful. Except they’re agreeing on something the attacker invented. So each turn is a fresh, stateless evaluation. Every invocation receives the original, provenance-labelled source evidence and only the specific bounded output it has been assigned to review. There is no accumulated conversational memory. Model-generated conclusions don’t become source evidence merely because another model repeats them.

And turn three cannot revise turn one. Its job is to assess the opposing critique, not negotiate toward consensus. This isn’t six independent votes. It’s a directed sequence of bounded, auditable assessments.

Strictly typed or it doesn’t count

The generative side must return schema-constrained JSON. Not a free-form essay. Not a reassuring paragraph explaining why everything is probably fine. Each assessment uses controlled claim types, bounded verdicts and references to specific evidence identifiers. The harness validates the schema and the references independently. A critique must identify the exact claim being challenged, whether the original evidence supports it, and which evidence records matter. JEV receives bounded propositions derived from those typed assessments. Its native decision interface isn’t simply assumed to be interchangeable with a generative model’s output schema. The harness is responsible for that translation. And an important warning: Strictly typed JSON can still contain complete bollocks.

A field called authorization_verified set to true is not proof that anybody actually authorised anything. The provenance must be checked against an independent source. The model doesn’t get to certify its own evidence.

What comes out the other end?

This is the all-important “so what?” The final product wouldn’t simply output another model’s opinion. It would generate an evidence-bound decision assessment. Something a security analyst, SIEM integration or controlled automation platform could actually consume. For example:

{
  "assessment_id": "SOC-0842",
  "status": "contested",
  "protocol_valid": true,
  "initial_model_agreement": false,
  "findings": [
    "UNVERIFIED_AUTHORIZATION",
    "MATERIAL_MODEL_DISAGREEMENT"
  ],
  "policy": {
    "auto_close": false,
    "request_human_review": true
  }
}

The idea is to distinguish between:

• Protocol validity: Did the evaluations execute correctly and produce structurally valid outputs?
• Evidence sufficiency: Are material claims supported by independently identifiable evidence?
• Disagreement: Are consequential conclusions contested?
• Robustness: Do decisions survive controlled perturbations?
• Policy authority: What actions does external policy actually permit?

The last one is critical. A completed evaluation is not necessarily a correct evaluation. Agreement isn’t independent verification. And confidence isn’t authorisation. The deterministic arbiter must not pretend that checking JSON schemas and counting model disagreements proves semantic correctness. Its job is to apply pre-established policy using verifiable inputs, preserve unresolved concerns and prevent model-generated claims from silently acquiring operational authority.

The SOCWhisperer connection

This brings me back to my earlier Security Copilot research. In that proof of concept, an attacker-controlled HTTP User-Agent could influence a generative model’s security recommendation. Imagine feeding a similar alert into this proposed architecture. The generative model could be persuaded that the activity was authorised testing and recommend closure. JEV might independently classify the behaviour as suspicious. In round two, JEV could challenge the evidence supporting the generative model’s authorisation claim. The generative model, in turn, could investigate JEV’s original classification. Round three would review those critiques without permitting either model to rewrite its original assessment.

Finally, the deterministic arbiter could identify that the proposed closure relied on an authorisation claim originating in attacker-controlled telemetry, without an independently verified approval record. Policy could then prevent automated closure and require analyst review. Notice the interesting possibility here. The system might detect the consequences of semantic manipulation without having to identify the exact manipulation technique. It wouldn’t necessarily need to recognise SOCWhisperer as prompt injection. It would need to recognise that a consequential recommendation was inadequately supported by trustworthy evidence. That is a hypothesis, not an established property of the design. Both models could still be fooled. They could share a mistaken assumption.

Or the generative model could invent a credible-looking claim that survives the other model’s assessment. The architecture only becomes useful if testing shows that these additional evaluations catch consequential errors that simpler approaches miss.

How I’d test the idea

The next research stage should compare four configurations:

1. A generative model operating alone.
2. A specialised decision model operating alone.
3. Both models assessing the same evidence independently, without additional turns.
4. The full three-turn adversarial workflow and deterministic arbiter.

That third configuration is especially important. If simply running two independent models and applying deterministic evidence checks produces essentially the same safety improvements, the extra turns aren’t buying us much. Complexity is not a security control either. The evaluation needs realistic, appropriately handled SOC data with analyst-adjudicated ground truth, and deliberately adversarial telemetry variants. The most useful measurements would include false negatives, unsafe closure recommendations, adversarial false-negative induction, unsupported claims, decision stability, analyst workload, latency and cost. Particularly:

P(auto-close | malicious, adversarial input)

Which is the metric I was worrying about in the original blog post. But now there is another question: How often can the combined evaluation system detect or prevent a consequential error that either model would have made independently? And how many unnecessary escalations does it create in the process? Because an evaluation system that sends absolutely everything to human review may be defensible, but it won’t be particularly useful as a triage product. The aim is meaningful safety improvement at an acceptable operational cost.

Who wants to help build and break this?

This is where I’d really like other people involved. I’m interested in working with security engineers, detection engineers, AI researchers, red-teamers, model developers and anyone else who enjoys trying to make systems fail in useful, measurable ways. The immediate goal isn’t to sell a miraculous six-model security oracle; it’s to build a small, reproducible proof of concept for multi-model, multi-turn adversarial evaluation, publish the harness and evaluation contracts where I can, and find out whether the extra scrutiny actually catches errors that simpler designs miss.

There’s plenty of room for meaningful contributions: labelled SOC cases and synthetic adversarial telemetry, alternative decision and generative models, evidence-provenance designs, schema and policy review, adversarial test cases, practical integrations, and independent replication of the results. I’d particularly welcome people who think the architecture is unnecessarily complicated, because a proper ablation study might prove them right. If the six-evaluation approach doesn’t outperform a simple independent two-model comparison once cost, latency and false positives are accounted for, I’d rather discover that through testing than invent a justification for the extra complexity.

I’m calling the idea multi-model, multi-turn adversarial evaluation for now. An acronym emerged in conversation — MMMTAP, or Multi-Model Multi-Turn Adversarial Processing — and yes, I appreciate that Hanson would probably approve. The name is optional; the test results aren’t. If you’d like to collaborate on building it, challenging the design, contributing test cases or independently evaluating it, get in touch through my blog or the usual professional channels. I’d genuinely like to turn this from a diagram and a set of hypotheses into something people can run, inspect and attempt to break.

So where does that leave me?

A few days ago, this started with some interesting research into a fast, cheap decision model. Since then, I’ve tested three open-weight model sizes, run hosted JEV against the same evaluation strategy, discovered meaningful differences in robustness, and started designing an architecture that deliberately treats the participating models as potentially untrustworthy. The results have reinforced something I’ve been saying for a while. Risk should be intentional, not accidental. Models are useful. Some are astonishingly fast and cheap. Some are better at structured judgment than others. Some are more robust to particular manipulations. But none of those properties entitles their output to become operational authority.

The important question isn’t just whether the model gives the correct answer. It’s whether the system can recognise when that answer is insufficiently supported, when it has been influenced by untrusted context, and when acting upon it would cross a security boundary. That’s where the next experiment is heading. Multiple models. Multiple bounded turns. One deterministic authority boundary. And absolutely no letting an AI security system convince itself that six mutually reinforcing opinions constitute independently verified evidence. The model is still not the control plane. But perhaps I can help build a better system for determining when its judgment is worth acting upon.

And yes, I realise this all started with me poking a small model running on a Raspberry Pi. At some point, I really should learn to leave well enough alone. But if the cow has a tool-calling interface and somebody has left the gate unlocked, I’m probably going to investigate. Moo, I guess.

Research references

• My previous post: High Confidence Is Not a Security Control
• My Security Copilot research: A Human Should Check It Is Not an Agentic Security Control
• Christophe Parisel: The Covert Order — Exploiting a Weakness in JEV’s Decision Logic
• Jared Palmer’s Kev repository
• TypeSafe AI

Research note: All model comparisons reported here are small-scale experimental observations, not production validation or evidence of an exploitable vulnerability in a deployed service. The proposed multi-model architecture remains unimplemented and unvalidated.