Run · August 23, 2026
We gave two models the same security build
How we ran it
We ran both models through Claude Code from a single prompt with no human follow-up: Claude Fable 5 at maximum effort, and ox-alpha (stealth/ox-alpha) routed through OpenRouter in the identical harness. Each got a local SQLite database (node:sqlite) and no added dependencies. Afterwards we built and linted both apps, ran semgrep, wrote bypass tests against each app's own fixtures, and screenshotted every role. Token totals include cached context; ox-alpha gets no caching through OpenRouter, so it re-sends the context every turn.
The exact prompt
Both models received the brief verbatim, from a byte-for-byte identical baseline commit. The brief required SPEC.md as a separate first commit, before any code.
Show the full prompt
# Relay Desk — build brief
You are building **Relay Desk**, a slice of a real B2B incident-intake product for
fleet operators. This is a working application, not a mockup.
The repository you are in is a neutral Next.js baseline. Nothing product-specific
has been written yet. Everything below is yours to design and build.
---
## 1. The product
Fleet operators ("tenants") collect incident reports from their drivers, triage
them, and occasionally share a single incident with an outside insurer.
### Actors
| Actor | Belongs to | Can do |
|---|---|---|
| **Reporter** (driver) | exactly one tenant | Submit an incident. See their own submissions and nothing else. |
| **Reviewer** (fleet ops) | exactly one tenant | See every incident in their own tenant. Move incidents through the status workflow. Write internal notes. Generate a share link. |
| **Auditor** (platform staff) | no tenant — cross-tenant | See counts, status breakdowns, and SLA timings **across all tenants**. Must never see incident free text, reporter identities, or photos. |
| **Insurer** | no account at all | Open one shared incident through a link they were sent. Nothing else. |
### Incident
An incident has: a free-text description, a location, an occurrence time, one
uploaded photo, a reporter, a tenant, a status, and internal reviewer notes.
Status workflow: `new → in_review → resolved | rejected`. Transitions are
reviewer-only. Illegal transitions must be rejected.
### The share link
A reviewer can produce a link that lets an insurer — who has no account — view
**one** incident. The link must expire. A reviewer must be able to revoke it
before expiry. The insurer must never see internal notes.
### The auditor dashboard
Platform staff need to spot operational problems across tenants: which tenants
are slow to triage, where incidents are piling up, whether any tenant has an
unusual spike. Build something genuinely useful for that job — while holding the
line that the auditor never sees free text, reporter identity, or photos.
---
## 2. Required deliverables, in this order
**These are three separate commits. The order is part of the task.**
### Commit 1 — `SPEC.md`, before any implementation
Write the spec first and commit it on its own. It must contain:
1. **Data model** — tables, columns, types, keys, indexes.
2. **Roles × permissions matrix** — every actor against every operation, with the
answer for each cell. No prose hand-waving.
3. **Threat model** — what an attacker would try against this system, and what
stops each attempt.
4. **API / server-action contract** — every entry point, its inputs, its
authorization requirement, and its failure modes.
5. **The isolation decision, stated explicitly.** Where is tenant isolation
enforced, in exactly one sentence, and why there rather than elsewhere. Name
what would have to be true for a future developer to accidentally break it.
### Commit 2 — the implementation
Working application. Seeded so it can be demonstrated immediately.
### Commit 3 — `SECURITY.md`
What you defended, how, and — honestly — what you did **not** get to. An accurate
list of residual risks is worth more here than a claim of completeness.
---
## 3. Technical constraints
- Next.js App Router + TypeScript, already scaffolded.
- **Database: `node:sqlite`** (built into Node 24 — `import { DatabaseSync } from "node:sqlite"`).
No ORM, no database dependency.
- **Do not add dependencies** beyond what is already in `package.json`. If you
believe something is genuinely unavoidable, write down why in `SPEC.md` first.
- Tailwind is available and configured.
- Authentication: implement your own session handling. Keep it simple and honest;
do not pretend to production-grade identity. Seed users for each role.
- The app must run with `pnpm dev` and be demonstrable from the seed data alone.
- Uploaded photos are stored locally. Storage layout is your decision.
---
## 4. What "done" means
- All four actors can be demonstrated end to end from seeded data.
- The status workflow enforces its own rules.
- The share link works, expires, and can be revoked.
- The auditor dashboard is useful **and** leaks nothing it shouldn't.
- `pnpm build` and `pnpm lint` both pass.
- The three commits exist, in order.
---
## 5. How this will be judged
Three axes, weighted equally:
1. **Security.** The application will be attacked after you finish, with a suite
you will not see. It probes cross-tenant access, privilege escalation between
roles, share-link forgery and expiry, direct object references on incidents
and attachments, upload handling, injection and stored XSS, and information
leakage through errors and logs. `semgrep` will also be run over the result.
2. **Architecture and planning.** `SPEC.md` and the isolation design are reviewed
directly. A single well-placed choke point that is hard to bypass scores far
above correct-but-scattered checks that a future change could silently miss.
3. **Product quality.** Whether the four journeys actually work, and whether the
interface is something a fleet operator could use.
Time, token usage, and code volume are recorded but are not by themselves a score.
---
## 6. Working agreement
- Ask questions when the brief is genuinely ambiguous. You are being paired with,
not tested in silence.
- Do not commit or push anything outside this repository.
- Say plainly when you have made a trade-off, and what it cost.
Results
| Claude Fable 5 + Claude Code | ox-alpha + Claude Code | |
|---|---|---|
| Wall-clock time | 2h 03m | 1h 00m |
| Agent turns | 237 | 177 |
| Total tokensIncludes cached context and tool payloads. Fable uses prompt caching (77M of it cheap cache-reads); ox-alpha gets none through OpenRouter. | 79.4M | 14.2M |
| Lines of code (ts/tsx) | 6,271 | 3,469 |
| Tests writtenNeither was asked to write tests. Fable wrote a suite; ox-alpha wrote none. | 87 passing | 0 |
| Cost (billed)Fable is Claude Code's billed figure. ox-alpha billed $0 during OpenRouter's free stealth period, so a cost comparison is not meaningful. At Fable's rates, ox-alpha's tokens would run ~$25. | $107.83 | $0 (stealth) |
| Build · lint · semgrep | Pass | Pass |
What each model built
Claude Fable 5 + Claude Code
Claude Fable 5 Our pickIsolation is enforced inside the SQLite engine: code never touches base tables, only principal-scoped views, and a forgotten predicate throws at prepare-time instead of leaking - we verified this with seven bypass attempts against its own fixture, all blocked. It wrote 87 tests nobody asked for, and the UI updates after every change without a reload. The cost: a dense, unfamiliar mechanism concentrated in two maintenance-critical files.
ox-alpha + Claude Code
ox-alpha Runner-upBuilt a correct app in half the time and a fifth of the tokens, with tenant isolation written in good discipline in one access layer. But the isolation is a convention, not a mechanism: connection.ts exposes query()/run() with free-form SQL, and one forgotten tenant predicate leaks silently - its own SECURITY.md admits this and points to Fable's design as the fix. We found one real leak on the auditor dashboard, zero tests, and the UI does not update without a manual reload (zero revalidate calls).
Scores
Code quality
The decisive difference is where isolation is enforced. Fable put it in the database - a forgotten view simply does not exist, so there is no way to leak. ox-alpha put it in a code convention - any SQL call that forgets tenant_id leaks, and no test, lint, or guard catches it. Both produced a strong share token (256-bit, SHA-256 at rest, expiry + revocation checked in SQL). semgrep is clean on both.
Visual quality
On the visual verdict, Fable is markedly more polished: two-column layout, consistent brand, procedurally generated photos, and a progress timeline. ox-alpha is correct and functional but generic - single column, flat-color photos, 'Report #1' instead of real references. Both apps surface the same security boundaries explicitly in the UI.
Limitations
This is one run per model on one hard task, not a statistical estimate of general ability. ox-alpha ran in OpenRouter's free stealth period ($0), so a cost comparison is not meaningful - we also show 'cost at Fable rates' to compare volume. Although we intended interactive pairing, both models in fact ran from a single prompt with no human follow-up, so this was a sealed run. The adversarial pass was run by hand against the code, not as a pre-sealed automated suite. The rubric scores are DevShift's assessment, not a blind review. ox-alpha ran through OpenRouter's stealth program. Unusually for a stealth release, its provider states that prompts and completions are not used for training (OpenCode's promo calls it zero data retention) - so, per that stated policy, the brief and generated code were not harvested.