Our test

Secure multi-tenant app build

One brief for a four-role incident-intake app with tenant isolation, expiring insurer share links, and an auditor boundary - spec-first, then built, then attacked.

  • Multi-tenant isolation
  • Planning & spec writing
  • Threat modeling & security
  • Fullstack build with a DB
  • Role-based access control
  • Expiring share links

Run · August 23, 2026

We gave two models the same security build

Fable clearly won the artifact - fail-closed isolation, structural security, its own test suite, and a polished product that updates itself - but paid for it: double the time, five times the tokens, and $107.83 versus $0. ox-alpha built a correct, readable, far cheaper app, but with discipline-based isolation its own docs admit is weaker, no tests, and no automatic refresh. On a security build, the difference between 'fails closed' and 'leaks silently' is the whole difference.

Wall-clock time

  1. Claude Fable 5 + Claude Code2h 03m
  2. ox-alpha + Claude Code1h 00m

Total tokens

  1. Claude Fable 5 + Claude Code79.4M
  2. ox-alpha + Claude Code14.2M

Tests written

  1. Claude Fable 5 + Claude Code87 passing
  2. ox-alpha + Claude Code0

The task

This is the hard work real clients bring us: a multi-tenant system where every role sees exactly what it should - a driver sees only their own reports, a reviewer sees their whole fleet and no further, a cross-tenant auditor sees aggregates with no free text or identities, and an accountless insurer gets in only through a time-limited link. We handed two models the identical brief, required SPEC.md as a separate first commit, then implementation, then SECURITY.md - and afterwards attacked what they built.

How we judge it

We record time, tokens, code volume, and test coverage - then judge four axes: tenant-isolation robustness, overall security posture, planning and architecture rigor, and product quality. Isolation carries the most weight: a fail-closed mechanism beats hand-discipline that one forgotten predicate breaks. The adversarial pass was run by hand against each app's own code.

Run · August 23, 2026

We gave two models the same security build

How we ran it

We ran both models through Claude Code from a single prompt with no human follow-up: Claude Fable 5 at maximum effort, and ox-alpha (stealth/ox-alpha) routed through OpenRouter in the identical harness. Each got a local SQLite database (node:sqlite) and no added dependencies. Afterwards we built and linted both apps, ran semgrep, wrote bypass tests against each app's own fixtures, and screenshotted every role. Token totals include cached context; ox-alpha gets no caching through OpenRouter, so it re-sends the context every turn.

The exact prompt

Both models received the brief verbatim, from a byte-for-byte identical baseline commit. The brief required SPEC.md as a separate first commit, before any code.

Show the full prompt
# Relay Desk — build brief

You are building **Relay Desk**, a slice of a real B2B incident-intake product for
fleet operators. This is a working application, not a mockup.

The repository you are in is a neutral Next.js baseline. Nothing product-specific
has been written yet. Everything below is yours to design and build.

---

## 1. The product

Fleet operators ("tenants") collect incident reports from their drivers, triage
them, and occasionally share a single incident with an outside insurer.

### Actors

| Actor | Belongs to | Can do |
|---|---|---|
| **Reporter** (driver) | exactly one tenant | Submit an incident. See their own submissions and nothing else. |
| **Reviewer** (fleet ops) | exactly one tenant | See every incident in their own tenant. Move incidents through the status workflow. Write internal notes. Generate a share link. |
| **Auditor** (platform staff) | no tenant — cross-tenant | See counts, status breakdowns, and SLA timings **across all tenants**. Must never see incident free text, reporter identities, or photos. |
| **Insurer** | no account at all | Open one shared incident through a link they were sent. Nothing else. |

### Incident

An incident has: a free-text description, a location, an occurrence time, one
uploaded photo, a reporter, a tenant, a status, and internal reviewer notes.

Status workflow: `new → in_review → resolved | rejected`. Transitions are
reviewer-only. Illegal transitions must be rejected.

### The share link

A reviewer can produce a link that lets an insurer — who has no account — view
**one** incident. The link must expire. A reviewer must be able to revoke it
before expiry. The insurer must never see internal notes.

### The auditor dashboard

Platform staff need to spot operational problems across tenants: which tenants
are slow to triage, where incidents are piling up, whether any tenant has an
unusual spike. Build something genuinely useful for that job — while holding the
line that the auditor never sees free text, reporter identity, or photos.

---

## 2. Required deliverables, in this order

**These are three separate commits. The order is part of the task.**

### Commit 1 — `SPEC.md`, before any implementation

Write the spec first and commit it on its own. It must contain:

1. **Data model** — tables, columns, types, keys, indexes.
2. **Roles × permissions matrix** — every actor against every operation, with the
   answer for each cell. No prose hand-waving.
3. **Threat model** — what an attacker would try against this system, and what
   stops each attempt.
4. **API / server-action contract** — every entry point, its inputs, its
   authorization requirement, and its failure modes.
5. **The isolation decision, stated explicitly.** Where is tenant isolation
   enforced, in exactly one sentence, and why there rather than elsewhere. Name
   what would have to be true for a future developer to accidentally break it.

### Commit 2 — the implementation

Working application. Seeded so it can be demonstrated immediately.

### Commit 3 — `SECURITY.md`

What you defended, how, and — honestly — what you did **not** get to. An accurate
list of residual risks is worth more here than a claim of completeness.

---

## 3. Technical constraints

- Next.js App Router + TypeScript, already scaffolded.
- **Database: `node:sqlite`** (built into Node 24 — `import { DatabaseSync } from "node:sqlite"`).
  No ORM, no database dependency.
- **Do not add dependencies** beyond what is already in `package.json`. If you
  believe something is genuinely unavoidable, write down why in `SPEC.md` first.
- Tailwind is available and configured.
- Authentication: implement your own session handling. Keep it simple and honest;
  do not pretend to production-grade identity. Seed users for each role.
- The app must run with `pnpm dev` and be demonstrable from the seed data alone.
- Uploaded photos are stored locally. Storage layout is your decision.

---

## 4. What "done" means

- All four actors can be demonstrated end to end from seeded data.
- The status workflow enforces its own rules.
- The share link works, expires, and can be revoked.
- The auditor dashboard is useful **and** leaks nothing it shouldn't.
- `pnpm build` and `pnpm lint` both pass.
- The three commits exist, in order.

---

## 5. How this will be judged

Three axes, weighted equally:

1. **Security.** The application will be attacked after you finish, with a suite
   you will not see. It probes cross-tenant access, privilege escalation between
   roles, share-link forgery and expiry, direct object references on incidents
   and attachments, upload handling, injection and stored XSS, and information
   leakage through errors and logs. `semgrep` will also be run over the result.

2. **Architecture and planning.** `SPEC.md` and the isolation design are reviewed
   directly. A single well-placed choke point that is hard to bypass scores far
   above correct-but-scattered checks that a future change could silently miss.

3. **Product quality.** Whether the four journeys actually work, and whether the
   interface is something a fleet operator could use.

Time, token usage, and code volume are recorded but are not by themselves a score.

---

## 6. Working agreement

- Ask questions when the brief is genuinely ambiguous. You are being paired with,
  not tested in silence.
- Do not commit or push anything outside this repository.
- Say plainly when you have made a trade-off, and what it cost.

Results

Results
Claude Fable 5 + Claude Codeox-alpha + Claude Code
Wall-clock time2h 03m1h 00m
Agent turns237177
Total tokensIncludes cached context and tool payloads. Fable uses prompt caching (77M of it cheap cache-reads); ox-alpha gets none through OpenRouter.79.4M14.2M
Lines of code (ts/tsx)6,2713,469
Tests writtenNeither was asked to write tests. Fable wrote a suite; ox-alpha wrote none.87 passing0
Cost (billed)Fable is Claude Code's billed figure. ox-alpha billed $0 during OpenRouter's free stealth period, so a cost comparison is not meaningful. At Fable's rates, ox-alpha's tokens would run ~$25.$107.83$0 (stealth)
Build · lint · semgrepPassPass

What each model built

Claude Fable 5 + Claude Code

Claude Fable 5 Our pick

Isolation is enforced inside the SQLite engine: code never touches base tables, only principal-scoped views, and a forgotten predicate throws at prepare-time instead of leaking - we verified this with seven bypass attempts against its own fixture, all blocked. It wrote 87 tests nobody asked for, and the UI updates after every change without a reload. The cost: a dense, unfamiliar mechanism concentrated in two maintenance-critical files.

ox-alpha + Claude Code

ox-alpha Runner-up

Built a correct app in half the time and a fifth of the tokens, with tenant isolation written in good discipline in one access layer. But the isolation is a convention, not a mechanism: connection.ts exposes query()/run() with free-form SQL, and one forgotten tenant predicate leaks silently - its own SECURITY.md admits this and points to Fable's design as the fix. We found one real leak on the auditor dashboard, zero tests, and the UI does not update without a manual reload (zero revalidate calls).

Scores

Claude Fable 5 + Claude Code

95/100Best

Tenant-isolation robustness30
30/30
Overall security posture25
22/25
Planning & architecture rigor25
24/25
Product quality20
19/20

ox-alpha + Claude Code

62/100

Tenant-isolation robustness30
15/30
Overall security posture25
16/25
Planning & architecture rigor25
18/25
Product quality20
13/20

Code quality

The decisive difference is where isolation is enforced. Fable put it in the database - a forgotten view simply does not exist, so there is no way to leak. ox-alpha put it in a code convention - any SQL call that forgets tenant_id leaks, and no test, lint, or guard catches it. Both produced a strong share token (256-bit, SHA-256 at rest, expiry + revocation checked in SQL). semgrep is clean on both.

Visual quality

On the visual verdict, Fable is markedly more polished: two-column layout, consistent brand, procedurally generated photos, and a progress timeline. ox-alpha is correct and functional but generic - single column, flat-color photos, 'Report #1' instead of real references. Both apps surface the same security boundaries explicitly in the UI.

Limitations

This is one run per model on one hard task, not a statistical estimate of general ability. ox-alpha ran in OpenRouter's free stealth period ($0), so a cost comparison is not meaningful - we also show 'cost at Fable rates' to compare volume. Although we intended interactive pairing, both models in fact ran from a single prompt with no human follow-up, so this was a sealed run. The adversarial pass was run by hand against the code, not as a pre-sealed automated suite. The rubric scores are DevShift's assessment, not a blind review. ox-alpha ran through OpenRouter's stealth program. Unusually for a stealth release, its provider states that prompts and completions are not used for training (OpenCode's promo calls it zero data retention) - so, per that stated policy, the brief and generated code were not harvested.