Procurement Three-Way Match with AI Agents: POs, Delivery Notes and Invoices

In most procurement and accounts payable teams, the real bottleneck is no longer typing a supplier name or invoice total from a PDF header. Header extraction has been routine for years. Friction lives inside line-level three-way matching: reconciling an internal purchase order, a delivery slip photographed at the loading bay with handwritten notes, and a supplier tax invoice arriving weeks later with unfamiliar catalog codes and pack sizes.

Illustrative synthetic scenario: the examples and figures in this article describe architectural patterns for document intelligence systems and do not represent a specific customer deployment, live model output or accounting advice.

Why Traditional OCR Breaks on Real Procurement Packets

In demos, purchase orders, delivery notes and invoices look like clean, single-language tables with identical columns. In practice, each document comes from a separate system. The purchase order comes out of the buyer's ERP using internal item codes. The delivery note prints at a supplier warehouse, collects rubber stamps and pen edits on the dock, and arrives as an angled phone photo. The tax invoice follows later as a bilingual PDF mixing right-to-left descriptions with left-to-right Latin part numbers and dimensions on the same row.

Flat text extraction without visual layout awareness scrambles columns: pack-size digits merge into catalog strings, or handwritten shortage notes drop outside table boundaries. Moreover, documents rarely arrive together. Orders stay open for weeks, partial delivery slips arrive per truck, and monthly invoices consolidate several drops. Reliable automation requires stateful orchestration through business workflow agents tracking open balances over time rather than a single stateless prompt.

Separating Document Roles Before Comparing Quantities

Sound reconciliation preserves the boundary between three distinct statements: the purchase order records what was authorized; the signed delivery slip records what physically crossed the dock; and the tax invoice records what the supplier seeks to collect. Summarizing a packet in prose blurs those roles and risks treating ordered units as proof of receipt.

Line-Item Resolution: SKU Mapping and Unit Normalization

Consider a synthetic scenario where a manufacturer orders sealing gaskets under purchase order PO-4820. Line three authorizes 24 master cartons at 10 gaskets per carton, totaling 240 base units. Four days later, the dock uploads a photographed delivery slip: the clerk crossed out 24, wrote 22 cartons received beside the stamp, and noted two cartons on backorder. At month-end, the supplier sends a PDF tax invoice listing its English catalog code instead of the buyer's SKU and billing for 240 individual units.

Simple string matching fails on three fronts: item codes differ, units shift from cartons to pieces, and billed units exceed physical delivery. To align lines without risky model guesses, the pipeline splits resolution into three governed stages:

• Approved supplier-to-internal SKU crosswalks: The agent resolves supplier catalog numbers, barcodes and dimensions against verified ERP crosswalk tables. Unmapped items surface as candidate pairings requiring buyer approval before reuse.
• Deterministic unit-of-measure conversion: Deterministic code converts cartons, pallets and rolls into a canonical base unit using catalog multipliers. When a document introduces an unknown pack size, the workflow pauses for clarification instead of guessing.
• Cumulative partial-delivery tracking: Each check compares open order balances against prior delivery slips and posted invoices so a second invoice never reuses an already consumed receipt.

Bounding-Box Evidence Links Instead of Bare Status Badges

Finance controllers rightly distrust tools that output a bare checkmark or generic alert on line four. When extracted rows appear detached from source pages, reviewers reopen all three files and scan every line manually anyway.

In an evidence-grounded pipeline, every field retains a pointer to its exact source: file ID, page number and normalized bounding-box coordinates around the cell, stamp or note. Selecting a disputed quantity displays the cropped dock photo beside the matching rows on the invoice and order.

Detecting Handwritten Edits and Receiving Dock Stamps

Pairing multimodal visual reading with structured extraction addresses a classic OCR blind spot: dockside pen overrides. When a printed 24 on a delivery slip has a pen strike-through and a handwritten 22 beside the stamp, visual inspection detects competing values, records both, lowers confidence and frames the signature crop for review.

Illustrative decision checkpoint: the signed delivery slip shows 22 cartons while the invoice bills 240 units

ERP voucher posting pauses automatically, and the 20-unit shortage routes to review with all three visual crops attached.

After deterministic conversion translates 24 ordered cartons (10 units per carton) into 240 base units, the signed delivery slip supports only 220 received units, leaving an unresolved 20-unit gap. Even when unit rates and payment terms match the order, the workflow agent holds full payment approval, keeps verified commercial checks intact, attaches the three bounding-box crops and routes an exception task to receiving and accounts payable to await a supplementary delivery slip or credit note.

Deterministic Tolerance Rules vs Mandatory Human Review

A costly mistake in financial AI is asking a generative model inside its prompt whether a discrepancy is small enough to ignore. Language models excel at parsing noisy layouts, aligning bilingual descriptions and flagging visual edits, but they must never act as the accounting rule engine. Arithmetic checks, tax ID lookups, open-balance math and tolerance bands belong strictly in deterministic, tested code.

Architectural boundary: The model reads layouts and proposes line pairings, but never decides whether a financial or quantity variance is acceptable. Vendor tax ID checks, duplicate invoice gates, pack-unit math and rounding tolerances run strictly in tested code.

Keeping extraction probabilistic and reconciliation deterministic lets finance teams enforce auditable control across four gates before general ledger posting:

1. Header identity and duplicate gates: Verify that the supplier tax ID matches the active vendor master, the invoice number has not been staged or posted previously, and currency and payment terms align before checking lines.
2. Rounding policy vs quantity or rate variances: Deterministic rules separate minor decimal rounding allowed by finance policy from physical shortages, unapproved SKU substitutions or unit-rate increases.
3. Role-specific exception routing: Physical receiving gaps route to warehouse leads with the signed dock photo crop, whereas pricing or contract variances route to the buyer managing the supplier agreement.
4. Staged ERP drafts with full lineage: Cleanly matched packets write staged draft vouchers with source links, while exceptions require recorded human sign-off and supporting evidence before ERP commitment.

Phasing a Rollout Without Disrupting Month-End Close

When scoping custom AI agent engineering for business operations, the primary target is zero false approvals on unsubstantiated invoices rather than maximum day-one automation. Responsible teams launch three-way matching in shadow mode against historical closed packets where the human accounting outcome is already documented.

Four practical steps keep rollout predictable across procurement and finance:

• Start with high-volume repeat suppliers: Onboard the vendor group generating most recurring line items first, locking down SKU crosswalks and pack multipliers cleanly.
• Test against messy dockside samples: Evaluate angled phone photos, stamped carbon copies, pen corrections, split shipments and multi-page consolidated invoices upfront.
• Define clear sign-off authority per exception: Specify who closes short-shipped receipts, who updates master pack ratios, and what proof must attach before a paused voucher resumes.
• Retain an immutable field-level audit trail: Store every extracted value, its source bounding box, the evaluated rule version and the identity and timestamp of every human decision.