Human-in-the-loop invoice classification

muhasip.ai — turning a week of monthly bookkeeping into a review queue an accountant can actually trust.

The problem

Every month, an accountant in Turkey sits down with a stack of purchase invoices. For each line on each invoice — the paper clip, the fuel, the office rent — they decide which expense category it belongs to, then enter it into the books. For a firm with about thirty clients, that work adds up to roughly a week of the month. Skilled, repetitive, and exactly the kind of work software should help with.

It is also exactly the kind of work that gets accountants into trouble. A miscategorised expense can trigger a tax correction. An expense the accountant never saw but which was accepted on their behalf is worse: they signed the books. So the obvious product idea — send the invoices to a language model, let it classify them, done — fails not on accuracy but on trust. Nobody wants a black box touching their clients' books.

muhasip.ai is my attempt at the version that survives that constraint.

The one rule everything else follows from

Before writing code I wrote a specification: 55 numbered requirements, each with an identifier, traced afterwards through the code, the tests and the decision log. Rule 32 is the one that shaped the product: nothing is ever auto-accepted. The system suggests, the accountant approves or corrects, and the books only ever contain accountant-approved categories.

Everything else is a consequence of that rule. If a human approves every line, reviewing has to be fast, which means the system must be right often enough to matter and explain itself well enough to check in a second. A confidence score of "0.87" from a language model does not meet that bar. "Eight of the ten most similar lines you approved for this client were category X" does.

How it works

• Extraction and routing (n8n). A client emails a PDF invoice to an address belonging to their firm. The workflow extracts it, rejects unreadable files, identifies which client it belongs to, and splits it into line items.
• Suggestion (kNN). For each line, the system finds this accountant's own previously approved lines that look similar, filtered to the same industry context. A weighted vote across those neighbours produces a candidate category and a confidence figure derived from vote share.
• The model fills the gaps. When the history is thin or the vote is split, a language model chooses from the defined category list, given the client's industry context and the nearest matches. It is told to prefer what this accountant has done before.
• Review. The accountant sees each line with its suggestion, its confidence, and the approved examples behind it. Approve, correct, or skip.
• Learning. Approved lines become training data for that accountant's future suggestions. Corrections are the most valuable examples of all.

Four decisions worth explaining

1. Confidence comes from the vote, not from the model

Models are confidently wrong. A vote across known-similar examples has a property a model's self-report lacks: it is auditable. If six of the nearest neighbours say "office supplies" and three say "equipment", the accountant can look at the nine lines that voted and decide in seconds. It also means the system improves immediately as more lines are approved — no retraining, no waiting for a model update.

2. Learning is per accountant, never shared

Each firm's approvals train only that firm's suggestions. Sharing would have made the system smarter faster, and would also have leaked one firm's commercial arrangements into another's suggestions. Isolation is stricter than most SaaS defaults: separate storage per firm, and no cross-firm queries in the similarity search.

3. The model only sees what it needs

Before anything reaches a language model, tax identifiers and addresses are removed, and they never enter the training data. Classification needs the item description, the amount and the industry context — not who the invoice belongs to. Token cost is tracked per accountant, because in a product with thin margins, per-tenant unit economics are a design requirement rather than an afterthought.

4. The application does not call the model

The .NET API never talks to a language model. Orchestration and the AI calls live in n8n workflows, stored as source and deployed by the same pipeline as the code. That keeps the API narrow and testable, and it means the model or provider can change without touching the domain logic. Lock-in is a design choice, not a fate.

Building it with agents, without losing control

The specification was written on 20 October 2026. The first commit is from the next day. Ten days later there were 71 commits and roughly 17,000 lines of C# and TypeScript. Most of that code was written by AI coding agents. My work was the specification, the architecture, the boundaries — and the review. Two moments show what that division of labour actually requires.

The API keys

An agent generated API keys and stored them in environment variables. It worked, which is the problem: no rotation, no per-user identity, no roles, no audit trail. I stopped it and moved authentication to Keycloak, with distinct roles for administrators, accountants and the automation service. An agent optimises for the shortest path to working code. Someone has to hold the line on the architecture, and that someone reviews everything.

The duplicated logic

An agent began rebuilding in a workflow what already existed in the application. I had deliberately given it limited access to the live automation instance; when I caught the duplication I made it change the workflow source in the repository instead, so the change shipped through the pipeline like everything else. One source of truth, least privilege for the automation, no direct edits in production. Treating an agent like a fast junior engineer with production access — supervise, bound, review — turned out to be the most useful mental model I found.

Stack

.NET 10 modular monolith with a schema per domain · n8n for orchestration and AI calls · embeddings (bge-m3) over pgvector for similarity · a language model for the hard cases · React and TypeScript front end · Keycloak for identity and roles · object storage per firm · PostgreSQL · GitHub Actions building to a private registry and deploying through Coolify to infrastructure I run myself · integration tests against a real database, not mocks.

Where it is now

Pre-pilot. The first demonstration with an accounting firm is scheduled in the coming weeks, and nothing has been auto-accepted into anyone's books, by design. The engineering question here was never "can a model classify an invoice". It was "will an accountant trust it enough to use it every month". That is the question the pilot answers.