Technical paper · SIFT
A Classifier That Teaches Itself.
A cost-tiered document classifier whose corpus grows from production traffic and whose autonomous retraining is made safe by a frozen evaluation gate.
- document classification
- continuous learning
- LLM-as-judge
- weak supervision
- model cascade
- active learning
- eval-gated promotion
- drift detection
A classifier that teaches itself
A cheap model answers most pages in milliseconds and sends only the unsure ones to an LLM judge. The judge’s answers become training data, and every retrained model has to pass a frozen gate before it goes live. The legal-NDA domain below is the paper’s illustrative example.
Pages flow through a cheap SPLADE and LightGBM classifier; confident ones get a label straight away, and the low-confidence minority escalates to an LLM judge. The judge’s verdicts are written back into the labeled corpus. A scheduler fires a retrain, the training kernel produces candidate v2, and the promote gate blocks it because NDA F1 dropped two points. After mislabeled pages are fixed, candidate v3 passes both the critical-label check and the frozen golden set, is promoted, and the latest pointer moves so serving uses it.
Core thesis
The bottleneck in production classification is not the model.It is the labeling project that must come before it and the fear of the retraining that must come after.Remove the first and tame the second.
Abstract
Document classification is a solved problem in the laboratory and an unsolved one in the enterprise. The blocker is rarely model architecture; it is the labeling project that must precede a model and the institutional fear of letting a model retrain itself once one exists.
We present SIFT (Self-Improving, Frozen-gate Training), the FlowX.AI Platform’s dynamic classifier service, which attacks both. SIFT serves classification from a deliberately cheap, CPU-bound pipeline, a SPLADE sparse encoder feeding a LightGBM head, and escalates only the low-confidence minority of pages to an LLM judge. The judge’s verdicts are written back into a labeled corpus, so the expensive model continuously teaches the cheap one: the escalation rate falls, the corpus grows from production traffic rather than from an up-front annotation effort, and accuracy compounds with use. Onboarding a new document family requires only a declarative bundle, label space, anchor phrases, and a judge glossary, not a labeling project.
The harder problem is safety: an autonomously retraining classifier can silently regress. SIFT resolves this with a two-part promote gate, a critical-label F1 regression check plus a frozen golden regression set the model is never trained on, either of which vetoes promotion. This turns retrain monthly without a human from reckless into routine.
We describe the architecture, the self-feeding corpus loop, the frozen-gate promotion mechanism, and an illustrative multi-domain deployment, and we discuss the economics of a classifier whose marginal labeling cost trends toward zero.
Part of the FlowX.AI paper series.
Each paper names a framework and shows it running in production: governance, reliability, memory and measurement, engineered rather than hoped for.