Technical paper · ORNA
Agent Self-Adaptation in Production.
Closing the loop between observability, validation, and node-level refinement for LLM agents in regulated industries.
- agent self-improvement
- LLM orchestration
- production AI systems
- observability
- compliance
- BFSI
- governed AI
- multi-tier validation
One adaptation cycle, validated
ORNA watches production traces, finds the node behind a failure pattern, and changes that node’s configuration, never the model’s weights. Each change has to pass schema, regression and shadow checks before it goes live, and the old version stays one step away. The case below is the paper’s cross-border loan workflow, with its reported numbers.
A six-node loan classification workflow with a strip of 28 cross-border applications, 15 of them flagged over three weeks. The entity extraction node is diagnosed with an extraction miss caused by nested LLC structures, a recognizer extension adds four subsidiary patterns, and the candidate passes three validation tiers: schema, a 120-trace regression suite with F1 rising from 87.2% to 90.4% and violations from 3 to 0, and a 200-invocation shadow run with confidence up 3.1 points. It is promoted with the old version kept for rollback, and compliance flags drop to zero over the following two weeks.
Core thesis
An agent that is 95% accurate but unauditable is operationally unusable.An agent that is 87% accurate but fully auditable, with a validated path to 90%, is deployable.
Abstract
Large language model (LLM) based agents deployed in regulated industries face a persistent tension: they operate in dynamic environments with evolving compliance constraints, yet manual agent tuning is slow, expensive, and fragile. Existing self-improvement frameworks assume benign operating conditions and largely ignore the governance requirements of production-grade systems.
We present a framework for autonomous agent self-adaptation that closes the loop between structured observability, multi-tier validation, and node-level workflow refinement, without retraining or model weight updates.
Our approach monitors production trace patterns through an observability layer (Observatory), identifies failure signatures at the node level, and applies targeted adaptations (prompt updates, recognizer extensions, threshold adjustments) subject to a three-tier validation protocol before safe rollout.
We evaluate the framework on a cross-border loan classification use case with real regulatory compliance constraints, demonstrating measurable accuracy improvement (87% to 90%+) while reducing compliance violations from 15 to 0 in a single adaptation cycle. We discuss how the framework generalizes to KYC screening, fraud detection, and insurance claims processing, and we address the governance mechanisms (audit trails, rollback, human escalation thresholds) required for production deployment in regulated environments.
Part of the FlowX.AI paper series.
Each paper names a framework and shows it running in production: governance, reliability, memory and measurement, engineered rather than hoped for.