Auditing a Legal-Ops
Post-Training Loop

Abstract

This article documents a single applied post-training session that produced a narrow legal-operations contract-intake adapter from a base Qwen3 4B instruction model, moved it through targeted repair and holdout gates, wrapped it with runtime controls, and released it publicly on Hugging Face as narcolepticchicken/qwen3-4b-legal-ops-contract-intake-lora.

The work did not attempt to build a general legal reasoning model. The target behavior was deliberately smaller: source-grounded intake labels, strict extraction formats, insufficient-evidence handling, privacy-safe refusal, prompt-injection resistance, and legal-boundary escalation. The final artifact is a PEFT/LoRA adapter for Qwen/Qwen3-4B-Instruct-2507, with a companion production wrapper that enforces authentication, redacted logging, human review routing, monitoring events, and rollback checks.

What Was Built

The session produced four concrete classes of artifact:

The final adapter is the result of model selection and interpolation rather than a single monolithic run. The winning public adapter was derived from v85 and v86 repair adapters at interpolation alpha 0.35 from v86.

Research Question

The practical question was:

Can a small instruction model be made useful for a narrow legal-ops workflow by repeatedly repairing concrete failure modes, without broadening the claim beyond what gates and runtime controls prove?

The answer from this session is conditionally yes. The adapter passed the defined gates and the wrapper implements the expected internal release controls. The result remains bounded by synthetic data, local validation, and the explicit legal-ops intake scope.

Repair Loop

The initial objective was not to make the model “smarter” in a general sense. The objective was to remove specific legal-ops failure modes that made the adapter unsafe or unreliable for contract-intake work.

The repair targets were:

1. Privacy refusal for tokens, employee IDs, credentials, secrets, and sensitive placeholders.
2. Prompt-injection resistance when malicious instructions appear inside contract text.
3. Legal-boundary escalation for prompts such as “will we win,” “can we rely on this,” “should I sign,” and “did we waive.”
4. Risk-label calibration for high_risk, medium_risk, and low_risk.
5. Strict JSON and value-style repair for fields such as "cyber insurance", "$5 million", "ticket closure", and "99.5%".
6. Regression preservation across corrected proof, usability, privacy, and production-readiness gates.

Immediate Training Data

The final public adapter is an interpolation of v85 and v86. Those adapters were trained with supervised fine-tuning on synthetic repair rows generated from prior gate failures and preservation cases.

v85 concentrated on the failed prod_v9_boundary_003 family while preserving v1-v9, privacy, and usability cases. v86 made the same family denser and more literal after v85 did not move that row enough.

Category Mix

The row distribution reflects the observed failure pattern: most additional data was legal-boundary data, with risk and extraction repairs added to prevent regressions in adjacent areas.

Failure-Driven Construction

The session repeatedly used this pattern:

1. Run a gate.
2. Identify exact failed rows.
3. Generate narrow SFT repair examples around the failure family.
4. Train the next adapter.
5. Re-run known gates.
6. Introduce or preserve a fresh non-overlapping holdout.
7. Stop only when the adapter passed the defined matrix.

This is closer to a targeted repair loop than an open-ended RLHF program. The useful property was not the number of attempts; it was that every attempt was tied to a named failure class and a regression matrix.

Evaluation

The original release reported corrected proof, privacy, usability, and v1-v9 gate results. A later independent audit showed that the final training sets contained every scored v1-v9 prompt. The public model card and this article therefore treat those scores as regression evidence, not holdout evidence.

Baseline Comparison

The corrected v6 proof gate includes an explicit base-model comparison against Qwen/Qwen3-4B-Instruct-2507.

The adapter also reached category score 1.0 across clause classification, contract NLI, evidence extraction, insufficient evidence, legal boundary, and risk flag on that proof gate.

Internal Regression Gates

The adapter passed v1 through v9 after those cases were used in repair and preservation training. Each gate had 56 tasks and the selected adapter passed 56 of 56.

The v9 validator found no exact overlap with data through v83. It did not and could not establish independence from v84-v86: all 504 v1-v9 prompts appeared in the final v85 and v86 training sets. v9 was also a near-copy of v8, so exact-string checks overstated freshness.

Independent Replacement Gate

A new 351-task benchmark was sealed after the public adapter was frozen. It draws from LegalBench configurations absent from the seed builder and adds independent prompt-injection, legal-boundary, and missing-context cases. Nine source rows with long copied spans were removed before sealing. The final artifact has zero exact prompt overlap and zero substantial 13-word overlap against 161 historical training/evaluation files.

The paired GPU run evaluated the unmodified base and adapter with identical deterministic decoding. Promotion required an adapter pass rate of at least 85%, a paired-bootstrap 95% lower lift bound of at least two percentage points, exact McNemar p <= 0.05, a 70% category floor, and zero critical failures.

The paired lift was 9.69 points with bootstrap 95% interval 5.41-14.25 points, and exact McNemar p=0.000033. This is strong evidence that the adapter improves the bounded aggregate task. Promotion was still rejected because three critical prompt-injection cases failed and contract NLI regressed by 4.76 points.

Interpretation

The old gates show memorized or preserved behavior on known cases. Only the sealed paired comparison can address improvement over baseline, and even that bounded benchmark cannot prove broad legal competence or deployment readiness.

Runtime Wrapper Prototype

After the adapter passed the internal gate matrix, the session moved from model behavior to runtime behavior. The local prototype added a wrapper around generation rather than trusting the model by itself.

The service wrapper implements:

• GET /health
• GET /metrics
• POST /v1/legal-ops/intake
• POST /v1/legal-ops/reviews
• persistent redacted monitoring logs
• review queue and review state
• review updates
• metrics JSON
• rollback decision log
• backend generator injection or backend command support

The implementation lives in scripts/legal_ops_production_service.py, with reusable guardrails in scripts/legal_ops_production_controls.py.

Runtime Controls

Test Evidence

The final local test suite covered:

• anonymous access denial;
• redaction and blocking of secret-reveal requests;
• prompt-injection and legal-decision escalation;
• rollback on adapter SHA drift and privacy echo;
• service persistence of monitoring events, review state, and metrics;
• production signoff validation;
• adapter artifact validation.

The independent-review test run passed 17 tests after adding sealed-holdout and paired-statistics checks.

Historical Configuration State

The active local config marks the release as:

These values were self-declared by the local configuration. They are not evidence of a deployed production service. No hosted inference backend, verified authentication system, tenant-isolated storage, or real review queue was exercised end to end.

Public Release

The model was renamed from an internal lineage-style name to a public-facing descriptive name:

narcolepticchicken/qwen3-4b-legal-ops-contract-intake-lora

The public name encodes the important user-facing facts:

• Qwen3;
• 4B parameter base;
• legal-ops domain;
• contract-intake task;
• LoRA adapter format.

Hugging Face Artifacts

The public Hugging Face repository contains:

• adapter_model.safetensors
• adapter_config.json
• tokenizer files
• chat template
• complete model card
• examples/simple_inference.py
• examples/production_wrapper_minimal.py

The corrected model card documents training data, intended use, out-of-scope use, the contamination audit, bounded baseline comparison, limitations, and usage examples.

Artifact Index

Example Use

from peft import PeftConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

adapter_id = "narcolepticchicken/qwen3-4b-legal-ops-contract-intake-lora"
peft_config = PeftConfig.from_pretrained(adapter_id)
base_id = peft_config.base_model_name_or_path

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    low_cpu_mem_usage=True,
).to("cuda")

model = PeftModel.from_pretrained(base, adapter_id).to("cuda")

Example Prompt

For intake coding, choose the clause family for this passage:
production content remains in Ireland and Canada.
Choose one: data_residency, escrow, payment_terms.

Expected style:

data_residency

Usage Boundary

The release is an experimental artifact for testing bounded legal-ops intake behavior. The existing evidence does not justify production deployment. It is the wrong tool for broad legal advice, jurisdictional analysis, legal conclusions, or unsupervised contract approval.

Lessons

The session suggests four practical lessons.

1. Narrow failure classes are easier to repair than broad capability goals.

The productive unit of work was a named failure row or failure family. Privacy echo, prompt injection, legal-boundary overreach, risk-label drift, and JSON formatting each needed distinct repair examples and distinct gates.

2. Freshness must be checked against the final training corpus.

Known gates are regression tests. v9 was fresh only relative to data through v83; it was then copied into v84-v86 repair data. A gate is not a holdout if it is used for failure repair, preservation training, interpolation selection, or prompt tuning.

3. A model card is part of release engineering.

The first public name exposed internal training lineage. Renaming the model to qwen3-4b-legal-ops-contract-intake-lora made the artifact understandable. The model card then needed the same discipline as the model: clear scope, data provenance, eval results, examples, and limitations.

4. A wrapper design is not a production deployment.

The local wrapper implements useful redaction, review-state, monitoring, and rollback logic under unit-test assumptions. It does not provide verified authentication, tenant isolation, a deployed review queue, or an end-to-end run with the real model backend. Those are integration requirements, not labels that can be supplied by a configuration file.

Remaining Work

The next meaningful work would test this system with messier documents and a real reviewer workflow:

• OCR-corrupted contracts;
• multi-document conflict sets;
• longer agreements;
• real review queue integration;
• scheduled regression automation;
• latency and cost measurement under load;
• human correction-rate monitoring;
• a larger document-family-isolated holdout generated independently of all repair scripts.

The session demonstrates both targeted repair and how repeated repair can invalidate evaluation. The public artifact is experimental until independent evidence supports a stronger claim.