Trained to Betray: Poisoned Data and Checkpoints Redirect Coding Agents
Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.
Model PoisoningMalicious ModelsModel ProvenanceRuntime IsolationAgentic AI
Two points of entry: In one study, a small poisoned portion of agent-training data teaches a victim-fine-tuned model to select an unsafe execution tool. In Kerkoporta, the adversary controls checkpoint training and downstream adoption is the supply-chain premise. The studies should not be collapsed into one confirmed campaign.
Useful behavior can conceal a backdoor: Kerkoporta’s SFT implants trigger-conditioned structured calls; subsequent RL can restore normal coding performance without removing them. Triggers appear in otherwise ordinary task context, and schema-valid calls are accepted by the harness.
Results have boundaries: The mock-harness dataset result is roughly 78% on seen interfaces, versus 11% and 0% on two unseen interfaces. The checkpoint study separately measures transfer across repositories and harnesses. Configuring a harness with deliberately poisoned weights does not establish a flaw in its default model configuration. Supporting primary records: Kerkoporta paper.
The training owner should accept external trajectory datasets only from approved publisher identities and immutable manifests, verifying every staged file before training reads it. This blocks unauthorized substitution or an unapproved supplier; a trusted publisher signature still does not prove the examples are safe.
CI/CD Release Gating, Model Artifact Signing & Secure Distribution
High
Require approved model-release identity, exact artifact digests and a signed release record before a custom checkpoint can enter the agent runtime. Admission must include an independent behavioral review result, not merely the attacker’s signature or benchmark claims. Verify the artifact actually loaded by each worker.
Give the agent only task-required tool capabilities, file roots and backend credentials. Prefer bounded repository operations over an unrestricted shell when the task permits. Enforce restrictions in the dispatcher and OS/backend authority, including direct calls; renaming the shell tool or hiding it from a prompt does not constrain a model trained to discover another execution tool.
Execute untrusted-checkpoint tool effects inside a verified microVM or supported user-space-kernel sandbox with only the necessary repository snapshot and no production credentials or host sockets. Verify the actual runtime, not just a configured class name. Local changes can still damage the sandbox workspace, so preserve recoverable source versions.
Default-deny tool-runtime egress outside the model’s control, allowing only task-required destinations. Test raw-IP, alternate tool and proxy routes from the actual runtime. This constrains remote-script retrieval and external disclosure, while local deletion and harmful changes through an allowed service remain separate risks.
Once a checkpoint is confirmed unsafe, quarantine its exact digest in every model registry and loader admission path while retaining forensic bytes. Resolve mutable aliases to digests and verify that restarts and new workers cannot reload it. Active sessions need separate termination; registry quarantine alone does not unload a model already in memory.
Recover with an independently approved pre-incident model and matching serving/harness configuration from immutable backup. Verify the loaded digest and rerun task plus incident-trigger tests before resuming. Restoring only the same poisoned artifact under a new tag, or improving coding scores through more RL, is not recovery.
Once analysts identify poisoned trajectory rows, the dataset owner should authorize their exact IDs in a signed removal manifest bound to the immutable input and candidate report. Produce a new dataset and independently verify that only those rows were removed. Preserve the forensic source and deny reuse of the compromised version. Removing rows prevents their reuse; it does not remove a backdoor already learned by a model.
Training, Fine-Tuning & Evaluation Data Sanitization
Medium
When building agent-trajectory data, preserve independently verified observation origins, content findings and applicable reward/judge contracts. Quarantine unknown or flagged records before materializing the training snapshot. Review whether each assistant action follows the user’s request; syntactically valid tool calls are not evidence of safe supervision. The filter requires trustworthy content findings and does not discover a hidden backdoor on its own.
Bind the exact admitted dataset version to each training run and produced model. Read the lineage back under a separate identity and restore the recorded dataset in a clean workspace. If poisoned rows are later found, this identifies the derived checkpoints that require review and avoids treating every unrelated model as affected.
Evaluate the exact fine-tuned candidate against a trusted baseline using held-out tasks, trigger-bearing repository fixtures, paraphrases and the production harness/tool schemas. Record benign utility and unsafe effects separately and block candidates that violate the release policy. Tests must exercise actual tool proposals and permitted sandbox effects; normal benchmark scores cannot establish backdoor absence.
RL Behavioral Promotion Validation in Isolated Environments
Medium
After the checkpoint’s RL stage, rerun complete behavioral episodes in an isolated deployment-equivalent harness. Bind actions, side effects, rewards and task outcomes to the exact post-RL digest and block promotion on forbidden effects even when coding reward improves. Do not reuse the pre-RL safety result for the changed model.
AI Goal Integrity Monitoring & Deviation Detection
Medium
Observe each proposed tool action against the independently approved task, and flag unrelated remote execution, destructive changes or disclosure. Calibrate on benign repository workflows as well as trigger cases; treat model explanations as untrusted. The detector produces a finding and needs a separate enforcement or response consumer to stop an action.
If no trustworthy trained version can be restored, a team that owns its training workflow can create a fresh model from a separately trusted base and verified clean inputs. Bind the sanitized record population, code, initialization policy and resulting model to immutable manifests; independently prove what the training job consumed. This route applies only where the team can acquire and verify the actual output artifact, such as its self-managed Qwen workflow. Submitting another hosted fine-tuning job alone cannot satisfy that artifact-verification requirement. Before serving the replacement, an evaluator independent of training and deployment must test the exact recovered model with the declared clean-task population and incident-specific trigger/action cases. Use the intended tool schema and harmless execution fixtures; separately measure valid unsafe calls and actual effects. Bind thresholds, sample IDs, evaluator release and observation window to the same model digest. Keep a candidate blocked if coverage is incomplete or the incident behavior remains.
What Defenders Should Do Now
Separate supplier/provenance admission from record-content review. Preserve complete data-to-model lineage, and compare each training action with its user task rather than trusting valid JSON or a familiar publisher name.
Evaluate exact custom weights with the actual harness, tool schemas, held-out tasks and trigger variations. Repeat after RL; utility improvement is not evidence of safety recovery.
Enforce tool permissions and sandbox/egress boundaries outside the model, monitor unrelated actions, and prepare digest-specific quarantine plus verified known-good restoration. These measures still have value when the trigger itself looks benign.
Conclusion
AIDEFEND ↗ covers the full defender lifecycle here: data provenance and content admission, traceable training, model-release checks, post-training behavioral validation, tool containment, deviation detection, artifact eviction and trusted restoration. Each protects a different boundary; none certifies every unknown trigger absent.