Insert hostile, misleading, or policy-changing content into data the model trains on, retrieves from, or stores as memory.
LLM04:2025
Data and Model Poisoning
Data and model poisoning introduces manipulated data or model artifacts that change the system's behavior. Training, fine-tuning and embedding pipelines all need integrity checks.
Step 01
Input
Step 02
Model
Step 03
Tool / Data
Step 04
Impact
What it is
An attacker changes data used to train, fine-tune or build retrieval context, or supplies a tampered model. The resulting errors or backdoors may appear only on particular inputs. Persistent memory and feedback can also carry poisoned content forward.
Why it matters
Poisoned data can degrade decisions quietly, steer agents toward unsafe actions, pollute customer answers, and create recurring failures that look like model quality problems.
Failure path
One way this failure can develop.
Use this example to locate the input, permission or application check that would interrupt the sequence in your system.
Cause the model to reuse that material in future conversations or workflows.
Exploit the poisoned behavior after it becomes part of normal system state.
Defenses
Controls worth checking.
Check where each control runs, who owns it and how you will test it again after a prompt, model or workflow change.
Classify source trust
Label training, fine-tuning, RAG, feedback, and memory sources by owner, trust level, freshness, and review status.
Gate ingestion
Scan and approve high-impact corpus changes, quarantine untrusted content, and maintain rollback paths for bad updates.
Monitor behavioral drift
Use regression suites and answer-quality checks to catch new unsafe behavior after data, model, or retrieval updates.
Signals and control gaps
- New instructions embedded inside retrieved content or long-term memory.
- Answer drift after dataset, embedding, fine-tuning, or corpus updates.
- User feedback loops that can directly rewrite future behavior.
Questions for your team
- Which data sources can change model behavior over time?
- Can untrusted users influence memory, RAG, or fine-tuning inputs?
- How quickly can a bad corpus or model update be rolled back?