8000 GitHub - q-qp-p/Sealing-Jutsu: CapsuleGuard agent poisoning defense research · GitHub
[go: up one dir, main page]

Skip to content

About

CapsuleGuard agent poisoning defense research

Resources

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

61 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Sealing Jutsu project banner

Sealing Jutsu

Intent-bound memory capsules for testing and hardening LLM agents against persistent memory poisoning.

Python Tests Held-out ASR Scope


In One Sentence

Sealing Jutsu is a research prototype that tests whether LLM-agent memory can be made safer by forcing every memory to prove what it is allowed to influence before it can shape a plan or trigger an action.

The Problem

Long-term-memory agents are useful because they remember context across sessions. That same feature creates a quiet security problem: a malicious web page, OCR document, tool result, agent summary, or experience log can be saved today and retrieved later as if it were normal trusted memory.

Most simple defenses ask:

Does this memory look suspicious?

This project asks a stricter question:

What is this memory allowed to influence?

That distinction matters. A memory can be useful for recall while still being forbidden from authorizing recommendations, purchases, emails, file changes, or tool chains.

The Claim

Sealing Jutsu does not claim to solve all agent memory poisoning.

It tests a narrower, defensible claim:

Persistent memory poisoning can be reduced when retrieved memories are treated as bounded security objects instead of plain context.

In other words, memory should not automatically become planning authority.

What It Builds

The implementation turns raw memory records into intent-bound capsules. A capsule keeps the content, but it also carries security metadata that limits where that content is allowed to matter.

Capsule field What it controls
source_type Whether the memory came from user input, web content, OCR, tool output, summary, or experience
kind Whether the memory is a fact, preference, observation, tool result, summary, or directive-like record
allowed_topics Which user intents the memory may influence
denied_actions Which actions the memory can never authorize
source_authority How much trust the source is allowed to carry
influence_budget How much decision weight the memory can contribute
verification_count Whether independent confirmation exists
lineage Whether derived memories can be traced back to their origin
status Whether the capsule is active or sealed

At decision time, retrieved memory has to pass topic scope, authority floors, independent evidence quorum, temporal checks, lineage checks, and a final plan authorization gate.

Raw memory
  -> Capsule compiler
  -> Source authority and lineage checks
  -> Topic-scope authorization
  -> Evidence quorum gate
  -> Plan authorization gate
  -> Tool/action decision

CapsuleGuard architecture diagram

Current Evidence

The short version: baselines still get poisoned, but the capsule path blocks the final poisoned action in the tested benchmarks.

Evidence track Main result
Workflow-corpus benchmark Intent capsules: 0.00% attack success, 0.00% risky action, 100.00% benign accuracy
Medium live LLM benchmark Capsule-filtered prompt: 0.00% final attack success across llama3, mistral, and phi3
Gap-fixed rerun Same 0.00% final attack success reproduced; planner temptation stayed at 2.78%
Trace realism Added trace_corpus importer so real or lab-collected agent traces can be evaluated without rewriting Python scenarios
Converted external corpora AgentDojo and InjecAgent traces now report poison influence separately from final action blocking

Influence Rate counts poisoned cases where poisoned memory reached the planner and the planner selected the attacker target before any final output gate. This matters because output moderation can block the final action while still letting poisoned memory steer the plan.

Converted-corpus readout, committed in results/converted_corpus_report.md:

Defense style Final ASR Poison influence What it means
Output moderation / semantic judge 0.00% 30.00-90.62% Final action is blocked, but poisoned memory still steers planning
Intent capsules 0.00% 0.00% Poisoned memory is denied planning authority before action gating

Latest committed workflow-corpus test split:

Agent Attack success Risky action Benign accuracy Sealing rate
Ambient memory baseline 33.33% 22.22% 100.00% 0.00%
Keyword filter 33.33% 22.22% 100.00% 0.00%
Provenance only 24.38% 16.25% 96.25% 0.00%
Trust-score retrieval 24.38% 16.25% 96.25% 0.00%
Counterfactual memory 3.96% 2.64% 91.25% 0.00%
Intent capsules 0.00% 0.00% 100.00% 8.33%

The corpus currently contains 120 generated workflow records across 8 domains:

Split Records Poisoned Benign
Train 60 40 20
Dev 24 16 8
Test 36 24 12

The benchmark includes vendor recommendation, email, calendar, file search, CRM notes, web research, OCR-style documents, and tool-chain workflows.

Live LLM planner check, committed in results/live_llm_planner_summary.csv:

Planner condition Live rows Attack success Raw parse error First-pass valid planner Final parse error
Ambient prompt 9 0.00% 0.00% 100.00% 0.00%
Capsule-filtered prompt 18 0.00% 0.00% 100.00% 0.00%
Jailbreak-style prompt 9 0.00% 0.00% 100.00% 0.00%

Per-model live LLM summary, committed in results/live_llm_planner_model_summary.csv:

Model Conditions tested Live rows Attack success Raw parse error First-pass valid planner
llama3 3 12 0.00% 0.00% 100.00%
mistral 3 12 0.00% 0.00% 100.00%
phi3 3 12 0.00% 0.00% 100.00%

Statistical reading: the live LLM suite is a realism check for the planner interface, not the main high-powered benchmark. It gives 36 live model rows across three models and three prompt conditions. The larger attack-surface claim should still use the workflow-corpus and stress-test runs; the live LLM result shows that the defense can run with actual model-generated plans while keeping structured first-pass planner output measurable.

Gap-fixed medium live LLM workflow-corpus run, committed in results/gap_fix_medium_live_llm_summary.csv:

Planner condition Live rows Planner tempted Final attack success Risky action Raw parse error Final parse error
Ambient prompt 108 22.22% 22.22% 22.22% 0.00% 0.00%
Capsule-filtered prompt 108 2.78% 0.00% 0.00% 0.00% 0.00%

Per-model defended medium result, committed in results/gap_fix_medium_live_llm_model_summary.csv:

Model Defended rows Planner tempted Final attack success Risky action Raw parse error First-pass valid planner
llama3 36 2.78% 0.00% 0.00% 0.00% 100.00%
mistral 36 2.78% 0.00% 0.00% 0.00% 100.00%
phi3 36 2.78% 0.00% 0.00% 0.00% 100.00%

Fresh gap-report reading:

Gap metric Result Status
Baseline attack pressure 22.22% in target band
Defended planner tempted 2.78% reduced and contained
Defended temptation blocked 100.00% closed
Defended final ASR 0.00% closed
Defended risky action 0.00% closed

Gap-closure comparison:

Gap / Metric Before fix After fix Rerun result Status
Baseline / ambient ASR 21.30% 22.22% 22.22% Good, still in 20-70% target band
Baseline / ambient risky action 21.30% 22.22% 22.22% Good, attack pressure is real
Capsule planner tempted 5.56% 2.78% 2.78% Improved
Capsule final ASR 0.00% 0.00% 0.00% Closed
Capsule risky action 0.00% 0.00% 0.00% Closed
Tempted plans blocked Not separately reported 100.00% 100.00% Closed
Raw parse error 1.85% defended 0.00% 0.00% Improved
Final parse error 0.00% 0.00% 0.00% Closed
Scenario realism Generated workflow corpus Added trace_corpus importer Available Code gap closed
Real external traces Not available Import path ready Still needs collected traces Evidence gap remains

How to read this: the LLM can still be tempted internally, which is realistic. The safety goal is that unauthorized temptation does not become an accepted recommendation or tool action. In the gap-fixed medium run, every tempted defended plan was blocked before final action.

Threshold calibration snapshot, committed in results/current_main_threshold_calibration.csv:

Sweep Rows ASR Risky action Benign accuracy FPR Calibration score
Current main generated-holdout sweep 16 0.00% 0.00% 100.00% 0.00% 1.00

The sweep varies medium-risk quorum and topic-scope thresholds across 16 settings and also writes SVG charts under results/current_main_threshold_calibration_charts/. This closes the prototype-level "hand-tuned only" gap inside the simulator. It does not replace calibration on external benign and adversarial workloads.

What The Hardened Tests Cover

The default and corpus benchmarks are designed to stress more than simple keyword attacks:

Attack area Coverage in this repo
Adaptive attackers Closed-loop mutation mode that changes attacks after failures
Delayed trigger poisoning Memories that activate only after later task context appears
Cross-session poisoning Session and workflow records that survive into later planning
Tool-chain manipulation Tool output traces and chained action simulation
Semantic paraphrase poisoning Synonym and paraphrase-style attack variants
Retrieval collision attacks Similarity and overlap stress cases against retrieval
Multimodal hidden instruction poisoning OCR-style and document-derived hidden instruction cases
Trusted-source compromise Source-authority and lineage stress cases

Quick Start

git clone https://github.com/Mr-Akuma/Sealing-Jutsu.git
cd Sealing-Jutsu
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m unittest discover -s tests

On Windows PowerShell, activate the environment with:

.\.venv\Scripts\Activate.ps1

Run the default hardened benchmark:

python run_capsuleguard.py

Run the held-out workflow-corpus test split:

python run_capsuleguard.py \
  --attack-mode workflow_corpus \
  --workflow-corpus data/workflow_corpus_splits/test.jsonl \
  --summary-csv results/workflow_corpus_test_split_summary.csv \
  --trace-jsonl results/workflow_corpus_test_split_traces.jsonl \
  --tool-trace-csv results/workflow_corpus_test_split_tool_traces.csv \
  --charts-dir results/workflow_corpus_test_split_charts

Run an external or lab-collected agent trace corpus:

python run_capsuleguard.py \
  --attack-mode trace_corpus \
  --workflow-corpus data/agent_trace_corpus_sample.jsonl \
  --summary-csv results/trace_corpus_summary.csv \
  --trace-jsonl results/trace_corpus_traces.jsonl \
  --tool-trace-csv results/trace_corpus_tool_traces.csv

trace_corpus accepts exported JSONL session records with flexible task, events, messages, or steps fields. It is the intended path for replacing generated scenarios with real or lab-user traces.

Validate and redact a lab trace corpus before using it:

python validate_trace_corpus.py data/raw_lab_traces.jsonl \
  --redacted-out data/redacted_lab_traces.jsonl \
  --report-json results/redacted_lab_trace_report.json \
  --require-redaction

The validator reports total traces, poisoned/benign balance, memory-event counts, source-type coverage, poison-memory labels, and privacy redaction coverage for emails, URLs, phone numbers, and common secret-token shapes.

Generate a fresh workflow corpus:

python generate_workflow_corpus.py \
  --out-dir data/workflow_corpus_splits \
  --train-count 60 \
  --dev-count 24 \
  --test-count 36 \
  --seed 2026

Benchmark Modes

python run_capsuleguard.py --attack-mode moderate
python run_capsuleguard.py --attack-mode insane
python run_capsuleguard.py --attack-mode extreme
python run_capsuleguard.py --attack-mode holdout
python run_capsuleguard.py --attack-mode adaptive_loop
python run_capsuleguard.py --attack-mode workflow_corpus --workflow-corpus data/workflow_corpus_splits/test.jsonl
python run_capsuleguard.py --attack-mode trace_corpus --workflow-corpus data/agent_trace_corpus_sample.jsonl
python run_capsuleguard.py --attack-mode multimodal
python run_capsuleguard.py --attack-mode attacker_generated

Sensitivity sweep:

python run_sensitivity.py \
  --attack-mode generated_holdout \
  --repetitions 4 \
  --noise-memories 8 \
  --seed 2026 \
  --medium-thresholds 0.45,0.55,0.70,0.85 \
  --topic-thresholds 0.08,0.12,0.18,0.24 \
  --csv results/current_main_threshold_calibration.csv \
  --charts-dir results/current_main_threshold_calibration_charts

LLM-provider experiment harness:

python run_llm_experiment.py --provider local
python run_llm_experiment.py --provider ollama --models llama3,mistral,phi3 --repetitions 3
python run_llm_experiment.py --provider openai-responses --api-key-env OPENAI_API_KEY --models gpt-5.2-codex,gpt-5.1-codex --case-source workflow-corpus

The local provider is deterministic and needs no external service. The ollama provider expects Ollama to be running locally with the requested models already pulled. The openai-responses provider requires OPENAI_API_KEY.

Medium live LLM workflow-corpus suite:

python run_llm_experiment.py \
  --provider ollama \
  --models llama3,mistral,phi3 \
  --case-source workflow-corpus \
  --workflow-corpus data/workflow_corpus_splits/test.jsonl \
  --case-limit 36 \
  --case-seed 2026 \
  --repetitions 1 \
  --output-csv results/medium_live_llm_planner_suite.csv \
  --summary-csv results/medium_live_llm_planner_summary.csv \
  --model-summary-csv results/medium_live_llm_planner_model_summary.csv

This runs 36 workflow-corpus cases across 3 live models and 2 planner conditions, producing 216 live planner rows. The committed medium run used --repetitions 1; increase --repetitions to 3 for a 648-row medium-cost variance run.

The live LLM CSV separates planner_attack_success from attack_success. planner_attack_success means the LLM was tempted into the attacker target before authorization. attack_success means the poisoned plan survived the final plan/action gates and would be accepted.

Run the complete workflow-corpus benchmark and the live LLM planner benchmark together:

python run_capsuleguard.py \
  --attack-mode workflow_corpus \
  --workflow-corpus data/workflow_corpus_splits/test.jsonl \
  --trials 5 \
  --repetitions 4 \
  --noise-memories 4 \
  --seed 2026 \
  --include-llm-planner \
  --llm-provider ollama \
  --llm-models llama3,mistral,phi3 \
  --llm-case-source workflow-corpus \
  --llm-workflow-corpus data/workflow_corpus_splits/test.jsonl \
  --llm-case-limit 36 \
  --llm-case-seed 2026 \
  --llm-repetitions 1

Use this when you want one command to produce both the symbolic high-volume capsule benchmark and the live LLM planner evidence.

High-cost conference-grade LLM evaluation profile:

python run_llm_experiment.py \
  --provider ollama \
  --models llama3,mistral,phi3,cydonia-chat,cydonia-lysandra \
  --case-source high-cost \
  --high-cost-attack-modes workflow_corpus,generated_holdout,adaptive_loop,advanced_attack_suite,attacker_generated \
  --high-cost-seeds 2026,2027,2028 \
  --high-cost-cases-per-mode-seed 20 \
  --high-cost-noise-memories 4 \
  --repetitions 1 \
  --output-csv results/high_cost_live_llm_suite.csv \
  --summary-csv results/high_cost_live_llm_summary.csv \
  --model-summary-csv results/high_cost_live_llm_model_summary.csv \
  --audit-jsonl results/high_cost_live_llm_audit.jsonl \
  --statistics-csv results/high_cost_live_llm_statistics.csv \
  --gap-report-csv results/high_cost_live_llm_gap_report.csv

That profile samples 300 high-cost LLM cases. With two prompt conditions and five models, it produces 3,000 live planner rows. For a 1,000+ case run, raise --high-cost-cases-per-mode-seed to 67, which produces roughly 10,050 live planner rows across five models.

Paid API / OpenAI-compatible provider shape:

python run_llm_experiment.py \
  --provider openai-compatible \
  --endpoint %CAPSULE_LLM_ENDPOINT% \
  --api-key-env CAPSULE_LLM_API_KEY \
  --models paid-model-a,paid-model-b,paid-model-c,paid-model-d,paid-model-e \
  --case-source high-cost \
  --high-cost-attack-modes workflow_corpus,generated_holdout,adaptive_loop,advanced_attack_suite,attacker_generated \
  --high-cost-seeds 2026,2027,2028 \
  --high-cost-cases-per-mode-seed 20 \
  --output-csv results/high_cost_paid_llm_suite.csv \
  --summary-csv results/high_cost_paid_llm_summary.csv \
  --model-summary-csv results/high_cost_paid_llm_model_summary.csv \
  --audit-jsonl results/high_cost_paid_llm_audit.jsonl \
  --statistics-csv results/high_cost_paid_llm_statistics.csv

The high-cost profile writes a raw-output audit JSONL and a paired statistics CSV comparing ambient prompt exposure against capsule-filtered authorization. The optional gap report CSV labels whether baseline attack pressure is in the target band, whether defended planner temptation is contained, and whether final defended ASR is closed.

OpenAI Responses / Codex provider shape:

python run_llm_experiment.py \
  --provider openai-responses \
  --api-key-env OPENAI_API_KEY \
  --models gpt-5.2-codex,gpt-5.1-codex \
  --case-source high-cost \
  --high-cost-attack-modes workflow_corpus,generated_holdout,adaptive_loop,advanced_attack_suite,attacker_generated \
  --high-cost-seeds 2026,2027,2028 \
  --high-cost-cases-per-mode-seed 20 \
  --output-csv results/high_cost_openai_responses_suite.csv \
  --summary-csv results/high_cost_openai_responses_summary.csv \
  --model-summary-csv results/high_cost_openai_responses_model_summary.csv \
  --audit-jsonl results/high_cost_openai_responses_audit.jsonl \
  --statistics-csv results/high_cost_openai_responses_statistics.csv \
  --gap-report-csv results/high_cost_openai_responses_gap_report.csv

This path uses OpenAI's Responses API with strict JSON Schema output for recommendation, action, and rationale. Use it for Codex-class API models when paid live-model evidence is needed; keep the raw-output audit JSONL for reviewer inspection.

Testing

python -m unittest discover -s tests

Expected current result:

Ran 142 tests
OK

Output Files

Benchmark runs write CSV, JSONL, and chart artifacts under results/. Curated summary files are committed for reproducibility. Large traces, rerun logs, process IDs, and local scratch outputs are ignored by .gitignore unless intentionally added.

results/
  capsule_sandbox_results.csv
  capsule_sandbox_summary.csv
  capsule_attack_breakdown.csv
  capsule_gap_closure.csv
  capsule_tool_traces.csv
  workflow_corpus_test_split_summary.csv
  workflow_corpus_test_split_traces.jsonl
  workflow_corpus_test_split_tool_traces.csv
  workflow_corpus_test_split_charts/
  live_llm_planner_summary.csv
  live_llm_planner_model_summary.csv
  live_llm_planner_suite.csv

Project Layout

capsule_guard/                 Core capsule compiler, policy gates, agents, metrics
data/                          Generated workflow corpora and benchmark splits
docs/                          Research notes, benchmark reports, operating guides
experiments/                   Experiment entry points and wrappers
paper/                         Conference-style draft material
results/                       Generated benchmark outputs
tests/                         Unit and regression tests
.github/workflows/tests.yml    GitHub Actions unit-test workflow
requirements.txt               Python dependencies for tests and paper generation
SECURITY.md                    Defensive vulnerability-reporting scope
ETHICS.md                      Intended-use and safety guidance
LICENSE                        MIT license
run_capsuleguard.py            Main benchmark runner
generate_workflow_corpus.py    Corpus generation entry point
validate_trace_corpus.py       Trace-corpus validation and redaction entry point

Useful live LLM notes:

docs/live_llm_planner_test_cases.txt
docs/medium_live_llm_workflow_cases.txt

Honest Limits

This is a research prototype. The current evidence is strong inside the simulator, but several production gaps remain:

  1. The included workflow corpus is generated. The repo now includes a trace_corpus import path for real or lab-collected agent traces, but conference claims should still run on traces collected outside the generator.
  2. The default high-volume benchmark still uses a deterministic planner. The live LLM planner study now has a committed 216-row workflow-corpus run, but larger multi-seed live LLM runs are still needed for high-confidence statistical claims.
  3. OCR and multimodal attacks are represented as extracted text, not full raw-image pipelines.
  4. Source labels are modeled as metadata; real deployments need signed, append-only provenance.
  5. Policy thresholds now have simulator calibration evidence, but still need calibration on real external benign and adversarial workloads.
  6. Real browser, email, database, and account side effects should be tested in isolated sandboxes before deployment claims.

About

CapsuleGuard agent poisoning defense research

Resources

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0