Benchmarking Indirect Prompt Injection Vulnerabilities and Mitigation Strategies in Tool-Calling Agent Trajectories

Team

Methodology

1. Build a lightweight Python execution sandbox with simulated tools: web search, file system reader, SQL database query, API webhook sender.
2. Create a benchmark dataset of 100 realistic agent tool-execution scenarios with hidden adversarial payloads in HTML comments, JSON fields, and natural text.
3. Evaluate Attack Success Rate (ASR) across multi-turn agent loops to measure safety mechanism degradation as context window length increases.
4. Test lightweight mitigation techniques: XML data-sandboxing in system prompts and secondary LLM guardrail sanitizers (GPT-4o-mini as guardrail).

Attack scenarios

• Data Exfiltration — getting the agent to read local files and send them to an external endpoint via a web request tool.
• Control Flow Hijacking — making the agent skip assigned sub-goals to execute the attacker's tasks.
• Privilege Escalation — forcing the agent to execute unauthorized administrative tool calls.

Evaluation metrics

• Attack Success Rate (ASR) — percentage of injections that successfully hijacked tool execution.
• Utility Retention Rate (URR) — whether adding guardrails breaks the agent's ability to solve normal benign tasks.
• Token / Latency Overhead — exact cost and latency added by mitigation layers.

Budget

$1,000 in OpenAI API credits — $400 for baseline vulnerability testing (2,000+ multi-turn loops), $400 for secondary guardrail testing, $200 for final sweeps and contingency.

Expected deliverables

• Public GitHub repository with evaluation harness, attack dataset, and mitigation scripts.
• Open benchmark dataset hosted on Hugging Face Datasets.
• Technical paper submitted to arXiv and/or international conference (AAAI, NeurIPS Workshop on AI Safety).