# Hunting Zero Days with T3MP3ST and Pi

> T3MP3ST claims to beat XBOW on its own 104-challenge suite. I ported it to the Pi agent harness to run any model, cut the prompt bloat and start building my own evals. First run found three low-severity bugs for under a dollar.

- Published: 2026-08-24
- Author: Ron Jansen (https://rxj.dev/about/)
- Canonical: https://rxj.dev/posts/hunting-zero-days-with-pi/

---
Pliny Elders along with BT6 released T3MP3ST; a offensive security framework that lets you turns Claude Code or Codex in a skilled red teaming offensive security agent. BT6, the organization behind [self reported T3MP3ST outcompetes XBOW (which is valued at $1 billion) in XBOW's own 104-challenge suite](https://github.com/elder-plinius/T3MP3ST). This is a huge deal and I'm surprised this does not get more attention.

## Porting
Why porting something that already proved it worked? First of all because I did not get it to run it on my machine - I ran into various bugs I needed to resolve. Secondly, the models are limited. I want to use GLM 5.3, while currently out of the box T3MP3ST only supports GLM 4.8.

Secondly in my humble opinion I believe the tool is doing too much. It has an API, a MCP server, a web interface. Not only from a interface perspective also, on prompt-level there is too much bloat. When using the Codex or Claude Code as the backbone of the process, T3MP3ST uses a prompt-based contract to simulate a tool call schema on top of Claude/Codex' own tool schema. This results in lots of wasted tokens and bloating the context window unnecesarily.  

Last but not least, I wanted to truely grasp the architecture and components they created to make this work. I mean, if you beat unicorn status startup it is definitly worth studying. 

Porting it was relative straight forward once you understand how the agents are defined and configured and how they communicate. All I needed was to derive a specification from the existing implementation and let another agent implement the spec, but now based on the Pi harness. 

## Pi
I believe conceptually most agents are just folders, with a basic agent configuration, SKILLS, MCP configurations and a AGENT.md file. This is easy to reason about for anybody. 

Conceptually, what I want is a basic programmatic interface that can be controlled by another agent or a basic TUI. Pi has has abstracted and decoupled so many components away that you can you fully change the core ReAct loopt, without touching the TUI itself, or breaking its extension system or the AI/LLM adapters. In my opinion it comes closest to what an agent framework should be built like. You get a lot of functionality for free, yet you swap out any component you do not like. I trully believe much more agents should be build like on top of the Pi ecosystem. 

The benefits of this approach are plenty fold
- it's easier to reason about
- easier to change - although this can also be a trap in itself
- it unlocks not only using Claude Code with your subscription but you can now use ANY model or provider you want. 
- you can use it programmatically from the command line, or even connect it a desktop application using the Agent Client Protocol which Pi supports out of the box. 
- deployment to a remote microVM or container is just a single install away

## Benchmarking
The main challenge with porting to a completely different architecture is that you lose their entire track record of evals and benchmarks. I do not have the funds to run a 104 challenge suite, so I started small and will built up till I can properly test the entire suite and publish a benchmark. 

## Results
It's still an LLM that is generating vulnerabilities; so you definitly will get a false positives as well as false negatives. During recon for example any prioritized vulnerability should be taken with a grain of salt, until its validated to be a real vulnerability.

However, in a single run I found 3 low severity vulnerability. It costs me $0,82 on GLM 5.2 tokens. Unfortunatelly the guardrails for the Anthropic models are too strict.

While benchmarking is work in progress, the first port has been a great succes. The first results are promising. Next up is building the evals and climbin the hill to reach benchmarks scores beyond what XBOW self reports.