Evaluation
We test whether AI agents choose fitting services more often with our directory than without it. Here are the results of each round so far, including the rounds where we made no difference.
How the agent tests work
An AI agent gets a task written the way agents ask for a service, for example a paid web search with no account, or an MCP server that reads academic papers without an API key. It chooses one service and says how to call it. Each task runs in two or three arms: with web search only, with web search and our directory (our MCP server), and, from round 3, with no tools at all. A second AI model judges each choice: whether the service does the job, whether it meets every constraint the task states, and whether it exists. The judge reads the service's own pages; it does not read our site.
The same small AI model, from one provider, plays the agent and the judge. Each run may use at most three web searches. A round costs about US$1.
Results by round
| Round | Arm | Fits | Partly fits | Does not fit or does not exist | Every constraint met | Median time per task |
|---|---|---|---|---|---|---|
| Pilot2026-10-19 · 8 tasks × 1 · judge lenient (any service that does the job) | Web only | 8 of 8 | 0 | 0 | 8 | – |
| Web and our directory | 8 of 8 | 0 | 0 | 8 | – | |
| Round 22026-10-19 · 8 tasks × 2 · judge strict (from here on) | Web only | 9 of 16 | 6 | 1 | 7 | – |
| Web and our directory | 10 of 16 | 6 | 0 | 8 | – | |
| Round 32026-10-19 · 9 tasks × 2 · judge strict | Web only | 8 of 18 | 8 | 1 | 4 | 13 s |
| Web and our directory | 7 of 18 | 10 | 1 | 7 | 21 s | |
| No tools (memory) | 1 of 18 | 9 | 8 | 1 | 5 s | |
| Round 42026-10-20 · 9 tasks × 2 · judge strict | Web only | 7 of 18 | 9 | 2 | 6 | 15 s |
| Web and our directory | 7 of 18 | 11 | 0 | 5 | 17 s | |
| No tools (memory) | 3 of 18 | 7 | 7 | 3 | 5 s |
What the numbers say
- With our directory, agents chose a fitting service about as often as with the web alone: 7 of 18 runs each way in rounds 3 and 4. At 18 runs an arm, a difference of two is within chance.
- In round 4, no run with our directory chose a service that does not exist; two runs with the web alone did.
- In round 4, a task with our directory took a median 2 seconds longer than with the web alone (8 seconds longer in round 3).
- With no tools, agents named a service that does not exist in 4 of 18 runs in round 3 and 6 of 18 in round 4.
- Of the fitting services agents found on the web alone, our search showed 0 of 8 in its first ten results in round 3, and 2 of 7 in round 4.
Fit benchmark
Between agent tests we run a cheaper check of our search alone. A set of needs, each written as an agent would ask, goes to our search, and a small AI model judges whether each of the first three results does the job (and, separately, whether it meets every constraint). The fixed set stays the same so runs compare; a new weekly set is written each week by an AI model that does not see our site, and is never used to tune search. Until 21 October about one judge reply in eight held no verdict and counted as a miss; the runs below count only real verdicts.
| Date | Set | Search | Needs judged | A fit in the first 3 | A fit meeting every constraint | |
|---|---|---|---|---|---|---|
| 2026-10-21 | fixed set, 59 needs | search/6 | 56 | 67.9% | 57.1% | needs judged in full |
| 2026-10-21 | fixed set, 59 needs | search/7 | 58 | 70.7% | 56.9% | needs judged in full |
| 2026-10-21 | weekly set 2026-W43 | search/7 | 9 | 22.2% | 11.1% | written that week, never used to tune search |
The first weekly set scored far below the fixed set. We read that as gaps in what we hold and in how search finds the right part of the tree, made larger by a stricter set; nine needs are too few to say more.
Limits
- Samples are small: 16 to 18 runs an arm.
- One AI model, from one provider, plays both agent and judge. Agents built on other models may choose differently.
- We wrote the agent-test tasks ourselves. The weekly benchmark sets are written by an AI model without access to our site.
- The judge is an AI model reading public pages. It can be wrong, and a service it cannot read counts against us.
Planned, not promised
The next agent-test round after the changes we are making to how search finds places and orders results. We will add it here, whatever it shows.
Dates are in UTC. How we check services: methodology. Something wrong here? Tell us.