1499 lines
46 KiB
Markdown
1499 lines
46 KiB
Markdown
# Web Search Tools Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Add modular `web_search` and `web_fetch` pi tools backed by Exa, with runtime config loaded from `~/.pi/agent/web-search.json` and an internal provider adapter layer for future providers.
|
|
|
|
**Architecture:** Build a package-style extension in `.pi/agent/extensions/web-search/` so it can carry `exa-js` and local tests cleanly. Keep config parsing, Exa request mapping, and output formatting in pure modules covered by `node:test` via `tsx`, then layer pi-specific tool registration and renderers on top.
|
|
|
|
**Tech Stack:** Pi extensions API, `exa-js`, `@sinclair/typebox`, `@mariozechner/pi-coding-agent`, `@mariozechner/pi-tui`, Node `node:test`, `tsx`, TypeScript
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/package.json`
|
|
- Local extension package manifest, npm scripts, and dependencies.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/index.ts`
|
|
- Extension entrypoint that registers `web_search` and `web_fetch`.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/schema.ts`
|
|
- TypeBox schemas for config and tool parameter validation.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.ts`
|
|
- Loads and validates `~/.pi/agent/web-search.json`, normalizes the provider list, and resolves the default provider.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.test.ts`
|
|
- Tests config parsing, normalization, and helpful error messages.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/types.ts`
|
|
- Provider-agnostic request/response interfaces used by both tools.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.ts`
|
|
- Exa-backed implementation of the provider interface.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.test.ts`
|
|
- Tests generic request → Exa call mapping and per-URL fetch failure handling.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.ts`
|
|
- Shared compact text formatting and truncation helpers.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.test.ts`
|
|
- Tests metadata-only search formatting, text truncation, and batch failure formatting.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.ts`
|
|
- `web_search` tool factory, execute path, and tool renderers.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.test.ts`
|
|
- Tests non-empty query validation and happy-path execution with a fake provider.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.ts`
|
|
- `web_fetch` tool factory, input normalization, execute path, and tool renderers.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.test.ts`
|
|
- Tests URL normalization, default text mode, and invalid URL rejection.
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/extension.test.ts`
|
|
- Smoke test that the extension entrypoint registers both tool names.
|
|
- Modify: `/home/alex/dotfiles/docs/superpowers/plans/2026-04-09-web-search-tools.md`
|
|
- Check boxes as work progresses.
|
|
|
|
## Environment Notes
|
|
|
|
- `/home/alex/.pi/agent/extensions` resolves to `/home/alex/dotfiles/.pi/agent/extensions`, so editing files in the repo updates the live global pi extension path.
|
|
- The config file for this feature lives outside the repo at `~/.pi/agent/web-search.json`.
|
|
- Use `/reload` inside pi after file edits to reload the extension.
|
|
- Run package-local tests from `/home/alex/dotfiles/.pi/agent/extensions/web-search`.
|
|
|
|
### Task 1: Scaffold the extension package and build the config loader first
|
|
|
|
**Files:**
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/package.json`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.test.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/schema.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.test.ts`
|
|
|
|
- [x] **Step 1: Create the extension package scaffold and install dependencies**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/package.json` with this content:
|
|
|
|
```json
|
|
{
|
|
"name": "pi-web-search-extension",
|
|
"private": true,
|
|
"version": "0.0.0",
|
|
"type": "module",
|
|
"scripts": {
|
|
"test": "tsx --test src/*.test.ts src/**/*.test.ts"
|
|
},
|
|
"pi": {
|
|
"extensions": [
|
|
"./index.ts"
|
|
]
|
|
},
|
|
"dependencies": {
|
|
"@sinclair/typebox": "^0.34.49",
|
|
"exa-js": "^2.11.0"
|
|
},
|
|
"devDependencies": {
|
|
"@mariozechner/pi-coding-agent": "^0.66.1",
|
|
"@mariozechner/pi-tui": "^0.66.1",
|
|
"@types/node": "^25.5.2",
|
|
"tsx": "^4.21.0",
|
|
"typescript": "^6.0.2"
|
|
}
|
|
}
|
|
```
|
|
|
|
Then run:
|
|
|
|
```bash
|
|
mkdir -p /home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers
|
|
mkdir -p /home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npm install
|
|
```
|
|
|
|
Expected: `node_modules/` is created and `npm install` exits with code `0`.
|
|
|
|
- [x] **Step 2: Write the failing config tests**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { mkdtemp, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { loadWebSearchConfig, WebSearchConfigError } from "./config.ts";
|
|
|
|
async function writeTempConfig(contents: unknown) {
|
|
const dir = await mkdtemp(join(tmpdir(), "pi-web-search-config-"));
|
|
const file = join(dir, "web-search.json");
|
|
const body = typeof contents === "string" ? contents : JSON.stringify(contents, null, 2);
|
|
await writeFile(file, body, "utf8");
|
|
return file;
|
|
}
|
|
|
|
test("loadWebSearchConfig returns a normalized default provider and provider lookup", async () => {
|
|
const file = await writeTempConfig({
|
|
defaultProvider: "exa-main",
|
|
providers: [
|
|
{
|
|
name: "exa-main",
|
|
type: "exa",
|
|
apiKey: "exa-test-key",
|
|
options: {
|
|
defaultSearchLimit: 7,
|
|
defaultFetchTextMaxCharacters: 9000
|
|
}
|
|
}
|
|
]
|
|
});
|
|
|
|
const config = await loadWebSearchConfig(file);
|
|
|
|
assert.equal(config.defaultProviderName, "exa-main");
|
|
assert.equal(config.defaultProvider.name, "exa-main");
|
|
assert.equal(config.providersByName.get("exa-main")?.apiKey, "exa-test-key");
|
|
assert.equal(config.providers[0]?.options?.defaultSearchLimit, 7);
|
|
});
|
|
|
|
test("loadWebSearchConfig rejects a missing default provider target", async () => {
|
|
const file = await writeTempConfig({
|
|
defaultProvider: "missing",
|
|
providers: [
|
|
{
|
|
name: "exa-main",
|
|
type: "exa",
|
|
apiKey: "exa-test-key"
|
|
}
|
|
]
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => loadWebSearchConfig(file),
|
|
(error) =>
|
|
error instanceof WebSearchConfigError &&
|
|
/defaultProvider \"missing\"/.test(error.message),
|
|
);
|
|
});
|
|
|
|
test("loadWebSearchConfig rejects a missing file with a helpful example message", async () => {
|
|
const file = join(tmpdir(), "pi-web-search-does-not-exist.json");
|
|
|
|
await assert.rejects(
|
|
() => loadWebSearchConfig(file),
|
|
(error) =>
|
|
error instanceof WebSearchConfigError &&
|
|
error.message.includes(file) &&
|
|
error.message.includes('"defaultProvider"') &&
|
|
error.message.includes('"providers"'),
|
|
);
|
|
});
|
|
```
|
|
|
|
- [x] **Step 3: Run the config tests to verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/config.test.ts
|
|
```
|
|
|
|
Expected: `FAIL` with an `ERR_MODULE_NOT_FOUND` error for `./config.ts`.
|
|
|
|
- [x] **Step 4: Write the minimal schema and config loader implementation**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/schema.ts` with this content:
|
|
|
|
```ts
|
|
import { Type, type Static } from "@sinclair/typebox";
|
|
|
|
export const ProviderOptionsSchema = Type.Object({
|
|
defaultSearchLimit: Type.Optional(Type.Integer({ minimum: 1 })),
|
|
defaultFetchTextMaxCharacters: Type.Optional(Type.Integer({ minimum: 1 })),
|
|
defaultFetchHighlightsMaxCharacters: Type.Optional(Type.Integer({ minimum: 1 })),
|
|
});
|
|
|
|
export const ExaProviderConfigSchema = Type.Object({
|
|
name: Type.String({ minLength: 1 }),
|
|
type: Type.Literal("exa"),
|
|
apiKey: Type.String({ minLength: 1 }),
|
|
options: Type.Optional(ProviderOptionsSchema),
|
|
});
|
|
|
|
export const WebSearchConfigSchema = Type.Object({
|
|
defaultProvider: Type.String({ minLength: 1 }),
|
|
providers: Type.Array(ExaProviderConfigSchema, { minItems: 1 }),
|
|
});
|
|
|
|
export const WebSearchParamsSchema = Type.Object({
|
|
query: Type.String({ minLength: 1, description: "Search query" }),
|
|
limit: Type.Optional(Type.Integer({ minimum: 1, maximum: 25 })),
|
|
includeDomains: Type.Optional(Type.Array(Type.String())),
|
|
excludeDomains: Type.Optional(Type.Array(Type.String())),
|
|
startPublishedDate: Type.Optional(Type.String()),
|
|
endPublishedDate: Type.Optional(Type.String()),
|
|
category: Type.Optional(Type.String()),
|
|
provider: Type.Optional(Type.String()),
|
|
});
|
|
|
|
export const WebFetchParamsSchema = Type.Object({
|
|
urls: Type.Array(Type.String(), { minItems: 1 }),
|
|
text: Type.Optional(Type.Boolean()),
|
|
highlights: Type.Optional(Type.Boolean()),
|
|
summary: Type.Optional(Type.Boolean()),
|
|
textMaxCharacters: Type.Optional(Type.Integer({ minimum: 1 })),
|
|
provider: Type.Optional(Type.String()),
|
|
});
|
|
|
|
export type ProviderOptions = Static<typeof ProviderOptionsSchema>;
|
|
export type ExaProviderConfig = Static<typeof ExaProviderConfigSchema>;
|
|
export type WebSearchConfig = Static<typeof WebSearchConfigSchema>;
|
|
export type WebSearchParams = Static<typeof WebSearchParamsSchema>;
|
|
export type WebFetchParams = Static<typeof WebFetchParamsSchema>;
|
|
```
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/config.ts` with this content:
|
|
|
|
```ts
|
|
import { readFile } from "node:fs/promises";
|
|
import { homedir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { Value } from "@sinclair/typebox/value";
|
|
import { WebSearchConfigSchema, type ExaProviderConfig, type WebSearchConfig } from "./schema.ts";
|
|
|
|
export interface ResolvedWebSearchConfig {
|
|
path: string;
|
|
defaultProviderName: string;
|
|
defaultProvider: ExaProviderConfig;
|
|
providers: ExaProviderConfig[];
|
|
providersByName: Map<string, ExaProviderConfig>;
|
|
}
|
|
|
|
export class WebSearchConfigError extends Error {
|
|
constructor(message: string) {
|
|
super(message);
|
|
this.name = "WebSearchConfigError";
|
|
}
|
|
}
|
|
|
|
export function getDefaultWebSearchConfigPath() {
|
|
return join(homedir(), ".pi", "agent", "web-search.json");
|
|
}
|
|
|
|
function exampleConfigSnippet() {
|
|
return JSON.stringify(
|
|
{
|
|
defaultProvider: "exa-main",
|
|
providers: [
|
|
{
|
|
name: "exa-main",
|
|
type: "exa",
|
|
apiKey: "exa_..."
|
|
}
|
|
]
|
|
},
|
|
null,
|
|
2,
|
|
);
|
|
}
|
|
|
|
export function normalizeWebSearchConfig(config: WebSearchConfig, path: string): ResolvedWebSearchConfig {
|
|
const providersByName = new Map<string, ExaProviderConfig>();
|
|
|
|
for (const provider of config.providers) {
|
|
if (!provider.apiKey.trim()) {
|
|
throw new WebSearchConfigError(`Provider \"${provider.name}\" in ${path} is missing a literal apiKey.`);
|
|
}
|
|
if (providersByName.has(provider.name)) {
|
|
throw new WebSearchConfigError(`Duplicate provider name \"${provider.name}\" in ${path}.`);
|
|
}
|
|
providersByName.set(provider.name, provider);
|
|
}
|
|
|
|
const defaultProvider = providersByName.get(config.defaultProvider);
|
|
if (!defaultProvider) {
|
|
throw new WebSearchConfigError(
|
|
`defaultProvider \"${config.defaultProvider}\" does not match any configured provider in ${path}.`,
|
|
);
|
|
}
|
|
|
|
return {
|
|
path,
|
|
defaultProviderName: config.defaultProvider,
|
|
defaultProvider,
|
|
providers: [...providersByName.values()],
|
|
providersByName,
|
|
};
|
|
}
|
|
|
|
export async function loadWebSearchConfig(path = getDefaultWebSearchConfigPath()) {
|
|
let raw: string;
|
|
try {
|
|
raw = await readFile(path, "utf8");
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
|
|
throw new WebSearchConfigError(
|
|
`Missing web-search config at ${path}.\nCreate ${path} with contents like:\n${exampleConfigSnippet()}`,
|
|
);
|
|
}
|
|
throw error;
|
|
}
|
|
|
|
let parsed: unknown;
|
|
try {
|
|
parsed = JSON.parse(raw);
|
|
} catch (error) {
|
|
throw new WebSearchConfigError(`Invalid JSON in ${path}: ${(error as Error).message}`);
|
|
}
|
|
|
|
if (!Value.Check(WebSearchConfigSchema, parsed)) {
|
|
const [firstError] = [...Value.Errors(WebSearchConfigSchema, parsed)];
|
|
throw new WebSearchConfigError(
|
|
`Invalid web-search config at ${path}: ${firstError?.path ?? "/"} ${firstError?.message ?? "failed validation"}`,
|
|
);
|
|
}
|
|
|
|
return normalizeWebSearchConfig(parsed as WebSearchConfig, path);
|
|
}
|
|
```
|
|
|
|
- [x] **Step 5: Run the config tests to verify they pass**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/config.test.ts
|
|
```
|
|
|
|
Expected: `PASS` for all three tests.
|
|
|
|
- [x] **Step 6: Commit the config layer**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
git -C /home/alex/dotfiles add \
|
|
.pi/agent/extensions/web-search/package.json \
|
|
.pi/agent/extensions/web-search/src/schema.ts \
|
|
.pi/agent/extensions/web-search/src/config.ts \
|
|
.pi/agent/extensions/web-search/src/config.test.ts \
|
|
docs/superpowers/plans/2026-04-09-web-search-tools.md
|
|
|
|
git -C /home/alex/dotfiles commit -m "test: add web search config loader"
|
|
```
|
|
|
|
### Task 2: Add the provider-agnostic types and the Exa adapter
|
|
|
|
**Files:**
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.test.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/types.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.test.ts`
|
|
|
|
- [x] **Step 1: Write the failing Exa adapter tests**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { createExaProvider } from "./exa.ts";
|
|
|
|
const baseConfig = {
|
|
name: "exa-main",
|
|
type: "exa" as const,
|
|
apiKey: "exa-test-key",
|
|
options: {
|
|
defaultSearchLimit: 7,
|
|
defaultFetchTextMaxCharacters: 9000,
|
|
defaultFetchHighlightsMaxCharacters: 1200,
|
|
},
|
|
};
|
|
|
|
test("createExaProvider maps generic search requests to Exa search with contents disabled", async () => {
|
|
let captured: { query: string; options: Record<string, unknown> } | undefined;
|
|
|
|
const provider = createExaProvider(baseConfig, () => ({
|
|
async search(query, options) {
|
|
captured = { query, options };
|
|
return {
|
|
requestId: "req-search-1",
|
|
searchTime: 123,
|
|
results: [
|
|
{
|
|
id: "doc-1",
|
|
title: "Exa Docs",
|
|
url: "https://exa.ai/docs",
|
|
publishedDate: "2026-04-09",
|
|
author: "Exa",
|
|
score: 0.98,
|
|
},
|
|
],
|
|
};
|
|
},
|
|
async getContents() {
|
|
throw new Error("not used");
|
|
},
|
|
}));
|
|
|
|
const result = await provider.search({
|
|
query: "exa docs",
|
|
includeDomains: ["exa.ai"],
|
|
});
|
|
|
|
assert.deepEqual(captured, {
|
|
query: "exa docs",
|
|
options: {
|
|
contents: false,
|
|
numResults: 7,
|
|
includeDomains: ["exa.ai"],
|
|
excludeDomains: undefined,
|
|
startPublishedDate: undefined,
|
|
endPublishedDate: undefined,
|
|
category: undefined,
|
|
},
|
|
});
|
|
assert.equal(result.providerName, "exa-main");
|
|
assert.equal(result.results[0]?.url, "https://exa.ai/docs");
|
|
});
|
|
|
|
test("createExaProvider fetch defaults to text and preserves per-url failures", async () => {
|
|
const calls: Array<{ urls: string[]; options: Record<string, unknown> }> = [];
|
|
|
|
const provider = createExaProvider(baseConfig, () => ({
|
|
async search() {
|
|
throw new Error("not used");
|
|
},
|
|
async getContents(urls, options) {
|
|
const requestUrls = Array.isArray(urls) ? urls : [urls];
|
|
calls.push({ urls: requestUrls, options });
|
|
|
|
if (requestUrls[0] === "https://bad.example") {
|
|
throw new Error("429 rate limited");
|
|
}
|
|
|
|
return {
|
|
requestId: `req-${calls.length}`,
|
|
results: [
|
|
{
|
|
url: requestUrls[0],
|
|
title: "Fetched page",
|
|
text: "Fetched body",
|
|
},
|
|
],
|
|
};
|
|
},
|
|
}));
|
|
|
|
const result = await provider.fetch({
|
|
urls: ["https://good.example", "https://bad.example"],
|
|
});
|
|
|
|
assert.equal((calls[0]?.options.text as { maxCharacters: number }).maxCharacters, 9000);
|
|
assert.deepEqual(result.results, [
|
|
{
|
|
url: "https://good.example",
|
|
title: "Fetched page",
|
|
text: "Fetched body",
|
|
highlights: undefined,
|
|
summary: undefined,
|
|
},
|
|
{
|
|
url: "https://bad.example",
|
|
title: null,
|
|
error: "429 rate limited",
|
|
},
|
|
]);
|
|
});
|
|
```
|
|
|
|
- [x] **Step 2: Run the Exa adapter tests to verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/providers/exa.test.ts
|
|
```
|
|
|
|
Expected: `FAIL` with an `ERR_MODULE_NOT_FOUND` error for `./exa.ts`.
|
|
|
|
- [x] **Step 3: Write the provider interface and Exa adapter implementation**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/types.ts` with this content:
|
|
|
|
```ts
|
|
export interface NormalizedSearchRequest {
|
|
query: string;
|
|
limit?: number;
|
|
includeDomains?: string[];
|
|
excludeDomains?: string[];
|
|
startPublishedDate?: string;
|
|
endPublishedDate?: string;
|
|
category?: string;
|
|
provider?: string;
|
|
}
|
|
|
|
export interface NormalizedSearchResult {
|
|
id?: string;
|
|
title: string | null;
|
|
url: string;
|
|
publishedDate?: string;
|
|
author?: string;
|
|
score?: number;
|
|
}
|
|
|
|
export interface NormalizedSearchResponse {
|
|
providerName: string;
|
|
requestId?: string;
|
|
searchTime?: number;
|
|
results: NormalizedSearchResult[];
|
|
}
|
|
|
|
export interface NormalizedFetchRequest {
|
|
urls: string[];
|
|
text?: boolean;
|
|
highlights?: boolean;
|
|
summary?: boolean;
|
|
textMaxCharacters?: number;
|
|
provider?: string;
|
|
}
|
|
|
|
export interface NormalizedFetchResult {
|
|
url: string;
|
|
title: string | null;
|
|
text?: string;
|
|
highlights?: string[];
|
|
summary?: string;
|
|
error?: string;
|
|
}
|
|
|
|
export interface NormalizedFetchResponse {
|
|
providerName: string;
|
|
requestIds?: string[];
|
|
results: NormalizedFetchResult[];
|
|
}
|
|
|
|
export interface WebProvider {
|
|
name: string;
|
|
type: string;
|
|
search(request: NormalizedSearchRequest): Promise<NormalizedSearchResponse>;
|
|
fetch(request: NormalizedFetchRequest): Promise<NormalizedFetchResponse>;
|
|
}
|
|
```
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/providers/exa.ts` with this content:
|
|
|
|
```ts
|
|
import Exa from "exa-js";
|
|
import type { ExaProviderConfig } from "../schema.ts";
|
|
import type {
|
|
NormalizedFetchRequest,
|
|
NormalizedFetchResponse,
|
|
NormalizedSearchRequest,
|
|
NormalizedSearchResponse,
|
|
WebProvider,
|
|
} from "./types.ts";
|
|
|
|
export interface ExaClientLike {
|
|
search(query: string, options?: Record<string, unknown>): Promise<any>;
|
|
getContents(urls: string[] | string, options?: Record<string, unknown>): Promise<any>;
|
|
}
|
|
|
|
export type ExaClientFactory = (apiKey: string) => ExaClientLike;
|
|
|
|
export function buildSearchOptions(config: ExaProviderConfig, request: NormalizedSearchRequest) {
|
|
return {
|
|
contents: false,
|
|
numResults: request.limit ?? config.options?.defaultSearchLimit ?? 5,
|
|
includeDomains: request.includeDomains,
|
|
excludeDomains: request.excludeDomains,
|
|
startPublishedDate: request.startPublishedDate,
|
|
endPublishedDate: request.endPublishedDate,
|
|
category: request.category,
|
|
};
|
|
}
|
|
|
|
export function buildFetchOptions(config: ExaProviderConfig, request: NormalizedFetchRequest) {
|
|
const text = request.text ?? (!request.highlights && !request.summary);
|
|
|
|
return {
|
|
...(text
|
|
? {
|
|
text: {
|
|
maxCharacters: request.textMaxCharacters ?? config.options?.defaultFetchTextMaxCharacters ?? 12000,
|
|
},
|
|
}
|
|
: {}),
|
|
...(request.highlights
|
|
? {
|
|
highlights: {
|
|
maxCharacters: config.options?.defaultFetchHighlightsMaxCharacters ?? 1000,
|
|
},
|
|
}
|
|
: {}),
|
|
...(request.summary ? { summary: true } : {}),
|
|
};
|
|
}
|
|
|
|
export function createExaProvider(
|
|
config: ExaProviderConfig,
|
|
createClient: ExaClientFactory = (apiKey) => new Exa(apiKey) as unknown as ExaClientLike,
|
|
): WebProvider {
|
|
const client = createClient(config.apiKey);
|
|
|
|
return {
|
|
name: config.name,
|
|
type: config.type,
|
|
|
|
async search(request: NormalizedSearchRequest): Promise<NormalizedSearchResponse> {
|
|
const response = await client.search(request.query, buildSearchOptions(config, request));
|
|
return {
|
|
providerName: config.name,
|
|
requestId: response.requestId,
|
|
searchTime: response.searchTime,
|
|
results: (response.results ?? []).map((item: any) => ({
|
|
id: item.id,
|
|
title: item.title ?? null,
|
|
url: item.url,
|
|
publishedDate: item.publishedDate,
|
|
author: item.author,
|
|
score: item.score,
|
|
})),
|
|
};
|
|
},
|
|
|
|
async fetch(request: NormalizedFetchRequest): Promise<NormalizedFetchResponse> {
|
|
const requestIds: string[] = [];
|
|
const options = buildFetchOptions(config, request);
|
|
|
|
const results = await Promise.all(
|
|
request.urls.map(async (url) => {
|
|
try {
|
|
const response = await client.getContents([url], options);
|
|
if (response.requestId) {
|
|
requestIds.push(response.requestId);
|
|
}
|
|
|
|
const item = response.results?.[0];
|
|
if (!item) {
|
|
return {
|
|
url,
|
|
title: null,
|
|
error: "No content returned",
|
|
};
|
|
}
|
|
|
|
return {
|
|
url: item.url ?? url,
|
|
title: item.title ?? null,
|
|
text: typeof item.text === "string" ? item.text : undefined,
|
|
highlights: Array.isArray(item.highlights) ? item.highlights : undefined,
|
|
summary: typeof item.summary === "string" ? item.summary : undefined,
|
|
};
|
|
} catch (error) {
|
|
return {
|
|
url,
|
|
title: null,
|
|
error: (error as Error).message,
|
|
};
|
|
}
|
|
}),
|
|
);
|
|
|
|
return {
|
|
providerName: config.name,
|
|
requestIds,
|
|
results,
|
|
};
|
|
},
|
|
};
|
|
}
|
|
```
|
|
|
|
- [x] **Step 4: Run the Exa adapter tests to verify they pass**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/providers/exa.test.ts
|
|
```
|
|
|
|
Expected: `PASS` for both tests.
|
|
|
|
- [x] **Step 5: Commit the provider layer**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
git -C /home/alex/dotfiles add \
|
|
.pi/agent/extensions/web-search/src/providers/types.ts \
|
|
.pi/agent/extensions/web-search/src/providers/exa.ts \
|
|
.pi/agent/extensions/web-search/src/providers/exa.test.ts \
|
|
docs/superpowers/plans/2026-04-09-web-search-tools.md
|
|
|
|
git -C /home/alex/dotfiles commit -m "test: add exa web provider adapter"
|
|
```
|
|
|
|
### Task 3: Add the shared formatter and truncation helpers
|
|
|
|
**Files:**
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.test.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.test.ts`
|
|
|
|
- [x] **Step 1: Write the failing formatter tests**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { formatFetchOutput, formatSearchOutput, truncateText } from "./format.ts";
|
|
|
|
test("formatSearchOutput renders a compact metadata-only list", () => {
|
|
const output = formatSearchOutput({
|
|
providerName: "exa-main",
|
|
results: [
|
|
{
|
|
title: "Exa Docs",
|
|
url: "https://exa.ai/docs",
|
|
publishedDate: "2026-04-09",
|
|
author: "Exa",
|
|
score: 0.98,
|
|
},
|
|
],
|
|
});
|
|
|
|
assert.match(output, /Found 1 web result via exa-main:/);
|
|
assert.match(output, /Exa Docs/);
|
|
assert.match(output, /https:\/\/exa.ai\/docs/);
|
|
});
|
|
|
|
test("truncateText shortens long fetch bodies with an ellipsis", () => {
|
|
assert.equal(truncateText("abcdef", 4), "abc…");
|
|
assert.equal(truncateText("abc", 10), "abc");
|
|
});
|
|
|
|
test("formatFetchOutput includes both successful and failed URLs", () => {
|
|
const output = formatFetchOutput(
|
|
{
|
|
providerName: "exa-main",
|
|
results: [
|
|
{
|
|
url: "https://good.example",
|
|
title: "Good",
|
|
text: "This is a very long body that should be truncated in the final output.",
|
|
},
|
|
{
|
|
url: "https://bad.example",
|
|
title: null,
|
|
error: "429 rate limited",
|
|
},
|
|
],
|
|
},
|
|
{ maxCharactersPerResult: 20 },
|
|
);
|
|
|
|
assert.match(output, /Status: ok/);
|
|
assert.match(output, /Status: failed/);
|
|
assert.match(output, /429 rate limited/);
|
|
assert.match(output, /This is a very long…/);
|
|
});
|
|
```
|
|
|
|
- [x] **Step 2: Run the formatter tests to verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/format.test.ts
|
|
```
|
|
|
|
Expected: `FAIL` with an `ERR_MODULE_NOT_FOUND` error for `./format.ts`.
|
|
|
|
- [x] **Step 3: Write the minimal formatter implementation**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/format.ts` with this content:
|
|
|
|
```ts
|
|
import type { NormalizedFetchResponse, NormalizedSearchResponse } from "./providers/types.ts";
|
|
|
|
export function truncateText(text: string, maxCharacters = 4000) {
|
|
if (text.length <= maxCharacters) {
|
|
return text;
|
|
}
|
|
return `${text.slice(0, Math.max(0, maxCharacters - 1))}…`;
|
|
}
|
|
|
|
export function formatSearchOutput(response: NormalizedSearchResponse) {
|
|
if (response.results.length === 0) {
|
|
return `No web results via ${response.providerName}.`;
|
|
}
|
|
|
|
const lines = [
|
|
`Found ${response.results.length} web result${response.results.length === 1 ? "" : "s"} via ${response.providerName}:`,
|
|
];
|
|
|
|
for (const [index, result] of response.results.entries()) {
|
|
lines.push(`${index + 1}. ${result.title ?? "(untitled)"}`);
|
|
lines.push(` URL: ${result.url}`);
|
|
|
|
const meta = [result.publishedDate, result.author].filter(Boolean);
|
|
if (meta.length > 0) {
|
|
lines.push(` Meta: ${meta.join(" • ")}`);
|
|
}
|
|
|
|
if (typeof result.score === "number") {
|
|
lines.push(` Score: ${result.score}`);
|
|
}
|
|
}
|
|
|
|
return lines.join("\n");
|
|
}
|
|
|
|
export interface FetchFormatOptions {
|
|
maxCharactersPerResult?: number;
|
|
}
|
|
|
|
export function formatFetchOutput(response: NormalizedFetchResponse, options: FetchFormatOptions = {}) {
|
|
const maxCharactersPerResult = options.maxCharactersPerResult ?? 4000;
|
|
const lines = [
|
|
`Fetched ${response.results.length} URL${response.results.length === 1 ? "" : "s"} via ${response.providerName}:`,
|
|
];
|
|
|
|
for (const result of response.results) {
|
|
lines.push("");
|
|
lines.push(`URL: ${result.url}`);
|
|
|
|
if (result.error) {
|
|
lines.push("Status: failed");
|
|
lines.push(`Error: ${result.error}`);
|
|
continue;
|
|
}
|
|
|
|
lines.push("Status: ok");
|
|
if (result.title) {
|
|
lines.push(`Title: ${result.title}`);
|
|
}
|
|
if (result.summary) {
|
|
lines.push(`Summary: ${result.summary}`);
|
|
}
|
|
if (result.highlights?.length) {
|
|
lines.push("Highlights:");
|
|
for (const highlight of result.highlights) {
|
|
lines.push(`- ${highlight}`);
|
|
}
|
|
}
|
|
if (result.text) {
|
|
lines.push("Text:");
|
|
lines.push(truncateText(result.text, maxCharactersPerResult));
|
|
}
|
|
}
|
|
|
|
return lines.join("\n");
|
|
}
|
|
```
|
|
|
|
- [x] **Step 4: Run the formatter tests to verify they pass**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/format.test.ts
|
|
```
|
|
|
|
Expected: `PASS` for all three tests.
|
|
|
|
- [x] **Step 5: Commit the formatter layer**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
git -C /home/alex/dotfiles add \
|
|
.pi/agent/extensions/web-search/src/format.ts \
|
|
.pi/agent/extensions/web-search/src/format.test.ts \
|
|
docs/superpowers/plans/2026-04-09-web-search-tools.md
|
|
|
|
git -C /home/alex/dotfiles commit -m "test: add web tool output formatting"
|
|
```
|
|
|
|
### Task 4: Add the `web_search` tool and verify live search
|
|
|
|
**Files:**
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.test.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.test.ts`
|
|
|
|
- [x] **Step 1: Write the failing `web_search` tool tests**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { createWebSearchTool } from "./web-search.ts";
|
|
|
|
test("web_search executes metadata-only search through the resolved provider", async () => {
|
|
let resolvedProviderName: string | undefined;
|
|
|
|
const tool = createWebSearchTool({
|
|
resolveProvider: async (providerName) => {
|
|
resolvedProviderName = providerName;
|
|
return {
|
|
name: "exa-main",
|
|
type: "exa",
|
|
async search(request) {
|
|
assert.equal(request.query, "exa docs");
|
|
return {
|
|
providerName: "exa-main",
|
|
results: [
|
|
{
|
|
title: "Exa Docs",
|
|
url: "https://exa.ai/docs",
|
|
score: 0.98,
|
|
},
|
|
],
|
|
};
|
|
},
|
|
async fetch() {
|
|
throw new Error("not used");
|
|
},
|
|
};
|
|
},
|
|
});
|
|
|
|
const result = await tool.execute("tool-1", { query: "exa docs" }, undefined, undefined, undefined);
|
|
|
|
assert.equal(resolvedProviderName, undefined);
|
|
assert.match((result.content[0] as { text: string }).text, /Exa Docs/);
|
|
assert.equal((result.details as { results: Array<{ url: string }> }).results[0]?.url, "https://exa.ai/docs");
|
|
});
|
|
|
|
test("web_search rejects a blank query before resolving a provider", async () => {
|
|
const tool = createWebSearchTool({
|
|
resolveProvider: async () => {
|
|
throw new Error("should not resolve provider for a blank query");
|
|
},
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => tool.execute("tool-1", { query: " " }, undefined, undefined, undefined),
|
|
/non-empty query/,
|
|
);
|
|
});
|
|
```
|
|
|
|
- [x] **Step 2: Run the `web_search` tests to verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/tools/web-search.test.ts
|
|
```
|
|
|
|
Expected: `FAIL` with an `ERR_MODULE_NOT_FOUND` error for `./web-search.ts`.
|
|
|
|
- [x] **Step 3: Write the minimal `web_search` tool implementation**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-search.ts` with this content:
|
|
|
|
```ts
|
|
import { Text } from "@mariozechner/pi-tui";
|
|
import { formatSearchOutput } from "../format.ts";
|
|
import type { NormalizedSearchResponse, WebProvider } from "../providers/types.ts";
|
|
import { WebSearchParamsSchema, type WebSearchParams } from "../schema.ts";
|
|
|
|
interface SearchToolDeps {
|
|
resolveProvider(providerName?: string): Promise<WebProvider>;
|
|
}
|
|
|
|
function normalizeSearchQuery(query: string) {
|
|
const trimmed = query.trim();
|
|
if (!trimmed) {
|
|
throw new Error("web_search requires a non-empty query.");
|
|
}
|
|
return trimmed;
|
|
}
|
|
|
|
export function createWebSearchTool({ resolveProvider }: SearchToolDeps) {
|
|
return {
|
|
name: "web_search",
|
|
label: "Web Search",
|
|
description: "Search the web through the configured provider. Returns result metadata by default.",
|
|
parameters: WebSearchParamsSchema,
|
|
|
|
async execute(_toolCallId: string, params: WebSearchParams) {
|
|
const provider = await resolveProvider(params.provider);
|
|
const response = await provider.search({
|
|
query: normalizeSearchQuery(params.query),
|
|
limit: params.limit,
|
|
includeDomains: params.includeDomains,
|
|
excludeDomains: params.excludeDomains,
|
|
startPublishedDate: params.startPublishedDate,
|
|
endPublishedDate: params.endPublishedDate,
|
|
category: params.category,
|
|
provider: params.provider,
|
|
});
|
|
|
|
return {
|
|
content: [{ type: "text" as const, text: formatSearchOutput(response) }],
|
|
details: response,
|
|
};
|
|
},
|
|
|
|
renderCall(args: Partial<WebSearchParams>, theme: any) {
|
|
let text = theme.fg("toolTitle", theme.bold("web_search "));
|
|
text += theme.fg("muted", args.query ?? "");
|
|
return new Text(text, 0, 0);
|
|
},
|
|
|
|
renderResult(result: { details?: NormalizedSearchResponse }, _options: unknown, theme: any) {
|
|
const details = result.details;
|
|
if (!details) {
|
|
return new Text("", 0, 0);
|
|
}
|
|
|
|
const lines = [
|
|
`${theme.fg("success", "✓ ")}${details.results.length} result${details.results.length === 1 ? "" : "s"} via ${details.providerName}`,
|
|
];
|
|
|
|
for (const [index, item] of details.results.slice(0, 5).entries()) {
|
|
lines.push(` ${theme.fg("muted", `${index + 1}.`)} ${item.title ?? "(untitled)"} ${theme.fg("dim", item.url)}`);
|
|
}
|
|
|
|
return new Text(lines.join("\n"), 0, 0);
|
|
},
|
|
};
|
|
}
|
|
```
|
|
|
|
- [x] **Step 4: Run the `web_search` tests to verify they pass**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/tools/web-search.test.ts
|
|
```
|
|
|
|
Expected: `PASS` for both tests.
|
|
|
|
- [x] **Step 5: Create a real Exa config file for the live smoke test**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
read -rsp "Exa API key: " EXA_KEY && printf '\n'
|
|
mkdir -p /home/alex/.pi/agent
|
|
cat > /home/alex/.pi/agent/web-search.json <<JSON
|
|
{
|
|
"defaultProvider": "exa-main",
|
|
"providers": [
|
|
{
|
|
"name": "exa-main",
|
|
"type": "exa",
|
|
"apiKey": "${EXA_KEY}",
|
|
"options": {
|
|
"defaultSearchLimit": 5,
|
|
"defaultFetchTextMaxCharacters": 12000
|
|
}
|
|
}
|
|
]
|
|
}
|
|
JSON
|
|
chmod 600 /home/alex/.pi/agent/web-search.json
|
|
unset EXA_KEY
|
|
```
|
|
|
|
Expected: `/home/alex/.pi/agent/web-search.json` exists with mode `600`.
|
|
|
|
- [x] **Step 6: Add a temporary entrypoint that registers `web_search` and manually verify it live**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/index.ts` with this content:
|
|
|
|
```ts
|
|
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
|
import { loadWebSearchConfig } from "./src/config.ts";
|
|
import { createExaProvider } from "./src/providers/exa.ts";
|
|
import type { WebProvider } from "./src/providers/types.ts";
|
|
import { createWebSearchTool } from "./src/tools/web-search.ts";
|
|
|
|
async function resolveProvider(providerName?: string): Promise<WebProvider> {
|
|
const config = await loadWebSearchConfig();
|
|
const selectedName = providerName ?? config.defaultProviderName;
|
|
const providerConfig = config.providersByName.get(selectedName);
|
|
|
|
if (!providerConfig) {
|
|
throw new Error(
|
|
`Unknown web-search provider \"${selectedName}\". Configured providers: ${[...config.providersByName.keys()].join(", ")}`,
|
|
);
|
|
}
|
|
|
|
switch (providerConfig.type) {
|
|
case "exa":
|
|
return createExaProvider(providerConfig);
|
|
default:
|
|
throw new Error(`Unsupported web-search provider type: ${(providerConfig as { type: string }).type}`);
|
|
}
|
|
}
|
|
|
|
export default function webSearch(pi: ExtensionAPI) {
|
|
pi.registerTool(createWebSearchTool({ resolveProvider }));
|
|
}
|
|
```
|
|
|
|
Then run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles
|
|
pi
|
|
```
|
|
|
|
Inside pi, run this sequence:
|
|
|
|
```text
|
|
/reload
|
|
Search the web for Exa docs using the web_search tool.
|
|
```
|
|
|
|
Expected manual checks:
|
|
|
|
- the agent has a `web_search` tool available
|
|
- the tool call row shows `web_search` and the query text
|
|
- the result contains metadata-only output, not fetched page text
|
|
- at least one result includes `https://exa.ai/docs`
|
|
|
|
- [x] **Step 7: Commit the `web_search` tool**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
git -C /home/alex/dotfiles add \
|
|
.pi/agent/extensions/web-search/index.ts \
|
|
.pi/agent/extensions/web-search/src/tools/web-search.ts \
|
|
.pi/agent/extensions/web-search/src/tools/web-search.test.ts \
|
|
docs/superpowers/plans/2026-04-09-web-search-tools.md
|
|
|
|
git -C /home/alex/dotfiles commit -m "feat: add web_search tool"
|
|
```
|
|
|
|
### Task 5: Add `web_fetch`, register both tools, and verify batch fetch behavior
|
|
|
|
**Files:**
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.test.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.ts`
|
|
- Create: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/extension.test.ts`
|
|
- Modify: `/home/alex/dotfiles/.pi/agent/extensions/web-search/index.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.test.ts`
|
|
- Test: `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/extension.test.ts`
|
|
|
|
- [x] **Step 1: Write the failing `web_fetch` and extension registration tests**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { createWebFetchTool } from "./web-fetch.ts";
|
|
|
|
test("web_fetch prepareArguments folds a single url into urls", () => {
|
|
const tool = createWebFetchTool({
|
|
resolveProvider: async () => {
|
|
throw new Error("not used");
|
|
},
|
|
});
|
|
|
|
assert.deepEqual(tool.prepareArguments?.({ url: "https://exa.ai/docs" }), {
|
|
url: "https://exa.ai/docs",
|
|
urls: ["https://exa.ai/docs"],
|
|
});
|
|
});
|
|
|
|
test("web_fetch defaults to text and returns formatted fetch results", async () => {
|
|
let capturedRequest: Record<string, unknown> | undefined;
|
|
|
|
const tool = createWebFetchTool({
|
|
resolveProvider: async () => ({
|
|
name: "exa-main",
|
|
type: "exa",
|
|
async search() {
|
|
throw new Error("not used");
|
|
},
|
|
async fetch(request) {
|
|
capturedRequest = request as unknown as Record<string, unknown>;
|
|
return {
|
|
providerName: "exa-main",
|
|
results: [
|
|
{
|
|
url: "https://exa.ai/docs",
|
|
title: "Docs",
|
|
text: "Body",
|
|
},
|
|
],
|
|
};
|
|
},
|
|
}),
|
|
});
|
|
|
|
const result = await tool.execute("tool-1", { urls: ["https://exa.ai/docs"] }, undefined, undefined, undefined);
|
|
|
|
assert.equal(capturedRequest?.text, true);
|
|
assert.match((result.content[0] as { text: string }).text, /Body/);
|
|
assert.equal((result.details as { results: Array<{ title: string }> }).results[0]?.title, "Docs");
|
|
});
|
|
|
|
test("web_fetch rejects malformed URLs", async () => {
|
|
const tool = createWebFetchTool({
|
|
resolveProvider: async () => {
|
|
throw new Error("should not resolve provider for invalid URLs");
|
|
},
|
|
});
|
|
|
|
await assert.rejects(
|
|
() => tool.execute("tool-1", { urls: ["not-a-url"] }, undefined, undefined, undefined),
|
|
/Invalid URL/,
|
|
);
|
|
});
|
|
```
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/extension.test.ts` with this content:
|
|
|
|
```ts
|
|
import test from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import webSearchExtension from "../index.ts";
|
|
|
|
test("the extension entrypoint registers both web_search and web_fetch", () => {
|
|
const registeredTools: string[] = [];
|
|
|
|
webSearchExtension({
|
|
registerTool(tool: { name: string }) {
|
|
registeredTools.push(tool.name);
|
|
},
|
|
} as any);
|
|
|
|
assert.deepEqual(registeredTools, ["web_search", "web_fetch"]);
|
|
});
|
|
```
|
|
|
|
- [x] **Step 2: Run the new tests to verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/tools/web-fetch.test.ts src/extension.test.ts
|
|
```
|
|
|
|
Expected: `FAIL` with an `ERR_MODULE_NOT_FOUND` error for `./web-fetch.ts`.
|
|
|
|
- [x] **Step 3: Write the minimal `web_fetch` tool implementation**
|
|
|
|
Create `/home/alex/dotfiles/.pi/agent/extensions/web-search/src/tools/web-fetch.ts` with this content:
|
|
|
|
```ts
|
|
import { Text } from "@mariozechner/pi-tui";
|
|
import { formatFetchOutput } from "../format.ts";
|
|
import type { NormalizedFetchResponse, WebProvider } from "../providers/types.ts";
|
|
import { WebFetchParamsSchema, type WebFetchParams } from "../schema.ts";
|
|
|
|
interface FetchToolDeps {
|
|
resolveProvider(providerName?: string): Promise<WebProvider>;
|
|
}
|
|
|
|
function normalizeUrl(value: string) {
|
|
try {
|
|
return new URL(value).toString();
|
|
} catch {
|
|
throw new Error(`Invalid URL: ${value}`);
|
|
}
|
|
}
|
|
|
|
function normalizeFetchParams(params: WebFetchParams & { url?: string }) {
|
|
const urls = (Array.isArray(params.urls) ? params.urls : []).map(normalizeUrl);
|
|
if (urls.length === 0) {
|
|
throw new Error("web_fetch requires at least one URL.");
|
|
}
|
|
|
|
return {
|
|
urls,
|
|
text: params.text ?? (!params.highlights && !params.summary),
|
|
highlights: params.highlights ?? false,
|
|
summary: params.summary ?? false,
|
|
textMaxCharacters: params.textMaxCharacters,
|
|
provider: params.provider,
|
|
};
|
|
}
|
|
|
|
export function createWebFetchTool({ resolveProvider }: FetchToolDeps) {
|
|
return {
|
|
name: "web_fetch",
|
|
label: "Web Fetch",
|
|
description: "Fetch page contents through the configured provider. Returns text by default.",
|
|
parameters: WebFetchParamsSchema,
|
|
|
|
prepareArguments(args: unknown) {
|
|
if (!args || typeof args !== "object") {
|
|
return args;
|
|
}
|
|
|
|
const input = args as { url?: unknown; urls?: unknown };
|
|
if (typeof input.url === "string" && !Array.isArray(input.urls)) {
|
|
return {
|
|
...input,
|
|
urls: [input.url],
|
|
};
|
|
}
|
|
|
|
return args;
|
|
},
|
|
|
|
async execute(_toolCallId: string, params: WebFetchParams) {
|
|
const normalized = normalizeFetchParams(params as WebFetchParams & { url?: string });
|
|
const provider = await resolveProvider(normalized.provider);
|
|
const response = await provider.fetch(normalized);
|
|
|
|
return {
|
|
content: [{ type: "text" as const, text: formatFetchOutput(response) }],
|
|
details: response,
|
|
};
|
|
},
|
|
|
|
renderCall(args: Partial<WebFetchParams> & { url?: string }, theme: any) {
|
|
const urls = Array.isArray(args.urls) ? args.urls : typeof args.url === "string" ? [args.url] : [];
|
|
let text = theme.fg("toolTitle", theme.bold("web_fetch "));
|
|
text += theme.fg("muted", `${urls.length} url${urls.length === 1 ? "" : "s"}`);
|
|
return new Text(text, 0, 0);
|
|
},
|
|
|
|
renderResult(result: { details?: NormalizedFetchResponse }, _options: unknown, theme: any) {
|
|
const details = result.details;
|
|
if (!details) {
|
|
return new Text("", 0, 0);
|
|
}
|
|
|
|
const failed = details.results.filter((item) => item.error).length;
|
|
const succeeded = details.results.length - failed;
|
|
return new Text(
|
|
`${theme.fg("success", "✓ ")}${succeeded} ok${failed ? ` • ${theme.fg("warning", `${failed} failed`)}` : ""}`,
|
|
0,
|
|
0,
|
|
);
|
|
},
|
|
};
|
|
}
|
|
```
|
|
|
|
- [x] **Step 4: Replace the temporary entrypoint with the final entrypoint that registers both tools**
|
|
|
|
Replace `/home/alex/dotfiles/.pi/agent/extensions/web-search/index.ts` with this content:
|
|
|
|
```ts
|
|
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
|
import { loadWebSearchConfig } from "./src/config.ts";
|
|
import { createExaProvider } from "./src/providers/exa.ts";
|
|
import type { WebProvider } from "./src/providers/types.ts";
|
|
import { createWebFetchTool } from "./src/tools/web-fetch.ts";
|
|
import { createWebSearchTool } from "./src/tools/web-search.ts";
|
|
|
|
async function resolveProvider(providerName?: string): Promise<WebProvider> {
|
|
const config = await loadWebSearchConfig();
|
|
const selectedName = providerName ?? config.defaultProviderName;
|
|
const providerConfig = config.providersByName.get(selectedName);
|
|
|
|
if (!providerConfig) {
|
|
throw new Error(
|
|
`Unknown web-search provider \"${selectedName}\". Configured providers: ${[...config.providersByName.keys()].join(", ")}`,
|
|
);
|
|
}
|
|
|
|
switch (providerConfig.type) {
|
|
case "exa":
|
|
return createExaProvider(providerConfig);
|
|
default:
|
|
throw new Error(`Unsupported web-search provider type: ${(providerConfig as { type: string }).type}`);
|
|
}
|
|
}
|
|
|
|
export default function webSearch(pi: ExtensionAPI) {
|
|
pi.registerTool(createWebSearchTool({ resolveProvider }));
|
|
pi.registerTool(createWebFetchTool({ resolveProvider }));
|
|
}
|
|
```
|
|
|
|
- [x] **Step 5: Run the `web_fetch` and entrypoint smoke tests to verify they pass**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npx tsx --test src/tools/web-fetch.test.ts src/extension.test.ts
|
|
```
|
|
|
|
Expected: `PASS` for all four tests.
|
|
|
|
- [x] **Step 6: Reload pi and manually verify search + single fetch + batch fetch**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles
|
|
pi
|
|
```
|
|
|
|
Inside pi, run this sequence:
|
|
|
|
```text
|
|
/reload
|
|
Search the web for Exa docs using the web_search tool.
|
|
Then fetch https://exa.ai/docs using the web_fetch tool.
|
|
Then fetch https://exa.ai/docs and https://exa.ai using the web_fetch tool.
|
|
```
|
|
|
|
Expected manual checks:
|
|
|
|
- both `web_search` and `web_fetch` are available
|
|
- `web_search` still returns metadata only by default
|
|
- `web_fetch` returns page text by default
|
|
- `web_fetch` accepts multiple URLs in one call
|
|
- batch fetch output is clearly separated per URL
|
|
- if one URL fails, the result still includes the successful URL output and a failure section for the bad URL
|
|
|
|
- [x] **Step 7: Commit the completed web tools extension**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
git -C /home/alex/dotfiles add \
|
|
.pi/agent/extensions/web-search/index.ts \
|
|
.pi/agent/extensions/web-search/src/tools/web-fetch.ts \
|
|
.pi/agent/extensions/web-search/src/tools/web-fetch.test.ts \
|
|
.pi/agent/extensions/web-search/src/extension.test.ts \
|
|
docs/superpowers/plans/2026-04-09-web-search-tools.md
|
|
|
|
git -C /home/alex/dotfiles commit -m "feat: add web search tools extension"
|
|
```
|
|
|
|
## Final Verification Checklist
|
|
|
|
Run these before claiming the work is complete:
|
|
|
|
```bash
|
|
cd /home/alex/dotfiles/.pi/agent/extensions/web-search
|
|
npm test
|
|
```
|
|
|
|
Expected: `PASS` for all config, provider, formatter, tool, and entrypoint tests.
|
|
|
|
Then run pi from `/home/alex/dotfiles`, use `/reload`, and manually verify:
|
|
|
|
1. `web_search` returns metadata-only results
|
|
2. `web_search` handles a configured Exa provider successfully
|
|
3. `web_fetch` returns text by default
|
|
4. `web_fetch` accepts a single URL
|
|
5. `web_fetch` accepts multiple URLs
|
|
6. invalid URLs fail before the provider is called
|
|
7. missing or invalid `~/.pi/agent/web-search.json` produces a helpful error
|
|
8. both tools remain available after `/reload`
|
|
|
|
If any of those checks fail, do not mark the work complete.
|