diff --git a/AGENTS.md b/AGENTS.md index 21b401980ed..a08c8e3513a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -182,7 +182,7 @@ Telegraph style. Root rules only. Read scoped `AGENTS.md` before subtree work. ## Ops / Footguns - Remote install docs: `docs/install/{exe-dev,fly,hetzner}.md`. Parallels smoke: `$openclaw-parallels-smoke`; Discord roundtrip: `parallels-discord-roundtrip`. -- Memory wiki: keep prompt digest tiny. The prompt should only say the wiki exists, prefer `wiki_search` / `wiki_get`, start from `reports/maintainer-agent-directory.md` for people routing, and verify contact data before use. +- Memory wiki: keep prompt digest tiny. The prompt should only say the wiki exists, prefer `wiki_search` / `wiki_get`, start from `reports/person-agent-directory.md` for people routing, use search modes (`find-person`, `route-question`, `source-evidence`, `raw-claim`) when useful, and verify contact data before use. - People wiki provenance: generated identity, social, contact, and "fun detail" notes need explicit source class/confidence (`maintainer-whois`, Discrawl sample/stat, GitHub profile, maintainer repo file). Do not promote inferred details to facts. - Rebrand/migration/config warnings: run `openclaw doctor`. - Never edit `node_modules`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 16ce2bef13b..a693b78000e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ Docs: https://docs.openclaw.ai ### Changes - Providers/NVIDIA: add the NVIDIA provider with API-key onboarding, setup docs, static catalog metadata, and literal model-ref picker support so NVIDIA hosted models can be selected with their provider prefix intact. (#71204) Thanks @eleqtrizit. +- Memory/wiki: add agent-facing people wiki metadata, canonical aliases, person cards, relationship graphs, privacy/provenance reports, evidence-kind drilldown, and search modes for person lookup, question routing, source evidence, and raw claims. Thanks @vincentkoc. - Messages: add global `messages.visibleReplies` so operators can require visible output to go through `message(action=send)` for any source chat, while `messages.groupChat.visibleReplies` stays available as the group/channel override. Thanks @scoootscooob. - Gateway/dev: run `pnpm gateway:watch` through a named tmux session by default, with `gateway:watch:raw` and `OPENCLAW_GATEWAY_WATCH_TMUX=0` for foreground mode, so repeated starts respawn an inspectable watcher without trapping the invoking agent shell. Thanks @vincentkoc. - Plugin SDK: mark remaining legacy alias exports and diffs tool/config aliases with deprecation metadata, and add a guard so future legacy alias comments require `@deprecated` tags. Thanks @vincentkoc. diff --git a/docs/cli/wiki.md b/docs/cli/wiki.md index 477bf2b64be..50901f0aa28 100644 --- a/docs/cli/wiki.md +++ b/docs/cli/wiki.md @@ -38,6 +38,7 @@ openclaw wiki ingest ./notes/alpha.md openclaw wiki compile openclaw wiki lint openclaw wiki search "alpha" +openclaw wiki search "who should I ask about Teams?" --mode route-question openclaw wiki get entity.alpha --from 1 --lines 80 openclaw wiki apply synthesis "Alpha Summary" \ @@ -135,11 +136,34 @@ Behavior depends on config: - `search.backend`: `shared` or `local` - `search.corpus`: `wiki`, `memory`, or `all` +- `--mode`: `auto`, `find-person`, `route-question`, `source-evidence`, or + `raw-claim` Use `wiki search` when you want wiki-specific ranking or provenance details. For one broad shared recall pass, prefer `openclaw memory search` when the active memory plugin exposes shared search. +Search modes help the agent choose the right surface: + +- `find-person`: aliases, handles, socials, canonical IDs, and person pages +- `route-question`: ask-for/best-used-for hints and relationship context +- `source-evidence`: source pages and structured evidence fields +- `raw-claim`: structured claim text with claim/evidence metadata + +Examples: + +```bash +openclaw wiki search "bgroux" --mode find-person +openclaw wiki search "who knows Teams rollout?" --mode route-question +openclaw wiki search "maintainer-whois" --mode source-evidence +openclaw wiki search "strong route Teams" --mode raw-claim --json +``` + +Text output includes `Claim:` and `Evidence:` lines when a result matches a +structured claim. JSON output additionally exposes `matchedClaimId`, +`matchedClaimStatus`, `matchedClaimConfidence`, `evidenceKinds`, and +`evidenceSourceIds` for agent-side drilldown. + ### `wiki get ` Read a wiki page by id or relative path. diff --git a/docs/plugins/memory-wiki.md b/docs/plugins/memory-wiki.md index abe381439df..12719fe96da 100644 --- a/docs/plugins/memory-wiki.md +++ b/docs/plugins/memory-wiki.md @@ -150,16 +150,90 @@ Each claim can include: Evidence entries can include: +- `kind` - `sourceId` - `path` - `lines` - `weight` +- `confidence` +- `privacyTier` - `note` - `updatedAt` This is what makes the wiki act more like a belief layer than a passive note dump. Claims can be tracked, scored, contested, and resolved back to sources. +## Agent-facing entity metadata + +Entity pages can also carry routing metadata for agent use. This is generic +frontmatter, so it works for people, teams, systems, projects, or any other +entity type. + +Common fields include: + +- `entityType`: for example `person`, `team`, `system`, or `project` +- `canonicalId`: stable identity key used across aliases and imports +- `aliases`: names, handles, or labels that should resolve to the same page +- `privacyTier`: `public`, `local-private`, `sensitive`, or `confirm-before-use` +- `bestUsedFor` / `notEnoughFor`: compact routing hints +- `lastRefreshedAt`: source-refresh timestamp separate from page edit time +- `personCard`: optional person-specific routing card with handles, socials, + emails, timezone, lane, ask-for, avoid-asking-for, confidence, and privacy +- `relationships`: typed edges to related pages with target, kind, weight, + confidence, evidence kind, privacy tier, and note + +For a people wiki, the agent should usually start with +`reports/person-agent-directory.md`, then open the person page with `wiki_get` +before using contact details or inferred facts. + +Example: + +```yaml +pageType: entity +entityType: person +id: entity.brad-groux +canonicalId: maintainer.brad-groux +aliases: + - Brad + - bgroux +privacyTier: local-private +bestUsedFor: + - Microsoft Teams and Azure routing +notEnoughFor: + - legal approval +lastRefreshedAt: "2026-04-29T00:00:00.000Z" +personCard: + handles: + - "@bgroux" + socials: + - "https://x.example/bgroux" + emails: + - brad@example.com + timezone: America/Chicago + lane: Microsoft ecosystem + askFor: + - Teams rollout questions + avoidAskingFor: + - unrelated billing decisions + confidence: 0.8 + privacyTier: confirm-before-use +relationships: + - targetId: entity.alice + targetTitle: Alice + kind: collaborates-with + confidence: 0.7 + evidenceKind: discrawl-stat +claims: + - id: claim.brad.teams + text: Brad is useful for Microsoft Teams routing. + status: supported + confidence: 0.9 + evidence: + - kind: maintainer-whois + sourceId: source.maintainers + privacyTier: local-private +``` + ## Compile pipeline The compile step reads wiki pages, normalizes summaries, and emits stable @@ -190,6 +264,10 @@ Built-in reports include: - `reports/low-confidence.md` - `reports/claim-health.md` - `reports/stale-pages.md` +- `reports/person-agent-directory.md` +- `reports/relationship-graph.md` +- `reports/provenance-coverage.md` +- `reports/privacy-review.md` These reports track things like: @@ -199,6 +277,10 @@ These reports track things like: - low-confidence pages and claims - stale or unknown freshness - pages with unresolved questions +- person/entity routing cards +- structured relationship edges +- evidence class coverage +- non-public privacy tiers that need review before use ## Search and retrieval @@ -219,6 +301,8 @@ Important behavior: - claim ids can resolve back to the owning page - contested/stale/fresh claims influence ranking - provenance labels can survive into results +- search mode can bias ranking for person lookup, question routing, source + evidence, or raw claims Practical rule: @@ -226,6 +310,22 @@ Practical rule: - use `wiki_search` + `wiki_get` when you care about wiki-specific ranking, provenance, or page-level belief structure +Search modes: + +- `auto`: balanced default +- `find-person`: boost person-like entities, aliases, handles, socials, and + canonical IDs +- `route-question`: boost agent cards, ask-for hints, best-used-for hints, and + relationship context +- `source-evidence`: boost source pages and structured evidence metadata +- `raw-claim`: boost matching structured claims and return claim/evidence + metadata in results + +When a result matches a structured claim, `wiki_search` can return +`matchedClaimId`, `matchedClaimStatus`, `matchedClaimConfidence`, +`evidenceKinds`, and `evidenceSourceIds` in its details payload. Text output +also includes compact `Claim:` and `Evidence:` lines when available. + ## Agent tools The plugin registers these tools: @@ -239,7 +339,9 @@ The plugin registers these tools: What they do: - `wiki_status`: current vault mode, health, Obsidian CLI availability -- `wiki_search`: search wiki pages and, when configured, shared memory corpora +- `wiki_search`: search wiki pages and, when configured, shared memory corpora; + accepts `mode` for person lookup, question routing, source evidence, or raw + claim drilldown - `wiki_get`: read a wiki page by id/path or fall back to shared memory corpus - `wiki_apply`: narrow synthesis/metadata mutations without freeform page surgery - `wiki_lint`: structural checks, provenance gaps, contradictions, open questions diff --git a/extensions/memory-wiki/src/cli.ts b/extensions/memory-wiki/src/cli.ts index 8823f2b0de2..2c53668f7be 100644 --- a/extensions/memory-wiki/src/cli.ts +++ b/extensions/memory-wiki/src/cli.ts @@ -26,7 +26,12 @@ import { runObsidianOpen, runObsidianSearch, } from "./obsidian.js"; -import { getMemoryWikiPage, searchMemoryWiki } from "./query.js"; +import { + getMemoryWikiPage, + searchMemoryWiki, + WIKI_SEARCH_MODES, + type WikiSearchMode, +} from "./query.js"; import { syncMemoryWikiImportedSources } from "./source-sync.js"; import type { MemoryWikiImportedSourceSyncResult } from "./source-sync.js"; import { @@ -81,6 +86,7 @@ type WikiSearchCommandOptions = { maxResults?: number; backend?: ResolvedMemoryWikiConfig["search"]["backend"]; corpus?: ResolvedMemoryWikiConfig["search"]["corpus"]; + mode?: WikiSearchMode; }; type WikiGetCommandOptions = { @@ -567,9 +573,13 @@ export async function runWikiSearch(params: { maxResults?: number; searchBackend?: ResolvedMemoryWikiConfig["search"]["backend"]; searchCorpus?: ResolvedMemoryWikiConfig["search"]["corpus"]; + mode?: WikiSearchMode; json?: boolean; stdout?: Pick; }) { + if (params.mode && !(WIKI_SEARCH_MODES as readonly string[]).includes(params.mode)) { + throw new Error(`wiki search --mode must be one of: ${WIKI_SEARCH_MODES.join(", ")}.`); + } await syncMemoryWikiImportedSources({ config: params.config, appConfig: params.appConfig }); const results = await searchMemoryWiki({ config: params.config, @@ -578,6 +588,7 @@ export async function runWikiSearch(params: { maxResults: params.maxResults, searchBackend: params.searchBackend, searchCorpus: params.searchCorpus, + mode: params.mode, }); const summary = params.json ? JSON.stringify(results, null, 2) @@ -586,7 +597,7 @@ export async function runWikiSearch(params: { : results .map( (result, index) => - `${index + 1}. ${result.title} (${result.corpus}/${result.kind})\nPath: ${result.path}${typeof result.startLine === "number" && typeof result.endLine === "number" ? `\nLines: ${result.startLine}-${result.endLine}` : ""}${result.provenanceLabel ? `\nProvenance: ${result.provenanceLabel}` : ""}\nSnippet: ${result.snippet}`, + `${index + 1}. ${result.title} (${result.corpus}/${result.kind})\nPath: ${result.path}${typeof result.startLine === "number" && typeof result.endLine === "number" ? `\nLines: ${result.startLine}-${result.endLine}` : ""}${result.provenanceLabel ? `\nProvenance: ${result.provenanceLabel}` : ""}${result.matchedClaimId ? `\nClaim: ${result.matchedClaimId}` : ""}${result.evidenceKinds && result.evidenceKinds.length > 0 ? `\nEvidence: ${result.evidenceKinds.join(", ")}` : ""}\nSnippet: ${result.snippet}`, ) .join("\n\n"); writeOutput(summary, params.stdout); @@ -935,7 +946,8 @@ export function registerWikiCli( .command("search") .description("Search wiki pages and, when configured, the active memory corpus") .argument("", "Search query") - .option("--max-results ", "Maximum results", (value: string) => Number(value)), + .option("--max-results ", "Maximum results", (value: string) => Number(value)) + .option("--mode ", `Search mode (${WIKI_SEARCH_MODES.join(", ")})`), ) .option("--json", "Print JSON") .action(async (query: string, opts: WikiSearchCommandOptions) => { @@ -946,6 +958,7 @@ export function registerWikiCli( maxResults: opts.maxResults, searchBackend: opts.backend, searchCorpus: opts.corpus, + mode: opts.mode, json: opts.json, }); }); diff --git a/extensions/memory-wiki/src/compile.test.ts b/extensions/memory-wiki/src/compile.test.ts index df1fe4e1d6f..4a3bc8bf1c5 100644 --- a/extensions/memory-wiki/src/compile.test.ts +++ b/extensions/memory-wiki/src/compile.test.ts @@ -353,6 +353,101 @@ describe("compileMemoryWikiVault", () => { await expect(fs.access(path.join(rootDir, "reports", "open-questions.md"))).rejects.toThrow(); }); + it("writes agent directory, relationship, provenance, and privacy reports", async () => { + const { rootDir, config } = await createVault({ + rootDir: nextCaseRoot(), + initialize: true, + }); + + await fs.writeFile( + path.join(rootDir, "entities", "brad.md"), + renderWikiMarkdown({ + frontmatter: { + pageType: "entity", + entityType: "person", + id: "entity.brad", + title: "Brad Groux", + canonicalId: "maintainer.brad-groux", + aliases: ["brad"], + privacyTier: "local-private", + bestUsedFor: ["Microsoft routing"], + lastRefreshedAt: "2026-04-29T00:00:00.000Z", + personCard: { + handles: ["@bgroux"], + lane: "Microsoft Teams", + askFor: ["Teams and Azure questions"], + privacyTier: "confirm-before-use", + }, + relationships: [ + { + targetId: "entity.alice", + targetTitle: "Alice", + kind: "collaborates-with", + evidenceKind: "discrawl-stat", + privacyTier: "local-private", + }, + ], + claims: [ + { + id: "claim.brad.teams", + text: "Brad is useful for Microsoft Teams routing.", + status: "supported", + confidence: 0.9, + evidence: [ + { + kind: "maintainer-whois", + sourceId: "source.maintainers", + privacyTier: "local-private", + }, + ], + }, + ], + }, + body: "# Brad Groux\n", + }), + "utf8", + ); + + await compileMemoryWikiVault(config); + + await expect( + fs.readFile(path.join(rootDir, "reports", "person-agent-directory.md"), "utf8"), + ).resolves.toContain("Microsoft Teams"); + await expect( + fs.readFile(path.join(rootDir, "reports", "relationship-graph.md"), "utf8"), + ).resolves.toContain("collaborates-with"); + await expect( + fs.readFile(path.join(rootDir, "reports", "provenance-coverage.md"), "utf8"), + ).resolves.toContain("maintainer-whois: 1"); + await expect( + fs.readFile(path.join(rootDir, "reports", "privacy-review.md"), "utf8"), + ).resolves.toContain("confirm-before-use"); + + const agentDigest = JSON.parse( + await fs.readFile(path.join(rootDir, ".openclaw-wiki", "cache", "agent-digest.json"), "utf8"), + ) as { + pages: Array<{ + path: string; + canonicalId?: string; + aliases?: string[]; + personCard?: { lane?: string }; + relationshipCount?: number; + }>; + }; + expect(agentDigest.pages).toContainEqual( + expect.objectContaining({ + path: "entities/brad.md", + canonicalId: "maintainer.brad-groux", + aliases: ["brad"], + personCard: expect.objectContaining({ lane: "Microsoft Teams" }), + relationshipCount: 1, + }), + ); + await expect( + fs.readFile(path.join(rootDir, ".openclaw-wiki", "cache", "claims.jsonl"), "utf8"), + ).resolves.toContain('"evidenceKinds":["maintainer-whois"]'); + }); + it("ignores generated related links when computing backlinks on repeated compile", async () => { const { rootDir, config } = await createVault({ rootDir: nextCaseRoot(), diff --git a/extensions/memory-wiki/src/compile.ts b/extensions/memory-wiki/src/compile.ts index 97948006d74..90e0fddb4e4 100644 --- a/extensions/memory-wiki/src/compile.ts +++ b/extensions/memory-wiki/src/compile.ts @@ -28,8 +28,10 @@ import { renderWikiMarkdown, toWikiPageSummary, type WikiClaim, + type WikiClaimEvidence, type WikiPageKind, type WikiPageSummary, + type WikiRelationship, WIKI_RELATED_END_MARKER, WIKI_RELATED_START_MARKER, } from "./markdown.js"; @@ -218,6 +220,106 @@ const DASHBOARD_PAGES: DashboardPageDefinition[] = [ ].join("\n"); }, }, + { + id: "report.person-agent-directory", + title: "Person Agent Directory", + relativePath: "reports/person-agent-directory.md", + buildBody: ({ config, pages, now }) => { + const matches = pages + .filter((page) => page.kind !== "report" && isPersonLikePage(page)) + .toSorted((left, right) => left.title.localeCompare(right.title)); + if (matches.length === 0) { + return "- No person-like entity pages with agent cards yet."; + } + const lines = [`- People with routing metadata: ${matches.length}`]; + for (const page of matches) { + const freshness = assessPageFreshness(page, now); + lines.push(`- ${formatPersonDirectoryLine(config, page, freshness)}`); + } + return lines.join("\n"); + }, + }, + { + id: "report.relationship-graph", + title: "Relationship Graph", + relativePath: "reports/relationship-graph.md", + buildBody: ({ config, pages }) => { + const relationships = pages + .flatMap((page) => page.relationships.map((relationship) => ({ page, relationship }))) + .toSorted((left, right) => { + const leftTitle = left.relationship.targetTitle ?? left.relationship.targetId ?? ""; + const rightTitle = right.relationship.targetTitle ?? right.relationship.targetId ?? ""; + return `${left.page.title} ${leftTitle}`.localeCompare( + `${right.page.title} ${rightTitle}`, + ); + }); + if (relationships.length === 0) { + return "- No structured relationships yet."; + } + return [ + `- Structured relationships: ${relationships.length}`, + "", + ...relationships.map( + ({ page, relationship }) => `- ${formatRelationshipLine(config, page, relationship)}`, + ), + ].join("\n"); + }, + }, + { + id: "report.provenance-coverage", + title: "Provenance Coverage", + relativePath: "reports/provenance-coverage.md", + buildBody: ({ config, pages }) => { + const evidenceEntries = pages.flatMap((page) => + page.claims.flatMap((claim) => + claim.evidence.map((evidence) => ({ page, claim, evidence })), + ), + ); + const missingEvidence = pages.flatMap((page) => + page.claims + .filter((claim) => claim.evidence.length === 0) + .map((claim) => ({ page, claim })), + ); + if (evidenceEntries.length === 0 && missingEvidence.length === 0) { + return "- No structured claims with provenance coverage yet."; + } + const kindCounts = countBy( + evidenceEntries.map(({ evidence }) => evidence.kind ?? "unspecified"), + ); + const sourceCounts = countBy( + evidenceEntries.map(({ evidence }) => evidence.sourceId ?? evidence.path ?? "inline"), + ); + const lines = [ + `- Evidence entries: ${evidenceEntries.length}`, + `- Claims missing evidence: ${missingEvidence.length}`, + "", + "### Evidence Classes", + ...formatCountLines(kindCounts), + "", + "### Top Evidence Sources", + ...formatCountLines(sourceCounts).slice(0, 20), + ]; + if (missingEvidence.length > 0) { + lines.push("", "### Missing Evidence"); + for (const { page, claim } of missingEvidence) { + lines.push(`- ${formatPageLink(config, page)}: ${formatClaimIdentityForPage(claim)}`); + } + } + return lines.join("\n"); + }, + }, + { + id: "report.privacy-review", + title: "Privacy Review", + relativePath: "reports/privacy-review.md", + buildBody: ({ config, pages }) => { + const entries = collectPrivacyReviewEntries(config, pages); + if (entries.length === 0) { + return "- No non-public privacy tiers flagged right now."; + } + return [`- Privacy review entries: ${entries.length}`, "", ...entries].join("\n"); + }, + }, ]; export type CompileMemoryWikiResult = { @@ -294,6 +396,165 @@ function formatFreshnessLabel(freshness: WikiFreshness): string { throw new Error("Unsupported wiki freshness level"); } +function formatListPreview(values: readonly string[], maxItems = 3): string | null { + if (values.length === 0) { + return null; + } + const shown = values.slice(0, maxItems).join(", "); + return values.length > maxItems ? `${shown}, +${values.length - maxItems}` : shown; +} + +function formatMaybeDetail(label: string, value: string | null | undefined): string | null { + return value ? `${label} ${value}` : null; +} + +function isPersonLikePage(page: WikiPageSummary): boolean { + const entityType = normalizeLowercaseStringOrEmpty(page.entityType); + const pageType = normalizeLowercaseStringOrEmpty(page.pageType); + return ( + Boolean(page.personCard) || + entityType === "person" || + entityType === "maintainer" || + pageType === "person" || + pageType === "maintainer" + ); +} + +function formatPersonDirectoryLine( + config: ResolvedMemoryWikiConfig, + page: WikiPageSummary, + freshness: WikiFreshness, +): string { + const card = page.personCard; + const details = [ + formatMaybeDetail("id", page.canonicalId ?? card?.canonicalId ?? page.id), + formatMaybeDetail("aliases", formatListPreview(page.aliases)), + formatMaybeDetail("handles", formatListPreview(card?.handles ?? [])), + formatMaybeDetail("lane", card?.lane), + formatMaybeDetail("ask", formatListPreview(card?.askFor ?? [])), + formatMaybeDetail( + "best", + formatListPreview([...page.bestUsedFor, ...(card?.bestUsedFor ?? [])]), + ), + formatMaybeDetail("privacy", page.privacyTier ?? card?.privacyTier), + formatMaybeDetail("refreshed", page.lastRefreshedAt ?? card?.lastRefreshedAt), + formatMaybeDetail("freshness", formatFreshnessLabel(freshness)), + ].filter(Boolean); + return `${formatPageLink(config, page)}${details.length > 0 ? `: ${details.join("; ")}` : ""}`; +} + +function formatRelationshipTarget( + config: ResolvedMemoryWikiConfig, + relationship: WikiRelationship, +) { + if (relationship.targetPath && relationship.targetTitle) { + return formatWikiLink({ + renderMode: config.vault.renderMode, + relativePath: relationship.targetPath, + title: relationship.targetTitle, + }); + } + return relationship.targetTitle ?? relationship.targetId ?? relationship.targetPath ?? "unknown"; +} + +function formatRelationshipLine( + config: ResolvedMemoryWikiConfig, + page: WikiPageSummary, + relationship: WikiRelationship, +): string { + const details = [ + relationship.kind ?? "related", + typeof relationship.weight === "number" ? `weight ${relationship.weight.toFixed(2)}` : null, + typeof relationship.confidence === "number" + ? `confidence ${relationship.confidence.toFixed(2)}` + : null, + relationship.evidenceKind ? `evidence ${relationship.evidenceKind}` : null, + relationship.privacyTier ? `privacy ${relationship.privacyTier}` : null, + relationship.note, + ].filter(Boolean); + return `${formatPageLink(config, page)} -> ${formatRelationshipTarget(config, relationship)}${ + details.length > 0 ? ` (${details.join(", ")})` : "" + }`; +} + +function countBy(values: readonly string[]): Map { + const counts = new Map(); + for (const value of values) { + counts.set(value, (counts.get(value) ?? 0) + 1); + } + return counts; +} + +function formatCountLines(counts: Map): string[] { + const lines = [...counts] + .toSorted((left, right) => { + if (left[1] !== right[1]) { + return right[1] - left[1]; + } + return left[0].localeCompare(right[0]); + }) + .map(([label, count]) => `- ${label}: ${count}`); + return lines.length > 0 ? lines : ["- None"]; +} + +function formatClaimIdentityForPage(claim: Pick): string { + return claim.id ? `\`${claim.id}\`: ${claim.text}` : claim.text; +} + +function isReviewablePrivacyTier(value: string | undefined): boolean { + const tier = normalizeLowercaseStringOrEmpty(value); + return tier !== "" && tier !== "public"; +} + +function formatEvidencePrivacyDetails(evidence: WikiClaimEvidence): string { + return [ + evidence.kind ? `kind ${evidence.kind}` : null, + evidence.sourceId ? `source ${evidence.sourceId}` : null, + evidence.path ? `path ${evidence.path}` : null, + evidence.lines ? `lines ${evidence.lines}` : null, + ] + .filter(Boolean) + .join(", "); +} + +function collectPrivacyReviewEntries( + config: ResolvedMemoryWikiConfig, + pages: WikiPageSummary[], +): string[] { + const entries: string[] = []; + for (const page of pages) { + if (isReviewablePrivacyTier(page.privacyTier)) { + entries.push(`- ${formatPageLink(config, page)}: page privacy ${page.privacyTier}`); + } + if (isReviewablePrivacyTier(page.personCard?.privacyTier)) { + entries.push( + `- ${formatPageLink(config, page)}: person card privacy ${page.personCard?.privacyTier}`, + ); + } + for (const relationship of page.relationships) { + if (isReviewablePrivacyTier(relationship.privacyTier)) { + entries.push( + `- ${formatPageLink(config, page)}: relationship privacy ${ + relationship.privacyTier + } -> ${formatRelationshipTarget(config, relationship)}`, + ); + } + } + for (const claim of page.claims) { + for (const evidence of claim.evidence) { + if (!isReviewablePrivacyTier(evidence.privacyTier)) { + continue; + } + const detail = formatEvidencePrivacyDetails(evidence); + entries.push( + `- ${formatPageLink(config, page)}: evidence privacy ${evidence.privacyTier} on ${formatClaimIdentityForPage(claim)}${detail ? ` (${detail})` : ""}`, + ); + } + } + } + return entries; +} + function formatClaimIdentity(claim: WikiClaimHealth): string { return claim.claimId ? `\`${claim.claimId}\`: ${claim.text}` : claim.text; } @@ -740,12 +1001,23 @@ type AgentDigestPage = { title: string; kind: WikiPageKind; path: string; + pageType?: string; + entityType?: string; + canonicalId?: string; + aliases: string[]; sourceIds: string[]; questions: string[]; contradictions: string[]; confidence?: number; + privacyTier?: string; + personCard?: WikiPageSummary["personCard"]; + bestUsedFor: string[]; + notEnoughFor: string[]; + relationshipCount: number; + topRelationships: WikiRelationship[]; freshnessLevel: WikiFreshnessLevel; lastTouchedAt?: string; + lastRefreshedAt?: string; claimCount: number; topClaims: AgentDigestClaim[]; }; @@ -878,13 +1150,24 @@ function buildAgentDigest(params: { title: page.title, kind: page.kind, path: page.relativePath, + aliases: [...page.aliases], sourceIds: [...page.sourceIds], questions: [...page.questions], contradictions: [...page.contradictions], + bestUsedFor: [...page.bestUsedFor], + notEnoughFor: [...page.notEnoughFor], + relationshipCount: page.relationships.length, + topRelationships: page.relationships.slice(0, 5), }, + page.pageType ? { pageType: page.pageType } : {}, + page.entityType ? { entityType: page.entityType } : {}, + page.canonicalId ? { canonicalId: page.canonicalId } : {}, typeof page.confidence === "number" ? { confidence: page.confidence } : {}, + page.privacyTier ? { privacyTier: page.privacyTier } : {}, + page.personCard ? { personCard: page.personCard } : {}, { freshnessLevel: pageFreshness.level }, pageFreshness.lastTouchedAt ? { lastTouchedAt: pageFreshness.lastTouchedAt } : {}, + page.lastRefreshedAt ? { lastRefreshedAt: page.lastRefreshedAt } : {}, { claimCount: page.claims.length, topClaims: sortClaims(page) @@ -931,10 +1214,24 @@ function buildClaimsDigestLines(params: { pages: WikiPageSummary[] }): string[] pageTitle: page.title, pageKind: page.kind, pagePath: page.relativePath, + pageType: page.pageType, + entityType: page.entityType, + canonicalId: page.canonicalId, + aliases: page.aliases, text: claim.text, status: normalizeClaimStatus(claim.status), confidence: claim.confidence, sourceIds: page.sourceIds, + evidenceKinds: [...new Set(claim.evidence.flatMap((entry) => entry.kind ?? []))], + privacyTiers: [ + ...new Set( + [ + page.privacyTier, + page.personCard?.privacyTier, + ...claim.evidence.map((entry) => entry.privacyTier), + ].flatMap((entry) => entry ?? []), + ), + ], evidenceCount: claim.evidence.length, missingEvidence: claim.evidence.length === 0, evidence: claim.evidence, diff --git a/extensions/memory-wiki/src/gateway.test.ts b/extensions/memory-wiki/src/gateway.test.ts index 5003a409479..d30615afa16 100644 --- a/extensions/memory-wiki/src/gateway.test.ts +++ b/extensions/memory-wiki/src/gateway.test.ts @@ -54,6 +54,7 @@ vi.mock("./obsidian.js", () => ({ vi.mock("./query.js", () => ({ getMemoryWikiPage: vi.fn(), searchMemoryWiki: vi.fn(), + WIKI_SEARCH_MODES: ["auto", "find-person", "route-question", "source-evidence", "raw-claim"], })); vi.mock("./source-sync.js", () => ({ @@ -356,6 +357,42 @@ describe("memory-wiki gateway methods", () => { ); }); + it("forwards wiki.search mode and corpus options over the gateway", async () => { + const { config } = await createVault({ prefix: "memory-wiki-gateway-" }); + const { api, registerGatewayMethod } = createPluginApi(); + + registerMemoryWikiGatewayMethods({ api, config }); + const handler = findGatewayHandler(registerGatewayMethod, "wiki.search"); + if (!handler) { + throw new Error("wiki.search handler missing"); + } + const respond = vi.fn(); + + await handler({ + params: { + query: "Teams Azure", + maxResults: 3, + corpus: "wiki", + backend: "local", + mode: "route-question", + }, + respond, + }); + + expect(searchMemoryWiki).toHaveBeenCalledWith( + expect.objectContaining({ + config, + appConfig: undefined, + query: "Teams Azure", + maxResults: 3, + searchBackend: "local", + searchCorpus: "wiki", + mode: "route-question", + }), + ); + expect(respond).toHaveBeenCalledWith(true, expect.anything()); + }); + it("forwards ingest requests over the gateway", async () => { const { config } = await createVault({ prefix: "memory-wiki-gateway-" }); const { api, registerGatewayMethod } = createPluginApi(); diff --git a/extensions/memory-wiki/src/gateway.ts b/extensions/memory-wiki/src/gateway.ts index 5cf93922b56..3267fc0e7db 100644 --- a/extensions/memory-wiki/src/gateway.ts +++ b/extensions/memory-wiki/src/gateway.ts @@ -19,7 +19,7 @@ import { runObsidianOpen, runObsidianSearch, } from "./obsidian.js"; -import { getMemoryWikiPage, searchMemoryWiki } from "./query.js"; +import { getMemoryWikiPage, searchMemoryWiki, WIKI_SEARCH_MODES } from "./query.js"; import { syncMemoryWikiImportedSources } from "./source-sync.js"; import { buildMemoryWikiDoctorReport, resolveMemoryWikiStatus } from "./status.js"; import { initializeMemoryWikiVault } from "./vault.js"; @@ -277,6 +277,7 @@ export function registerMemoryWikiGatewayMethods(params: { const maxResults = readNumberParam(requestParams, "maxResults"); const searchBackend = readEnumParam(requestParams, "backend", WIKI_SEARCH_BACKENDS); const searchCorpus = readEnumParam(requestParams, "corpus", WIKI_SEARCH_CORPORA); + const mode = readEnumParam(requestParams, "mode", WIKI_SEARCH_MODES); respond( true, await searchMemoryWiki({ @@ -286,6 +287,7 @@ export function registerMemoryWikiGatewayMethods(params: { maxResults, searchBackend, searchCorpus, + mode, }), ); } catch (error) { diff --git a/extensions/memory-wiki/src/markdown.test.ts b/extensions/memory-wiki/src/markdown.test.ts index 39d5f95f838..d78a5c0d803 100644 --- a/extensions/memory-wiki/src/markdown.test.ts +++ b/extensions/memory-wiki/src/markdown.test.ts @@ -1,6 +1,11 @@ import { createHash } from "node:crypto"; import { describe, expect, it } from "vitest"; -import { createWikiPageFilename, slugifyWikiSegment } from "./markdown.js"; +import { + createWikiPageFilename, + renderWikiMarkdown, + slugifyWikiSegment, + toWikiPageSummary, +} from "./markdown.js"; describe("slugifyWikiSegment", () => { it("preserves Unicode letters and numbers in wiki slugs", () => { @@ -40,3 +45,103 @@ describe("slugifyWikiSegment", () => { expect(createWikiPageFilename(stem)).toBe(fileName); }); }); + +describe("toWikiPageSummary", () => { + it("normalizes agent-facing people wiki metadata", () => { + const raw = renderWikiMarkdown({ + frontmatter: { + pageType: "entity", + entityType: "person", + id: "entity.brad", + title: "Brad Groux", + canonicalId: "maintainer.brad-groux", + aliases: ["brad", "bgroux"], + privacyTier: "local-private", + bestUsedFor: ["Microsoft ecosystem routing"], + notEnoughFor: ["legal approval"], + lastRefreshedAt: "2026-04-29T00:00:00.000Z", + personCard: { + handles: ["@bgroux"], + socials: ["https://x.example/bgroux"], + email: "brad@example.com", + timezone: "America/Chicago", + lane: "Microsoft Teams", + askFor: ["Teams and Azure questions"], + avoidAskingFor: ["unrelated billing"], + confidence: 0.8, + privacyTier: "confirm-before-use", + lastRefreshedAt: "2026-04-28T00:00:00.000Z", + }, + relationships: [ + { + targetId: "entity.alice", + targetTitle: "Alice", + kind: "collaborates-with", + weight: 0.7, + confidence: 0.6, + evidenceKind: "discrawl-stat", + privacyTier: "local-private", + }, + ], + claims: [ + { + id: "claim.brad.teams", + text: "Brad is useful for Microsoft Teams routing.", + confidence: 0.9, + evidence: [ + { + kind: "maintainer-whois", + sourceId: "source.maintainers", + confidence: 0.8, + privacyTier: "local-private", + }, + ], + }, + ], + }, + body: "# Brad Groux\n", + }); + + const summary = toWikiPageSummary({ + absolutePath: "/tmp/wiki/entities/brad.md", + relativePath: "entities/brad.md", + raw, + }); + + expect(summary).toEqual( + expect.objectContaining({ + entityType: "person", + canonicalId: "maintainer.brad-groux", + aliases: ["brad", "bgroux"], + privacyTier: "local-private", + bestUsedFor: ["Microsoft ecosystem routing"], + notEnoughFor: ["legal approval"], + lastRefreshedAt: "2026-04-29T00:00:00.000Z", + personCard: expect.objectContaining({ + handles: ["@bgroux"], + emails: ["brad@example.com"], + lane: "Microsoft Teams", + privacyTier: "confirm-before-use", + }), + relationships: [ + expect.objectContaining({ + targetId: "entity.alice", + kind: "collaborates-with", + evidenceKind: "discrawl-stat", + }), + ], + claims: [ + expect.objectContaining({ + id: "claim.brad.teams", + evidence: [ + expect.objectContaining({ + kind: "maintainer-whois", + privacyTier: "local-private", + }), + ], + }), + ], + }), + ); + }); +}); diff --git a/extensions/memory-wiki/src/markdown.ts b/extensions/memory-wiki/src/markdown.ts index 73755711fab..52cbe1fac1d 100644 --- a/extensions/memory-wiki/src/markdown.ts +++ b/extensions/memory-wiki/src/markdown.ts @@ -19,10 +19,13 @@ export type ParsedWikiMarkdown = { }; export type WikiClaimEvidence = { + kind?: string; sourceId?: string; path?: string; lines?: string; weight?: number; + confidence?: number; + privacyTier?: string; note?: string; updatedAt?: string; }; @@ -36,6 +39,35 @@ export type WikiClaim = { updatedAt?: string; }; +export type WikiPersonCard = { + canonicalId?: string; + handles: string[]; + socials: string[]; + emails: string[]; + timezone?: string; + lane?: string; + askFor: string[]; + avoidAskingFor: string[]; + bestUsedFor: string[]; + notEnoughFor: string[]; + confidence?: number; + privacyTier?: string; + lastRefreshedAt?: string; +}; + +export type WikiRelationship = { + targetId?: string; + targetPath?: string; + targetTitle?: string; + kind?: string; + weight?: number; + confidence?: number; + evidenceKind?: string; + privacyTier?: string; + note?: string; + updatedAt?: string; +}; + export type WikiPageSummary = { absolutePath: string; relativePath: string; @@ -43,12 +75,20 @@ export type WikiPageSummary = { title: string; id?: string; pageType?: string; + entityType?: string; + canonicalId?: string; + aliases: string[]; sourceIds: string[]; linkTargets: string[]; claims: WikiClaim[]; contradictions: string[]; questions: string[]; confidence?: number; + privacyTier?: string; + personCard?: WikiPersonCard; + relationships: WikiRelationship[]; + bestUsedFor: string[]; + notEnoughFor: string[]; sourceType?: string; provenanceMode?: string; sourcePath?: string; @@ -56,6 +96,7 @@ export type WikiPageSummary = { bridgeWorkspaceDir?: string; unsafeLocalConfiguredPath?: string; unsafeLocalRelativePath?: string; + lastRefreshedAt?: string; updatedAt?: string; }; @@ -153,21 +194,37 @@ function normalizeWikiClaimEvidence(value: unknown): WikiClaimEvidence | null { return null; } const record = value as Record; + const kind = normalizeOptionalString(record.kind); const sourceId = normalizeOptionalString(record.sourceId); const evidencePath = normalizeOptionalString(record.path); const lines = normalizeOptionalString(record.lines); const note = normalizeOptionalString(record.note); const updatedAt = normalizeOptionalString(record.updatedAt); + const privacyTier = normalizeOptionalString(record.privacyTier); const weight = typeof record.weight === "number" && Number.isFinite(record.weight) ? record.weight : undefined; - if (!sourceId && !evidencePath && !lines && !note && weight === undefined && !updatedAt) { + const confidence = normalizeOptionalNumber(record.confidence); + if ( + !kind && + !sourceId && + !evidencePath && + !lines && + !note && + weight === undefined && + confidence === undefined && + !privacyTier && + !updatedAt + ) { return null; } return { + ...(kind ? { kind } : {}), ...(sourceId ? { sourceId } : {}), ...(evidencePath ? { path: evidencePath } : {}), ...(lines ? { lines } : {}), ...(weight !== undefined ? { weight } : {}), + ...(confidence !== undefined ? { confidence } : {}), + ...(privacyTier ? { privacyTier } : {}), ...(note ? { note } : {}), ...(updatedAt ? { updatedAt } : {}), }; @@ -213,6 +270,101 @@ export function normalizeWikiClaims(value: unknown): WikiClaim[] { }); } +function normalizeOptionalNumber(value: unknown): number | undefined { + return typeof value === "number" && Number.isFinite(value) ? value : undefined; +} + +function normalizeWikiPersonCard(value: unknown): WikiPersonCard | undefined { + if (!value || typeof value !== "object" || Array.isArray(value)) { + return undefined; + } + const record = value as Record; + const card: WikiPersonCard = { + ...(normalizeOptionalString(record.canonicalId) + ? { canonicalId: normalizeOptionalString(record.canonicalId) } + : {}), + handles: normalizeSingleOrTrimmedStringList(record.handles), + socials: normalizeSingleOrTrimmedStringList(record.socials), + emails: normalizeSingleOrTrimmedStringList(record.emails ?? record.email), + ...(normalizeOptionalString(record.timezone) + ? { timezone: normalizeOptionalString(record.timezone) } + : {}), + ...(normalizeOptionalString(record.lane) ? { lane: normalizeOptionalString(record.lane) } : {}), + askFor: normalizeSingleOrTrimmedStringList(record.askFor), + avoidAskingFor: normalizeSingleOrTrimmedStringList(record.avoidAskingFor), + bestUsedFor: normalizeSingleOrTrimmedStringList(record.bestUsedFor), + notEnoughFor: normalizeSingleOrTrimmedStringList(record.notEnoughFor), + ...(normalizeOptionalNumber(record.confidence) !== undefined + ? { confidence: normalizeOptionalNumber(record.confidence) } + : {}), + ...(normalizeOptionalString(record.privacyTier) + ? { privacyTier: normalizeOptionalString(record.privacyTier) } + : {}), + ...(normalizeOptionalString(record.lastRefreshedAt) + ? { lastRefreshedAt: normalizeOptionalString(record.lastRefreshedAt) } + : {}), + }; + const hasAnyValue = + Boolean( + card.canonicalId || card.timezone || card.lane || card.privacyTier || card.lastRefreshedAt, + ) || + typeof card.confidence === "number" || + card.handles.length > 0 || + card.socials.length > 0 || + card.emails.length > 0 || + card.askFor.length > 0 || + card.avoidAskingFor.length > 0 || + card.bestUsedFor.length > 0 || + card.notEnoughFor.length > 0; + return hasAnyValue ? card : undefined; +} + +function normalizeWikiRelationships(value: unknown): WikiRelationship[] { + if (!Array.isArray(value)) { + return []; + } + return value.flatMap((entry) => { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + return []; + } + const record = entry as Record; + const relationship: WikiRelationship = { + ...(normalizeOptionalString(record.targetId) + ? { targetId: normalizeOptionalString(record.targetId) } + : {}), + ...(normalizeOptionalString(record.targetPath) + ? { targetPath: normalizeOptionalString(record.targetPath) } + : {}), + ...(normalizeOptionalString(record.targetTitle) + ? { targetTitle: normalizeOptionalString(record.targetTitle) } + : {}), + ...(normalizeOptionalString(record.kind) + ? { kind: normalizeOptionalString(record.kind) } + : {}), + ...(normalizeOptionalNumber(record.weight) !== undefined + ? { weight: normalizeOptionalNumber(record.weight) } + : {}), + ...(normalizeOptionalNumber(record.confidence) !== undefined + ? { confidence: normalizeOptionalNumber(record.confidence) } + : {}), + ...(normalizeOptionalString(record.evidenceKind) + ? { evidenceKind: normalizeOptionalString(record.evidenceKind) } + : {}), + ...(normalizeOptionalString(record.privacyTier) + ? { privacyTier: normalizeOptionalString(record.privacyTier) } + : {}), + ...(normalizeOptionalString(record.note) + ? { note: normalizeOptionalString(record.note) } + : {}), + ...(normalizeOptionalString(record.updatedAt) + ? { updatedAt: normalizeOptionalString(record.updatedAt) } + : {}), + }; + const hasAnyValue = Object.keys(relationship).length > 0; + return hasAnyValue ? [relationship] : []; + }); +} + export function extractWikiLinks(markdown: string): string[] { const searchable = markdown.replace(RELATED_BLOCK_PATTERN, ""); const links: string[] = []; @@ -297,6 +449,9 @@ export function toWikiPageSummary(params: { title, id: normalizeOptionalString(parsed.frontmatter.id), pageType: normalizeOptionalString(parsed.frontmatter.pageType), + entityType: normalizeOptionalString(parsed.frontmatter.entityType), + canonicalId: normalizeOptionalString(parsed.frontmatter.canonicalId), + aliases: normalizeSingleOrTrimmedStringList(parsed.frontmatter.aliases), sourceIds: normalizeSourceIds(parsed.frontmatter.sourceIds), linkTargets: extractWikiLinks(params.raw), claims: normalizeWikiClaims(parsed.frontmatter.claims), @@ -307,6 +462,11 @@ export function toWikiPageSummary(params: { Number.isFinite(parsed.frontmatter.confidence) ? parsed.frontmatter.confidence : undefined, + privacyTier: normalizeOptionalString(parsed.frontmatter.privacyTier), + personCard: normalizeWikiPersonCard(parsed.frontmatter.personCard), + relationships: normalizeWikiRelationships(parsed.frontmatter.relationships), + bestUsedFor: normalizeSingleOrTrimmedStringList(parsed.frontmatter.bestUsedFor), + notEnoughFor: normalizeSingleOrTrimmedStringList(parsed.frontmatter.notEnoughFor), sourceType: normalizeOptionalString(parsed.frontmatter.sourceType), provenanceMode: normalizeOptionalString(parsed.frontmatter.provenanceMode), sourcePath: normalizeOptionalString(parsed.frontmatter.sourcePath), @@ -316,6 +476,7 @@ export function toWikiPageSummary(params: { parsed.frontmatter.unsafeLocalConfiguredPath, ), unsafeLocalRelativePath: normalizeOptionalString(parsed.frontmatter.unsafeLocalRelativePath), + lastRefreshedAt: normalizeOptionalString(parsed.frontmatter.lastRefreshedAt), updatedAt: normalizeOptionalString(parsed.frontmatter.updatedAt), }; } diff --git a/extensions/memory-wiki/src/query.test.ts b/extensions/memory-wiki/src/query.test.ts index 49d3e66da16..b6af9cccf23 100644 --- a/extensions/memory-wiki/src/query.test.ts +++ b/extensions/memory-wiki/src/query.test.ts @@ -205,6 +205,114 @@ describe("searchMemoryWiki", () => { expect(results[0]?.snippet).toContain("Teams"); }); + it("supports people-routing search modes and claim evidence drilldown metadata", async () => { + const { rootDir, config } = await createQueryVault({ + initialize: true, + }); + await fs.writeFile( + path.join(rootDir, "entities", "brad.md"), + renderWikiMarkdown({ + frontmatter: { + pageType: "entity", + entityType: "person", + id: "entity.brad", + title: "Brad Groux", + canonicalId: "maintainer.brad-groux", + aliases: ["bgroux"], + privacyTier: "local-private", + personCard: { + handles: ["@bgroux"], + lane: "Microsoft Teams", + askFor: ["Teams and Azure rollout questions"], + }, + bestUsedFor: ["Microsoft ecosystem routing"], + relationships: [ + { + targetId: "entity.alice", + targetTitle: "Alice", + kind: "works-with", + note: "Teams escalation buddy", + }, + ], + claims: [ + { + id: "claim.brad.teams", + text: "Brad is a strong route for Microsoft Teams questions.", + status: "supported", + confidence: 0.88, + evidence: [ + { + kind: "maintainer-whois", + sourceId: "source.maintainers", + privacyTier: "local-private", + }, + ], + }, + ], + }, + body: "# Brad Groux\n\nAgent card summary.\n", + }), + "utf8", + ); + await fs.writeFile( + path.join(rootDir, "sources", "maintainers.md"), + renderWikiMarkdown({ + frontmatter: { + pageType: "source", + id: "source.maintainers", + title: "Maintainers Source", + }, + body: "# Maintainers Source\n\nmaintainer-whois Teams sample.\n", + }), + "utf8", + ); + + const personResults = await searchMemoryWiki({ + config, + query: "bgroux", + mode: "find-person", + }); + expect(personResults[0]).toEqual( + expect.objectContaining({ + path: "entities/brad.md", + canonicalId: "maintainer.brad-groux", + aliases: ["bgroux"], + privacyTier: "local-private", + searchMode: "find-person", + }), + ); + + const routeResults = await searchMemoryWiki({ + config, + query: "Teams Azure", + mode: "route-question", + }); + expect(routeResults[0]?.path).toBe("entities/brad.md"); + + const claimResults = await searchMemoryWiki({ + config, + query: "strong route Teams", + mode: "raw-claim", + }); + expect(claimResults[0]).toEqual( + expect.objectContaining({ + path: "entities/brad.md", + matchedClaimId: "claim.brad.teams", + matchedClaimConfidence: 0.88, + evidenceKinds: ["maintainer-whois"], + evidenceSourceIds: ["source.maintainers"], + }), + ); + + const evidenceResults = await searchMemoryWiki({ + config, + query: "maintainer-whois", + mode: "source-evidence", + maxResults: 2, + }); + expect(evidenceResults.map((result) => result.path)).toContain("sources/maintainers.md"); + }); + it("uses body text instead of frontmatter for fallback snippets", async () => { const { rootDir, config } = await createQueryVault({ initialize: true, diff --git a/extensions/memory-wiki/src/query.ts b/extensions/memory-wiki/src/query.ts index 5febc64f1fe..9ced9917a3c 100644 --- a/extensions/memory-wiki/src/query.ts +++ b/extensions/memory-wiki/src/query.ts @@ -22,14 +22,33 @@ const RELATED_BLOCK_PATTERN = /[\s\S]*?/g; const MARKDOWN_FRONTMATTER_PATTERN = /^\s*---\r?\n[\s\S]*?\r?\n---\r?\n?/; +export const WIKI_SEARCH_MODES = [ + "auto", + "find-person", + "route-question", + "source-evidence", + "raw-claim", +] as const; + +export type WikiSearchMode = (typeof WIKI_SEARCH_MODES)[number]; + type QueryDigestPage = { id?: string; title: string; kind: WikiPageSummary["kind"]; path: string; + pageType?: string; + entityType?: string; + canonicalId?: string; + aliases?: string[]; sourceIds: string[]; questions: string[]; contradictions: string[]; + privacyTier?: string; + personCard?: WikiPageSummary["personCard"]; + bestUsedFor?: string[]; + notEnoughFor?: string[]; + relationshipCount?: number; }; type QueryDigestClaim = { @@ -38,10 +57,16 @@ type QueryDigestClaim = { pageTitle: string; pageKind: WikiPageSummary["kind"]; pagePath: string; + pageType?: string; + entityType?: string; + canonicalId?: string; + aliases?: string[]; text: string; status?: string; confidence?: number; sourceIds?: string[]; + evidenceKinds?: string[]; + privacyTiers?: string[]; freshnessLevel?: string; lastTouchedAt?: string; }; @@ -68,6 +93,16 @@ export type WikiSearchResult = { sourcePath?: string; provenanceLabel?: string; updatedAt?: string; + searchMode?: WikiSearchMode; + entityType?: string; + canonicalId?: string; + aliases?: string[]; + privacyTier?: string; + matchedClaimId?: string; + matchedClaimStatus?: string; + matchedClaimConfidence?: number; + evidenceKinds?: string[]; + evidenceSourceIds?: string[]; }; export type WikiGetResult = { @@ -206,11 +241,50 @@ function buildPageSearchText(page: QueryableWikiPage): string { page.title, page.relativePath, page.id ?? "", + page.pageType ?? "", + page.entityType ?? "", + page.canonicalId ?? "", + page.aliases.join(" "), page.sourceIds.join(" "), page.questions.join(" "), page.contradictions.join(" "), + page.privacyTier ?? "", + page.bestUsedFor.join(" "), + page.notEnoughFor.join(" "), + page.personCard?.canonicalId ?? "", + page.personCard?.handles.join(" ") ?? "", + page.personCard?.socials.join(" ") ?? "", + page.personCard?.emails.join(" ") ?? "", + page.personCard?.timezone ?? "", + page.personCard?.lane ?? "", + page.personCard?.askFor.join(" ") ?? "", + page.personCard?.avoidAskingFor.join(" ") ?? "", + page.personCard?.bestUsedFor.join(" ") ?? "", + page.personCard?.notEnoughFor.join(" ") ?? "", + page.relationships + .flatMap((relationship) => [ + relationship.targetId ?? "", + relationship.targetPath ?? "", + relationship.targetTitle ?? "", + relationship.kind ?? "", + relationship.evidenceKind ?? "", + relationship.note ?? "", + ]) + .join(" "), page.claims.map((claim) => claim.text).join(" "), page.claims.map((claim) => claim.id ?? "").join(" "), + page.claims + .flatMap((claim) => + claim.evidence.flatMap((evidence) => [ + evidence.kind ?? "", + evidence.sourceId ?? "", + evidence.path ?? "", + evidence.lines ?? "", + evidence.note ?? "", + evidence.privacyTier ?? "", + ]), + ) + .join(" "), ] .filter(Boolean) .join("\n"); @@ -247,11 +321,30 @@ function buildDigestPageSearchText(page: QueryDigestPage, claims: QueryDigestCla page.title, page.path, page.id ?? "", + page.pageType ?? "", + page.entityType ?? "", + page.canonicalId ?? "", + page.aliases?.join(" ") ?? "", page.sourceIds.join(" "), page.questions.join(" "), page.contradictions.join(" "), + page.privacyTier ?? "", + page.bestUsedFor?.join(" ") ?? "", + page.notEnoughFor?.join(" ") ?? "", + page.personCard?.canonicalId ?? "", + page.personCard?.handles.join(" ") ?? "", + page.personCard?.socials.join(" ") ?? "", + page.personCard?.emails.join(" ") ?? "", + page.personCard?.timezone ?? "", + page.personCard?.lane ?? "", + page.personCard?.askFor.join(" ") ?? "", + page.personCard?.avoidAskingFor.join(" ") ?? "", + page.personCard?.bestUsedFor.join(" ") ?? "", + page.personCard?.notEnoughFor.join(" ") ?? "", claims.map((claim) => claim.text).join(" "), claims.map((claim) => claim.id ?? "").join(" "), + claims.map((claim) => claim.evidenceKinds?.join(" ") ?? "").join(" "), + claims.map((claim) => claim.privacyTiers?.join(" ") ?? "").join(" "), ] .filter(Boolean) .join("\n"); @@ -260,11 +353,13 @@ function buildDigestPageSearchText(page: QueryDigestPage, claims: QueryDigestCla function isClaimTextOrIdMatch( claim: Pick | Pick, queryLower: string, + queryTokens: readonly string[] = buildQueryTokens(queryLower), ): boolean { - if (normalizeLowercaseStringOrEmpty(claim.text).includes(queryLower)) { + const textLower = normalizeLowercaseStringOrEmpty(claim.text); + if (lineMatchesQuery(textLower, queryLower, [...queryTokens])) { return true; } - return normalizeLowercaseStringOrEmpty(claim.id).includes(queryLower); + return lineMatchesQuery(normalizeLowercaseStringOrEmpty(claim.id), queryLower, [...queryTokens]); } function scoreClaimMatch(params: { @@ -274,10 +369,18 @@ function scoreClaimMatch(params: { status?: string; freshnessLevel?: string; queryLower: string; + queryTokens?: readonly string[]; }): number { let score = 0; if (normalizeLowercaseStringOrEmpty(params.text).includes(params.queryLower)) { score += 25; + } else if ( + params.queryTokens?.length && + params.queryTokens.every((token) => + normalizeLowercaseStringOrEmpty(params.text).includes(token), + ) + ) { + score += 18; } if (normalizeLowercaseStringOrEmpty(params.id).includes(params.queryLower)) { score += 10; @@ -313,6 +416,7 @@ function scoreDigestClaimMatch(claim: QueryDigestClaim, queryLower: string): num status: claim.status, freshnessLevel: claim.freshnessLevel, queryLower, + queryTokens: buildQueryTokens(queryLower), }); } @@ -348,12 +452,212 @@ function scoreWikiMetadataMatch(params: { return score; } +function hasQueryMatch( + value: string | undefined, + queryLower: string, + queryTokens: readonly string[], +) { + const normalized = normalizeLowercaseStringOrEmpty(value); + return lineMatchesQuery(normalized, queryLower, [...queryTokens]); +} + +function hasAnyQueryMatch( + values: readonly (string | undefined)[], + queryLower: string, + queryTokens: readonly string[], +) { + return values.some((value) => hasQueryMatch(value, queryLower, queryTokens)); +} + +function isPersonLikeSummary( + page: Pick, +): boolean { + const entityType = normalizeLowercaseStringOrEmpty(page.entityType); + const pageType = normalizeLowercaseStringOrEmpty(page.pageType); + return ( + Boolean(page.personCard) || + entityType === "person" || + entityType === "maintainer" || + pageType === "person" || + pageType === "maintainer" + ); +} + +function scorePageSearchModeBoost(params: { + page: QueryableWikiPage; + matchingClaims: readonly WikiClaim[]; + queryLower: string; + queryTokens: readonly string[]; + mode: WikiSearchMode; +}): number { + const { page, queryLower, queryTokens } = params; + switch (params.mode) { + case "auto": + return 0; + case "find-person": { + let score = isPersonLikeSummary(page) ? 24 : -4; + if ( + hasAnyQueryMatch( + [ + page.canonicalId, + ...page.aliases, + page.personCard?.canonicalId, + ...(page.personCard?.handles ?? []), + ...(page.personCard?.emails ?? []), + ...(page.personCard?.socials ?? []), + ], + queryLower, + queryTokens, + ) + ) { + score += 24; + } + return score; + } + case "route-question": { + let score = isPersonLikeSummary(page) ? 14 : 0; + if ( + hasAnyQueryMatch( + [ + page.personCard?.lane, + ...(page.personCard?.askFor ?? []), + ...(page.personCard?.avoidAskingFor ?? []), + ...page.bestUsedFor, + ...page.notEnoughFor, + ...(page.personCard?.bestUsedFor ?? []), + ...(page.personCard?.notEnoughFor ?? []), + ...page.relationships.flatMap((relationship) => [ + relationship.kind, + relationship.targetTitle, + relationship.note, + ]), + ], + queryLower, + queryTokens, + ) + ) { + score += 32; + } + score += Math.min(8, page.relationships.length * 2); + return score; + } + case "source-evidence": { + let score = page.kind === "source" ? 22 : 0; + if ( + hasAnyQueryMatch( + [ + page.sourcePath, + ...page.sourceIds, + ...page.claims.flatMap((claim) => + claim.evidence.flatMap((evidence) => [ + evidence.kind, + evidence.sourceId, + evidence.path, + evidence.lines, + evidence.note, + ]), + ), + ], + queryLower, + queryTokens, + ) + ) { + score += 30; + } + return score; + } + case "raw-claim": + return params.matchingClaims.length > 0 ? 42 : 0; + } +} + +function scoreDigestSearchModeBoost(params: { + page: QueryDigestPage; + claims: readonly QueryDigestClaim[]; + matchingClaims: readonly QueryDigestClaim[]; + queryLower: string; + queryTokens: readonly string[]; + mode: WikiSearchMode; +}): number { + const { page, queryLower, queryTokens } = params; + switch (params.mode) { + case "auto": + return 0; + case "find-person": { + let score = isPersonLikeSummary(page) ? 24 : -4; + if ( + hasAnyQueryMatch( + [ + page.canonicalId, + ...(page.aliases ?? []), + page.personCard?.canonicalId, + ...(page.personCard?.handles ?? []), + ...(page.personCard?.emails ?? []), + ...(page.personCard?.socials ?? []), + ], + queryLower, + queryTokens, + ) + ) { + score += 24; + } + return score; + } + case "route-question": { + let score = isPersonLikeSummary(page) ? 14 : 0; + if ( + hasAnyQueryMatch( + [ + page.personCard?.lane, + ...(page.personCard?.askFor ?? []), + ...(page.personCard?.avoidAskingFor ?? []), + ...(page.bestUsedFor ?? []), + ...(page.notEnoughFor ?? []), + ...(page.personCard?.bestUsedFor ?? []), + ...(page.personCard?.notEnoughFor ?? []), + ], + queryLower, + queryTokens, + ) + ) { + score += 32; + } + score += Math.min(8, (page.relationshipCount ?? 0) * 2); + return score; + } + case "source-evidence": { + let score = page.kind === "source" ? 22 : 0; + if ( + hasAnyQueryMatch( + [ + ...page.sourceIds, + ...params.claims.flatMap((claim) => [ + ...(claim.sourceIds ?? []), + ...(claim.evidenceKinds ?? []), + ...(claim.privacyTiers ?? []), + ]), + ], + queryLower, + queryTokens, + ) + ) { + score += 30; + } + return score; + } + case "raw-claim": + return params.matchingClaims.length > 0 ? 42 : 0; + } +} + function buildDigestCandidatePaths(params: { digest: QueryDigestBundle; query: string; maxResults: number; + mode: WikiSearchMode; }): string[] { const queryLower = normalizeLowercaseStringOrEmpty(params.query); + const queryTokens = buildQueryTokens(queryLower); const claimsByPage = new Map(); for (const claim of params.digest.claims) { const current = claimsByPage.get(claim.pagePath) ?? []; @@ -380,7 +684,7 @@ function buildDigestCandidatePaths(params: { queryLower, }); const matchingClaims = claims - .filter((claim) => isClaimTextOrIdMatch(claim, queryLower)) + .filter((claim) => isClaimTextOrIdMatch(claim, queryLower, queryTokens)) .toSorted( (left, right) => scoreDigestClaimMatch(right, queryLower) - scoreDigestClaimMatch(left, queryLower), @@ -389,6 +693,14 @@ function buildDigestCandidatePaths(params: { score += scoreDigestClaimMatch(matchingClaims[0], queryLower); score += Math.min(10, (matchingClaims.length - 1) * 2); } + score += scoreDigestSearchModeBoost({ + page, + claims, + matchingClaims, + queryLower, + queryTokens, + mode: params.mode, + }); return { path: page.path, score }; }) .filter((candidate) => candidate.score > 0) @@ -402,11 +714,20 @@ function buildDigestCandidatePaths(params: { .map((candidate) => candidate.path); } -function isClaimMatch(claim: WikiClaim, queryLower: string): boolean { - return isClaimTextOrIdMatch(claim, queryLower); +function isClaimMatch( + claim: WikiClaim, + queryLower: string, + queryTokens: readonly string[], +): boolean { + return isClaimTextOrIdMatch(claim, queryLower, queryTokens); } -function rankClaimMatch(page: QueryableWikiPage, claim: WikiClaim, queryLower: string): number { +function rankClaimMatch( + page: QueryableWikiPage, + claim: WikiClaim, + queryLower: string, + queryTokens: readonly string[], +): number { const freshness = assessClaimFreshness({ page, claim }); return scoreClaimMatch({ text: claim.text, @@ -415,15 +736,18 @@ function rankClaimMatch(page: QueryableWikiPage, claim: WikiClaim, queryLower: s status: claim.status, freshnessLevel: freshness.level, queryLower, + queryTokens, }); } function getMatchingClaims(page: QueryableWikiPage, queryLower: string): WikiClaim[] { + const queryTokens = buildQueryTokens(queryLower); return page.claims - .filter((claim) => isClaimMatch(claim, queryLower)) + .filter((claim) => isClaimMatch(claim, queryLower, queryTokens)) .toSorted( (left, right) => - rankClaimMatch(page, right, queryLower) - rankClaimMatch(page, left, queryLower), + rankClaimMatch(page, right, queryLower, queryTokens) - + rankClaimMatch(page, left, queryLower, queryTokens), ); } @@ -436,7 +760,7 @@ function buildPageSnippet(page: QueryableWikiPage, query: string): string { return buildSnippet(page.raw, query); } -function scorePage(page: QueryableWikiPage, query: string): number { +function scorePage(page: QueryableWikiPage, query: string, mode: WikiSearchMode): number { const queryLower = normalizeLowercaseStringOrEmpty(query); const queryTokens = buildQueryTokens(queryLower); const titleLower = normalizeLowercaseStringOrEmpty(page.title); @@ -468,9 +792,16 @@ function scorePage(page: QueryableWikiPage, query: string): number { }); const matchingClaims = getMatchingClaims(page, queryLower); if (matchingClaims.length > 0) { - score += rankClaimMatch(page, matchingClaims[0], queryLower); + score += rankClaimMatch(page, matchingClaims[0], queryLower, queryTokens); score += Math.min(10, (matchingClaims.length - 1) * 2); } + score += scorePageSearchModeBoost({ + page, + matchingClaims, + queryLower, + queryTokens, + mode, + }); const bodyOccurrences = rawLower.split(queryLower).length - 1; score += Math.min(10, bodyOccurrences); for (const token of queryTokens) { @@ -585,6 +916,10 @@ function buildWikiProvenanceLabel( | "bridgeRelativePath" | "unsafeLocalRelativePath" | "relativePath" + | "entityType" + | "canonicalId" + | "aliases" + | "privacyTier" >, ): string | undefined { if (page.sourceType === "memory-bridge-events") { @@ -610,11 +945,24 @@ function buildWikiResultMetadata( | "bridgeRelativePath" | "unsafeLocalRelativePath" | "relativePath" + | "entityType" + | "canonicalId" + | "aliases" + | "privacyTier" >, ): Partial< Pick< WikiSearchResult, - "id" | "sourceType" | "provenanceMode" | "sourcePath" | "provenanceLabel" | "updatedAt" + | "id" + | "sourceType" + | "provenanceMode" + | "sourcePath" + | "provenanceLabel" + | "updatedAt" + | "entityType" + | "canonicalId" + | "aliases" + | "privacyTier" > > { const provenanceLabel = buildWikiProvenanceLabel(page); @@ -625,22 +973,50 @@ function buildWikiResultMetadata( ...(page.sourcePath ? { sourcePath: page.sourcePath } : {}), ...(provenanceLabel ? { provenanceLabel } : {}), ...(page.updatedAt ? { updatedAt: page.updatedAt } : {}), + ...("entityType" in page && page.entityType ? { entityType: page.entityType } : {}), + ...("canonicalId" in page && page.canonicalId ? { canonicalId: page.canonicalId } : {}), + ...("aliases" in page && page.aliases.length > 0 ? { aliases: [...page.aliases] } : {}), + ...("privacyTier" in page && page.privacyTier ? { privacyTier: page.privacyTier } : {}), }; } -function toWikiSearchResult(page: QueryableWikiPage, query: string): WikiSearchResult { +function buildClaimResultMetadata(claim: WikiClaim | undefined): Partial { + if (!claim) { + return {}; + } + return { + ...(claim.id ? { matchedClaimId: claim.id } : {}), + ...(claim.status ? { matchedClaimStatus: claim.status } : {}), + ...(typeof claim.confidence === "number" ? { matchedClaimConfidence: claim.confidence } : {}), + evidenceKinds: [...new Set(claim.evidence.flatMap((evidence) => evidence.kind ?? []))], + evidenceSourceIds: [...new Set(claim.evidence.flatMap((evidence) => evidence.sourceId ?? []))], + }; +} + +function toWikiSearchResult( + page: QueryableWikiPage, + query: string, + mode: WikiSearchMode, +): WikiSearchResult { + const queryLower = normalizeLowercaseStringOrEmpty(query); + const matchingClaim = getMatchingClaims(page, queryLower)[0]; return { corpus: "wiki", path: page.relativePath, title: page.title, kind: page.kind, - score: scorePage(page, query), + score: scorePage(page, query, mode), snippet: buildPageSnippet(page, query), + searchMode: mode, ...buildWikiResultMetadata(page), + ...buildClaimResultMetadata(matchingClaim), }; } -function toMemoryWikiSearchResult(result: MemorySearchResult): WikiSearchResult { +function toMemoryWikiSearchResult( + result: MemorySearchResult, + mode: WikiSearchMode, +): WikiSearchResult { return { corpus: "memory", path: result.path, @@ -651,6 +1027,7 @@ function toMemoryWikiSearchResult(result: MemorySearchResult): WikiSearchResult startLine: result.startLine, endLine: result.endLine, memorySource: result.source, + searchMode: mode, ...(result.citation ? { citation: result.citation } : {}), }; } @@ -659,6 +1036,7 @@ async function searchWikiCorpus(params: { rootDir: string; query: string; maxResults: number; + mode: WikiSearchMode; }): Promise { const digest = await readQueryDigestBundle(params.rootDir); const candidatePaths = digest @@ -666,6 +1044,7 @@ async function searchWikiCorpus(params: { digest, query: params.query, maxResults: params.maxResults, + mode: params.mode, }) : []; const seenPaths = new Set(); @@ -678,7 +1057,7 @@ async function searchWikiCorpus(params: { } const results = candidatePages - .map((page) => toWikiSearchResult(page, params.query)) + .map((page) => toWikiSearchResult(page, params.query, params.mode)) .filter((page) => page.score > 0); if (candidatePaths.length === 0 || results.length >= params.maxResults) { return results; @@ -691,7 +1070,7 @@ async function searchWikiCorpus(params: { return [ ...results, ...remainingPages - .map((page) => toWikiSearchResult(page, params.query)) + .map((page) => toWikiSearchResult(page, params.query, params.mode)) .filter((page) => page.score > 0), ]; } @@ -728,16 +1107,19 @@ export async function searchMemoryWiki(params: { maxResults?: number; searchBackend?: WikiSearchBackend; searchCorpus?: WikiSearchCorpus; + mode?: WikiSearchMode; }): Promise { const effectiveConfig = applySearchOverrides(params.config, params); await initializeMemoryWikiVault(effectiveConfig); const maxResults = Math.max(1, params.maxResults ?? 10); + const mode = params.mode ?? "auto"; const wikiResults = shouldSearchWiki(effectiveConfig) ? await searchWikiCorpus({ rootDir: effectiveConfig.vault.path, query: params.query, maxResults, + mode, }) : []; @@ -750,7 +1132,7 @@ export async function searchMemoryWiki(params: { : null; const memoryResults = sharedMemoryManager ? (await sharedMemoryManager.search(params.query, { maxResults })).map((result) => - toMemoryWikiSearchResult(result), + toMemoryWikiSearchResult(result, mode), ) : []; diff --git a/extensions/memory-wiki/src/tool.ts b/extensions/memory-wiki/src/tool.ts index 77621397a31..ef691b867e3 100644 --- a/extensions/memory-wiki/src/tool.ts +++ b/extensions/memory-wiki/src/tool.ts @@ -7,7 +7,7 @@ import { type ResolvedMemoryWikiConfig, } from "./config.js"; import { lintMemoryWikiVault } from "./lint.js"; -import { getMemoryWikiPage, searchMemoryWiki } from "./query.js"; +import { getMemoryWikiPage, searchMemoryWiki, WIKI_SEARCH_MODES } from "./query.js"; import { syncMemoryWikiImportedSources } from "./source-sync.js"; import { renderMemoryWikiStatus, resolveMemoryWikiStatus } from "./status.js"; @@ -17,12 +17,14 @@ const WikiSearchBackendSchema = Type.Union( WIKI_SEARCH_BACKENDS.map((value) => Type.Literal(value)), ); const WikiSearchCorpusSchema = Type.Union(WIKI_SEARCH_CORPORA.map((value) => Type.Literal(value))); +const WikiSearchModeSchema = Type.Union(WIKI_SEARCH_MODES.map((value) => Type.Literal(value))); const WikiSearchSchema = Type.Object( { query: Type.String({ minLength: 1 }), maxResults: Type.Optional(Type.Number({ minimum: 1 })), backend: Type.Optional(WikiSearchBackendSchema), corpus: Type.Optional(WikiSearchCorpusSchema), + mode: Type.Optional(WikiSearchModeSchema), }, { additionalProperties: false }, ); @@ -126,6 +128,7 @@ export function createWikiSearchTool( maxResults?: number; backend?: ResolvedMemoryWikiConfig["search"]["backend"]; corpus?: ResolvedMemoryWikiConfig["search"]["corpus"]; + mode?: (typeof WIKI_SEARCH_MODES)[number]; }; await syncImportedSourcesIfNeeded(config, appConfig); const results = await searchMemoryWiki({ @@ -137,6 +140,7 @@ export function createWikiSearchTool( maxResults: params.maxResults, ...(params.backend ? { searchBackend: params.backend } : {}), ...(params.corpus ? { searchCorpus: params.corpus } : {}), + ...(params.mode ? { mode: params.mode } : {}), }); const text = results.length === 0 @@ -144,7 +148,7 @@ export function createWikiSearchTool( : results .map( (result, index) => - `${index + 1}. ${result.title} (${result.corpus}/${result.kind})\nPath: ${result.path}${typeof result.startLine === "number" && typeof result.endLine === "number" ? `\nLines: ${result.startLine}-${result.endLine}` : ""}${result.provenanceLabel ? `\nProvenance: ${result.provenanceLabel}` : ""}\nSnippet: ${result.snippet}`, + `${index + 1}. ${result.title} (${result.corpus}/${result.kind})\nPath: ${result.path}${typeof result.startLine === "number" && typeof result.endLine === "number" ? `\nLines: ${result.startLine}-${result.endLine}` : ""}${result.provenanceLabel ? `\nProvenance: ${result.provenanceLabel}` : ""}${result.matchedClaimId ? `\nClaim: ${result.matchedClaimId}` : ""}${result.evidenceKinds && result.evidenceKinds.length > 0 ? `\nEvidence: ${result.evidenceKinds.join(", ")}` : ""}\nSnippet: ${result.snippet}`, ) .join("\n\n"); return {