kb CLI and Knowledge Base Pattern

Build and maintain a self-compiling Obsidian markdown knowledge base using the kb CLI. The LLM reads raw sources, writes cross-linked wiki articles, files Q&A results back into the corpus, and runs lint-and-heal passes. The CLI also supports codebase ingestion with deep inspection commands for code quality, architecture health, and symbol relationships.

Each topic lives in its own folder inside the Obsidian vault, either directly at the vault root (e.g. go-best-practices/) or nested by configured glob (e.g. harness/goclaw/). All topics share a single Obsidian vault, commonly the repo root. Read references/architecture.md for the full rationale and the four-phase pipeline (ingest → compile → query → lint).

Every topic has a lifecycle mode recorded in topic.yaml — wiki (the default) or okf:

• A wiki topic is the Karpathy research lab described throughout this skill. It contains raw/, wiki/, outputs/, and bases/ subtrees plus topic-level CLAUDE.md, topic.yaml, and log.md, and follows the ingest → compile → query → lint loop.
• An OKF topic is a portable Open Knowledge Format catalog (a "bundle") that other people's agents and tools consume. It is flat: typed concept files at the bundle root, plus a generated index.md, a log.md, an OKF-flavored CLAUDE.md, and the AGENTS.md symlink — no raw//wiki//outputs/ pyramid. See the OKF Dual-Mode section below and references/okf-mode.md.

Mode is opt-in per topic and invisible to existing users: absent/empty mode normalizes to wiki, so every existing topic behaves exactly as before.

The topic's CLAUDE.md is the schema document and topic marker — it tells the LLM the scope, conventions, current articles, and research gaps for that topic. topic.yaml is the structured source of truth for topic metadata (slug, title, domain). AGENTS.md may symlink to CLAUDE.md for Codex parity, but the valid-topic marker is CLAUDE.md.

Prerequisites

1. Verify the kb binary is available:

   kb version
   

2. For search and index commands, verify QMD is installed:

   qmd --version
   # If missing: npm install -g @tobilu/qmd
   

3. Supported source languages for codebase analysis: TypeScript (.ts), TSX (.tsx), JavaScript (.js), JSX (.jsx), Go (.go).
4. For repo-root vaults, configure kb.toml at the vault root:

   [vault]
   root = "."
   topic_globs = ["*", "harness/*", "social-media/*"]
   

Pattern Overview

Based on Andrej Karpathy's LLM Wiki pattern, the KB treats the LLM as a compiler that reads raw source documents and produces a structured, cross-linked markdown wiki. The four-phase loop:

1. Ingest — Scrape/curate sources via kb CLI → raw/ (immutable staging)
2. Compile — LLM reads raw/, writes wiki/concepts/ articles (3000-4000 words, dense wikilinks)
3. Query — Q&A against wiki → file answers to outputs/queries/, promote strong answers to wiki
4. Lint — Automated structural checks + LLM-driven semantic healing

Read references/architecture.md for the full rationale, context-window vs RAG tradeoffs, and multi-topic vault design.

OKF Dual-Mode

kb supports two distinct knowledge lifecycles, selected per topic by mode:

• wiki (default) — a research lab: ingest → compile → query → lint. Obsidian [[wikilinks]], the wiki frontmatter schema, the raw/+wiki/+outputs/ pyramid. This is the lifecycle the rest of this skill documents.
• okf — a portable catalog: declare → consume. A flat bundle of typed concepts that conforms to the Open Knowledge Format (OKF) v0.1, meant to be shared and consumed by other people's agents and tools. Plain relative markdown links, the OKF producer-field contract, a generated index.md.

The two are not derived from each other; the high-leverage move is distilling research into the catalog — turning a finished wiki concept into a typed OKF concept. kb makes that one command (kb promote), mechanical and non-LLM, then validates the result (kb okf check).

OKF bundle layout (what kb topic new --mode okf scaffolds):

<bundle>/
  CLAUDE.md        # OKF-flavored schema/marker (topic marker)
  AGENTS.md        # symlink → CLAUDE.md
  topic.yaml       # mode: okf
  index.md         # generated; frontmatter okf_version: "0.1"; concepts grouped by type
  log.md           # "# Directory Update Log"; ISO-date (## YYYY-MM-DD) headings, newest first
  <concept>.md     # typed concept files, flat at the bundle root (added by promote/authoring)

There is no raw//wiki//outputs//bases/ in OKF mode. index.md and log.md are auto-maintained by kb (regenerated on scaffold and on every promote).

OKF concept contract — every concept .md carries four producer fields in frontmatter (emitted alphabetically): description, timestamp (RFC3339, UTC), title, type, plus optional tags. type is the only field OKF v0.1 strictly requires; the other three and the relative-link style are two deliberate, documented deviations matching Google's reference tooling. Concept bodies use relative markdown links ([label](other-concept.md)), never [[wikilinks]].

Two deviations from the written OKF spec are intentional (ADR-002): emit the four producer fields (spec mandates only type) and emit relative links (spec recommends absolute /path.md, which breaks GitHub rendering).

Read references/okf-mode.md for the full bundle layout, frontmatter contract, promote transform rules, conformance ruleset, and the concept-path/link model.

Related Skills

This skill orchestrates several companion skills for the LLM-driven phases:

• obsidian-markdown — author wiki articles with valid Obsidian Flavored Markdown (wikilinks, callouts, embeds, properties).
• obsidian-bases — create .base files under <topic>/bases/ for dashboard views, filters, and formulas.
• obsidian-cli — interact with the running Obsidian vault from the command line (open notes, search, refresh indexes).

kb CLI Quick Reference

Topic management

kb topic new <slug> <title> <domain>              # scaffold a new wiki topic (mode: wiki, default)
kb topic new <slug> <title> <domain> --mode okf   # scaffold a flat OKF bundle (mode: okf)
kb topic list                                      # list all topics in the vault
kb topic info <topic-id>                           # topic metadata (counts, last log entry)

--mode accepts wiki (default) or okf. An OKF topic scaffolds the flat bundle layout (see the OKF Dual-Mode section), not the wiki pyramid.

OKF bundles (promote + conformance)

kb promote <wiki-doc> --to <okf-topic> --type <Type>                  # distill a wiki concept into an OKF bundle
kb promote <wiki-doc> --to <okf-topic> --type <Type> --description "…" # override the generated description
kb okf check <okf-topic>                                               # validate OKF v0.1 conformance (lenient)
kb okf check <okf-topic> --strict --format json                       # promote local-standard warnings to errors (CI gate)

kb promote is mechanical, non-LLM, and non-destructive: it reads a compiled wiki document, remaps its frontmatter to the OKF producer contract, rewrites [[wikilinks]] to relative markdown links, writes a new typed concept at the bundle root, regenerates the bundle's index.md, and inserts a newest-first log.md entry. The source wiki document is left untouched. Both --to and --type are required, and --to must resolve to a mode: okf topic or promote errors before writing. The concept filename is a slug of the source document's base name (collisions get a -2, -3 suffix), so its filename and every inbound link share one canonical key. kb promote emits the JSON ConceptResult (writtenPath, type, linksRewritten, unresolvedLinks, warnings).

kb okf check is lenient by default per OKF §9 (tolerates broken cross-links, unknown type values, missing optional fields) so externally produced bundles pass. It emits diagnostics as severity, kind, filePath, target, message and exits non-zero when any error is present (and on warnings too under --strict). Hard errors include a concept with a missing/empty type and unparseable frontmatter; local-standard warnings include a missing producer field (title/description/timestamp) and a type outside the configured vocabulary. The type vocabulary is the local standard in kb.toml:

[okf]
# Local OKF concept type vocabulary. Empty (the default) means `kb okf check`
# never warns about unknown types until you opt into a local standard.
types = ["Voice Profile", "Offer", "Playbook"]

When [okf].types is non-empty, a --type (or existing concept type) outside the list is a warning — --strict turns it into a CI-failing error, preventing type drift (Voice Profile vs voice-profile).

Ingestion (auto-generates frontmatter, auto-appends to log.md)

kb ingest url <url> --topic <topic-id>        # scrape a web URL via Firecrawl
kb ingest file <path> --topic <topic-id>      # convert local file (PDF, DOCX, EPUB, HTML, images w/OCR, etc.)
kb ingest youtube <url> --topic <topic-id>    # extract a single YouTube transcript -> raw/youtube/
kb ingest channel <url> --topic <topic-id>    # bulk-extract a YouTube channel/playlist -> raw/youtube/
kb ingest instagram <url> --topic <topic-id>  # Instagram reel/video (caption + transcript) -> raw/instagram/
kb ingest bookmarks <path> --topic <topic-id> # ingest a bookmark-cluster markdown file
kb ingest codebase <path> --topic <topic-id>  # analyze a codebase into raw/codebase/

Use path-relative topic identifiers for nested topics, e.g. --topic harness/goclaw.

YouTube extraction uses raw/youtube/ as the canonical transcript directory. kb requires yt-dlp for metadata, captions, and audio extraction. Install or update yt-dlp when public captions fail before treating the issue as only a proxy/cookie problem:

[youtube]
yt_dlp_path = "yt-dlp"
proxy = "http://127.0.0.1:8080"
cookies_file = "/path/to/youtube-cookies.txt"
user_agent = "Mozilla/5.0 ..."
retry_attempts = 3
retry_backoff = "1s"
transcription = "captions" # captions | auto | stt

Use kb ingest youtube <url> --topic <topic-id> --transcribe auto to use manual captions when present and STT when only automatic captions or no captions are available. Use --transcribe stt to force STT. The old shortcut flag is intentionally unsupported.

YouTube transcript frontmatter includes video metadata from yt-dlp for filtering and ordering, including view_count, like_count, comment_count, upload_date, duration, duration_string, channel, channel_id, uploader_id, channel_follower_count, categories, youtube_tags, language, live_status, was_live, and chapter_count. Use youtube_tags for video keywords because tags is KB taxonomy; see references/frontmatter-schemas.md for null, list, and count behavior.

YOUTUBE_YT_DLP_PATH, YOUTUBE_PROXY, YOUTUBE_COOKIES_FILE, and YOUTUBE_USER_AGENT override the matching TOML values for local runs. The CLI reports blocked caption/audio requests as network_blocked; after confirming yt-dlp is installed and current, treat that as a network or auth configuration issue, not a missing-transcript issue.

YouTube channels and playlists

kb ingest channel <url> --topic <topic-id> bulk-extracts a YouTube channel or playlist into raw/youtube/, writing one document per upload with the same youtube-transcript frontmatter as single videos. Flags: --limit <n> (newest n; 0 = all), --all (every upload, overrides --limit), --concurrency <n>, --throttle <dur>, --dry-run (resolve and list videos without ingesting), and --transcribe captions|auto|stt. The worker-pool size, inter-request throttle, adaptive-backoff ceiling, and per-video retries default from [youtube].bulk_concurrency, [youtube].bulk_throttle, [youtube].bulk_backoff_max, and [youtube].bulk_retries. Single-video URLs are rejected — use kb ingest youtube for those.

Instagram ingestion

kb ingest instagram <url> --topic <topic-id> ingests Instagram reels, video posts (/p/), and IGTV (/tv/) into raw/instagram/ using the same yt-dlp + STT engine as YouTube. The document body is the post caption under a ## Caption heading followed by the spoken transcript under ## Transcript. When the audio yields no transcript (e.g. a music-only reel) it degrades to a caption-only document with transcript_source: none and no ## Transcript section.

[instagram]
yt_dlp_path = "yt-dlp"
proxy = ""
cookies_file = "/path/to/instagram-cookies.txt" # separate from YouTube — Instagram needs its own session
user_agent = ""
transcription = "auto" # captions | auto | stt (default auto: reels rarely carry caption tracks)
retry_attempts = 3
retry_backoff = "1s"

[instagram] mirrors the [youtube] yt-dlp knobs but keeps cookies separate because the two platforms need distinct authenticated sessions. Public reels often work without cookies, but Instagram rate-limits aggressively — set cookies_file (and optionally proxy) for reliable access. The default --transcribe is auto (manual caption when present, otherwise Whisper STT). Whole-profile bulk ingestion is out of scope because yt-dlp's instagram:user extractor is currently broken. Blocked requests are reported as network_blocked; remediate with [instagram].cookies_file / [instagram].proxy or a trusted network.

Layout migrations

kb migrate transcripts --topic <topic-id> # move legacy raw/transcripts/*.md into raw/youtube/

Codebase inspection

kb inspect smells [--type <smell-type>] --format json
kb inspect dead-code --format json
kb inspect complexity [--top N] --format json
kb inspect blast-radius [--min N] [--top N] --format json
kb inspect coupling [--unstable] --format json
kb inspect circular-deps --format json
kb inspect symbol <name> --format json
kb inspect file <path> --format json
kb inspect backlinks <name-or-path> --format json
kb inspect deps <name-or-path> --format json

Structural linting

kb lint [<topic-id>] [--save]             # dead links, orphans, missing sources, format violations, stale content

Indexing and search (requires QMD)

kb index --topic <topic-id>               # create or update QMD collection
kb search "<query>" --topic <topic-id>    # hybrid BM25 + vector search
kb search "<query>" --lex --topic <topic-id>  # keyword-only search
kb search "<query>" --vec --topic <topic-id>  # vector-only search

After running kb ingest or kb lint --save, the CLI auto-appends entries to <topic>/log.md. kb promote (wiki→OKF) also auto-maintains the OKF bundle's log.md and index.md. Manual log entries are still needed for compile, query, the wiki-internal query→wiki promotion, and split operations (Procedure 5).

Command Dispatch

Map the user's intent to the correct command:

Intent	Command
Scaffold a new wiki topic	kb topic new <slug> <title> <domain>
Scaffold a new OKF bundle	kb topic new <slug> <title> <domain> --mode okf
Distill a wiki concept into an OKF bundle	kb promote <wiki-doc> --to <okf-topic> --type <Type>
Check an OKF bundle for conformance	kb okf check <okf-topic>
Gate OKF conformance in CI	kb okf check <okf-topic> --strict --format json
List all topics	kb topic list
Scrape a web URL	kb ingest url <url> --topic <topic-id>
Ingest a local file (PDF, DOCX, etc.)	kb ingest file <path> --topic <topic-id>
Extract a YouTube transcript	kb ingest youtube <url> --topic <topic-id>
Bulk-ingest a YouTube channel/playlist	kb ingest channel <url> --topic <topic-id>
Extract an Instagram reel/video	kb ingest instagram <url> --topic <topic-id>
Migrate legacy transcripts	kb migrate transcripts --topic <topic-id>
Ingest bookmark clusters	kb ingest bookmarks <path> --topic <topic-id>
Analyze a codebase	kb ingest codebase <path> --topic <topic-id> --progress never
Find code smells	kb inspect smells --format json
Find dead exports and orphan files	kb inspect dead-code --format json
Rank functions by complexity	kb inspect complexity --format json
Find high-impact symbols (blast radius)	kb inspect blast-radius --min 5 --format json
Find unstable files (coupling)	kb inspect coupling --unstable --format json
Find circular imports	kb inspect circular-deps --format json
Look up a specific symbol	kb inspect symbol <name> --format json
Look up a specific file	kb inspect file <path> --format json
Find what depends on X (incoming refs)	kb inspect backlinks <name-or-path> --format json
Find what X depends on (outgoing deps)	kb inspect deps <name-or-path> --format json
Run structural lint	kb lint <topic-id> --save
Index vault for search	kb index --topic <topic-id>
Search the knowledge base	kb search "<query>" --topic <topic-id> --format json

Codebase Analysis Workflow

For codebase-specific analysis, the kb ingest codebase command must run before any inspect command.

Workflow A -- Code Analysis (no QMD required):

kb ingest codebase <path> --topic <topic-id> --> kb inspect <subcommand>

Workflow B -- Full Pipeline (requires QMD):

kb ingest codebase <path> --topic <topic-id> --> kb index --> kb search <query>

On first run, kb ingest codebase uses the discovered vault from kb.toml or legacy .kb/vault/. If no vault marker is discoverable from the current working directory, it falls back to bootstrapping under <path>/.kb/vault/<topic-id>/. Later commands auto-discover kb.toml or .kb/vault/ from the current directory; otherwise pass --vault <path>.

Ingest a Codebase

kb ingest codebase <path> --topic <topic-id> --progress never

Always use --progress never in agent contexts to prevent TTY progress bars from corrupting stdout. Use --title and --domain only when bootstrapping a missing topic.

Parse the JSON output from stdout to extract key values:

• topicSlug -- the topic identifier for later commands
• vaultPath -- absolute path to the vault root
• topicPath -- absolute path to the topic directory
• filesScanned, filesParsed, symbolsExtracted -- summary statistics
• diagnostics -- check for warnings or errors

Stderr carries structured stage logs. Do not treat stderr content as failure evidence.

Key flags:

• --vault <dir> -- override vault root location
• --output <dir> -- deprecated alias for --vault
• --topic <topic-id> -- target topic id inside the vault; nested topics use a relative path such as harness/goclaw
• --title <value> -- bootstrap-only topic title override
• --domain <value> -- bootstrap-only topic domain override
• --include <pattern> -- re-include paths that would otherwise be ignored (repeatable)
• --exclude <pattern> -- exclude additional paths from scanning (repeatable)
• --semantic -- enable semantic analysis when adapters support it

Read references/cli-ingest-codebase.md for the full flag table and output schema.

Inspect the Vault

Run inspect subcommands to analyze code quality and architecture.

Shared flags for all inspect subcommands:

• --format json -- always use JSON for programmatic parsing
• --vault <path> -- explicit vault root (omit to auto-discover from cwd)
• --topic <topic-id> -- explicit topic id, including nested relative paths such as harness/goclaw (omit if only one topic exists)

Tabular Subcommands

These return a list of rows sorted by the primary metric:

1. smells -- List symbols and files with detected code smells.

   kb inspect smells --format json
   kb inspect smells --type high-complexity --format json
   

2. dead-code -- List dead exports and orphan files.

   kb inspect dead-code --format json
   

3. complexity -- Rank functions/methods by cyclomatic complexity. Default top 20.

   kb inspect complexity --format json
   kb inspect complexity --top 50 --format json
   

4. blast-radius -- Rank symbols by transitive dependent count.

   kb inspect blast-radius --format json
   kb inspect blast-radius --min 10 --top 20 --format json
   

5. coupling -- Rank files by instability (Ce / (Ca + Ce)).

   kb inspect coupling --format json
   kb inspect coupling --unstable --format json
   

6. circular-deps -- List files participating in circular import chains.

   kb inspect circular-deps --format json
   

Detail Lookup Subcommands

These return field-value pairs for a single matched entity:

7. symbol <name> -- Case-insensitive substring match. Returns detail fields for a single match, or a summary table for multiple matches.

   kb inspect symbol parseConfig --format json
   

8. file <path> -- Exact source path lookup. Use the source-relative path as stored in vault frontmatter.

   kb inspect file src/config.ts --format json
   

Relation Subcommands

These return relation edges (target_path, type, confidence):

9. backlinks <name-or-path> -- Incoming references. Accepts a symbol name or file path.

   kb inspect backlinks parseConfig --format json
   

10. deps <name-or-path> -- Outgoing dependencies. Accepts a symbol name or file path.

    kb inspect deps src/config.ts --format json
    

Read references/cli-inspect.md for all column schemas and flag details.

Index the Vault

Index the vault content into QMD for search. This step requires QMD on PATH.

kb index --topic <topic-id>

The command is idempotent: it checks whether the collection already exists and chooses add (create) or update (refresh) automatically.

Key flags:

• --embed (default true) -- run embedding after syncing files
• --force-embed -- force re-embedding all documents
• --context <text> -- attach human context to improve search relevance
• --name <name> -- override the derived collection name

Read references/cli-search-index.md for the full output schema.

Search the Vault

Search indexed vault content with QMD. Requires a prior kb index run.

kb search "<query>" --topic <topic-id> --format json

Search modes:

• Hybrid (default) -- combines lexical and vector search
• Lexical (--lex) -- BM25 keyword search only
• Vector (--vec) -- embedding-based semantic search

The --lex and --vec flags are mutually exclusive. Omit both for hybrid mode.

Key flags:

• --limit N (default 10) -- maximum results
• --min-score N -- minimum relevance threshold
• --full -- return full document content instead of snippets
• --all -- return all matches above the minimum score

Read references/cli-search-index.md for full details.

KB Maintenance Procedures

Procedure 1: Compile a wiki article