PULSE8.ai Cortex

PULSE8.ai Cortex is the open-source knowledge layer for AI agents: Git-native memory, a typed knowledge graph, and MCP-powered retrieval on top of plain Markdown — so agents can build, evolve, and reuse persistent knowledge instead of re-deriving it on every query.

Under the hood it's a unified vault for AI agents and humans, backed by a typed knowledge graph, full-text + hybrid search, and a MarkItDown-powered file compiler. Drop files in (PDF, DOCX, PPTX, XLSX, HTML, images, and more), let agents read, write, search, link, and compile knowledge — no database required.

Inspired by Andrej Karpathy's LLM Wiki pattern — a persistent, compounding knowledge base maintained by LLMs instead of re-derived on every query. Search powered by Tobi Lütke's QMD.

---

Why PULSE8.ai Cortex?

Most AI agents can access tools, but they cannot accumulate knowledge.

Traditional RAG systems retrieve documents. PULSE8.ai Cortex builds a persistent, evolving knowledge layer that grows over time and becomes more valuable the more agents and humans interact with it.

With PULSE8.ai Cortex, agents can:

• Learn from interactions — every read, write, ingest, and compile event is logged and graph-linked
• Build and traverse knowledge graphs — wikilinks, tags, and typed edges, maintained automatically
• Store structured insights — Markdown notes with typed nodes (note, agent_def, session, daily, feedback)
• Retrieve context across projects — vault_context builds a ranked subgraph from any seed query
• Share knowledge through MCP — one vault, every MCP-compatible client
• Maintain long-term memory — files survive sessions, deployments, and model upgrades
• Version knowledge through Git — the vault is a plain directory of Markdown, diff-friendly out of the box

Aspect	Traditional RAG	PULSE8.ai Cortex
Focus	Documents	Knowledge
Memory	Session-based	Persistent
Structure	Chunks	Markdown + typed graph
Evolution	Static index	Continuous, file-watched
Versioning	None	Git-native
Agent collaboration	Limited	First-class (MCP)

When to use PULSE8.ai Cortex

Ideal use cases

• ✅ Persistent memory for AI agents
• ✅ Shared knowledge across multiple agents
• ✅ Git-versioned organisational knowledge
• ✅ MCP-compatible knowledge retrieval
• ✅ Knowledge graphs without a dedicated graph database
• ✅ Long-term accumulation of institutional knowledge
• ✅ Human + AI collaborative knowledge management

Not ideal for

• ❌ Simple full-text document search (use a search engine)
• ❌ Pure vector-only retrieval with no graph (use a vector DB)
• ❌ Short-lived, stateless conversations
• ❌ Workflows that don't need persistent knowledge evolution

PULSE8.ai Cortex vs alternatives

Capability	PULSE8.ai Cortex	Traditional RAG	GraphRAG
Persistent knowledge	✅	❌	⚠️
Markdown-native storage	✅	❌	❌
MCP-compatible out of the box	✅	❌	❌
Knowledge graph	✅	❌	✅
Git versioning	✅	❌	❌
Agent memory layer	✅	❌	⚠️
Human + AI collaboration	✅	❌	⚠️
Continuous knowledge evolution	✅	❌	⚠️
Zero database required	✅	❌	❌

Works with

PULSE8.ai Cortex speaks MCP — so it plugs into any AI client that does. The same vault is reachable over streamable HTTP or stdio, and mirrored 1:1 by a REST API at /api/v1/.

Category	Compatible with
AI agents	Claude Desktop, Claude Code, OpenAI Agents, Gemini, custom agent frameworks
Development tools	Cursor, VS Code, JetBrains IDEs
Agent frameworks	LangGraph, LangChain, CrewAI, AutoGen
MCP ecosystem	MCP clients, MCP servers, MCP tool registries
Human tools	Obsidian, any Markdown editor, any Git client

Because the vault is just files, humans and agents collaborate on the same knowledge — no proprietary format, no lock-in.

Get started

[!NOTE] PULSE8.ai Cortex requires Docker. An OpenRouter API key is optional — needed only for LLM-powered cross-referencing between wiki articles. File conversion works out of the box without any API key.

1. Clone the repository:

  git clone https://github.com/synpulse8-opensource/pulse8-ai-cortex-knowledge-vault.git
  cd cortex-knowledge-vault

2. Launch PULSE8.ai Cortex:

  ./scripts/start.sh

This builds and starts both **PULSE8.ai Cortex** (API + MCP on `:8420`) and **QMD** (search on `:3100`), waits for health checks, and you're ready to go.

3. Connect your MCP client (e.g. Claude Desktop) to http://localhost:8420/mcp/.

To stop: ./scripts/stop.sh

Native QMD mode (macOS / Metal GPU)

Docker Desktop on macOS cannot expose the Metal GPU to containers, so containerized QMD embeds on CPU only — over an order of magnitude slower on non-trivial vaults. Run QMD natively instead; the qmd binary uses Metal automatically:

# One-time: install the qmd binary
brew install tobi/tap/qmd   # or: npm install -g @tobilu/qmd

# Start native QMD (background daemon) + Cortex in Docker
./scripts/start.sh --native-qmd

The QMD daemon's pid and log are kept in .qmd-native.pid / .qmd-native.log. To stop both: ./scripts/stop.sh --native-qmd (a plain ./scripts/stop.sh also cleans up a native QMD if one is running).

Cortex-only mode (external QMD)

If you manage QMD yourself (already running elsewhere), start only the Cortex container:

./scripts/start.sh --cortex-only   # set QMD_URL in .env if not http://host.docker.internal:3100

To stop: ./scripts/stop.sh --cortex-only

GPU-accelerated QMD (EC2 / Linux with NVIDIA GPU)

For production deployments with NVIDIA GPU acceleration:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build -d

See docs/ec2-gpu-setup.md for a full guide on instance selection, NVIDIA toolkit installation, and cost estimates.

Features

Knowledge Graph	Typed graph engine (NetworkX) — wikilinks, tags, and custom edges, auto-maintained on every file change
Full-Text Search	QMD search with hybrid (BM25 + vector + re-ranking) by default; keyword and semantic modes selectable. Results cached with a configurable TTL.
File Compiler	Converts raw sources (PDF, DOCX, PPTX, XLSX, HTML, images, etc.) to Markdown via MarkItDown. LLM used only for cross-referencing.
MCP Server	Streamable HTTP + stdio transport — works with Claude Desktop, Cursor, and any MCP client
Feedback & Notifications	vault_feedback captures quality feedback as notes; optional Microsoft Teams webhook posts an adaptive card per submission
Daily Activity Log	Every write/ingest/compile is mirrored into daily/<date>.md as a greppable, wikilinked timeline
Bulk Ingest	Ingest dozens or hundreds of files at once from a local directory with SHA-256 dedup and bounded concurrency
REST API	FastAPI endpoints mirroring all MCP tools at /api/v1/, including multipart file upload and bulk ingest
Vault Watcher	Real-time filesystem monitoring — graph stays in sync automatically
Lineage & Audit	Every edge labeled extracted / inferred / manual; vault_trace answers "why does the vault say X" back to the source document
Graph Queries	vault_path (what connects X to Y), vault_impact (what's downstream of this note), vault_explain (entity summary with provenance)
Curation Report	Read counters + outcome feedback (useful / dead-end / corrected) surface stale, contradicted, and never-read notes at GET /api/v1/curation/report
Zero Database	Everything persists as Markdown + JSON on your filesystem

Benchmarks

45.0% overall accuracy on LongMemEval-S (500 questions, full haystacks, hybrid search) with 65.6% evidence recall@8 and zero judge errors — measured end-to-end through the public REST API: ingest → compile → graph → search → answer.

Category	Accuracy	Recall@8
single-session-assistant	96.4%	98.2%
single-session-user	71.4%	78.6%
knowledge-update	60.3%	75.6%
temporal-reasoning	25.6%	51.1%
multi-session	24.8%	54.1%
single-session-preference	23.3%	63.3%

Every number is reproducible from a pinned config (dataset SHA-256, models, seed) with one command:

uv run python -m evals.run_longmemeval --config evals/configs/longmemeval-s-hybrid.yaml

The harness (evals/) publishes per-question JSONL traces, separates the judge model from the answer model, uses the official LongMemEval per-type grading prompts, and includes blind human validation of the judge. Full methodology, caveats, and raw results: docs/benchmarks/.

Runs without an LLM

Cortex is deterministic-first: ingestion (MarkItDown conversion), the knowledge graph (wikilinks, tags, derived_from edges), and QMD search all work with zero LLM calls. The LLM is an optional enrichment pass — cross-referencing, tagging, image captioning — not a dependency.

Pick a backend with LLM_BACKEND (env) / CORTEX_LLM_BACKEND (Python):

Backend	What it covers
openai-compatible (default)	OpenRouter, Azure OpenAI, Ollama, vLLM, LM Studio — anything speaking the OpenAI protocol. Point LLM_BASE_URL at your endpoint.
bedrock	AWS Bedrock via the standard AWS credential chain (no API key). Requires boto3.
none	Explicit zero-LLM mode. Guaranteed to construct no LLM client and make no model calls — suitable for air-gapped deployments.

Air-gapped example with a local Ollama:

LLM_BACKEND=openai-compatible \
LLM_BASE_URL=http://localhost:11434/v1 \
LLM_API_KEY=ollama \
COMPILER_MODEL=llama3.1 \
./scripts/start.sh

Or fully deterministic: LLM_BACKEND=none ./scripts/start.sh (no API key needed).

MCP resources (token-light large payloads)

PULSE8.ai Cortex implements the resources-as-tool-inputs pattern recommended by the Microsoft Copilot Studio CAT team: token-heavy tool outputs (large search result sets, full context windows) can be kept server-side and passed between tools as lightweight handles, so the LLM context window stays small.

How it works. Pass as_resource: true to vault_search or vault_context (or ?as_resource=true on GET /api/v1/search). Instead of inlining the full payload, you get a handle:

{
  "resource_id": "7f8a3c...",
  "resource_uri": "cortex://resource/7f8a3c...",
  "summary": { "query": "...", "count": 8, "paths": ["..."] }
}

Read it back through any of the three transports:

• MCP resources protocol — resources/read with the cortex://resource/{id} URI (Claude Desktop, Cursor, Copilot Studio MCP).
• Fallback tool — vault_resource_read for clients that only expose tools to the planning layer (some Copilot Studio configurations).
• REST — GET /api/v1/resources/{resource_id} (accepts the bare ID or the full URI).

The store is in-memory, asyncio-safe, TTL-evicted, and LRU-bounded:

Env var	Default	Purpose
CORTEX_RESOURCE_TTL_SECONDS	3600	Max age before a stored resource is evicted lazily on read
CORTEX_RESOURCE_MAX_ITEMS	1000	LRU cap before oldest entry is dropped

The same ResourceStore is shared between MCP and REST — produce a handle via MCP, read it back via REST (or vice versa).

Microsoft Copilot Studio setup — agent instructions, tool selection, and the Custom Connector fallback — is documented in docs/copilot-studio.md. No Cortex code change required.

MCP tools

Tool	Description
vault_read	Read a note by path
vault_write	Create or update a note
vault_search	Search the vault (keyword / semantic / hybrid). Supports as_resource=true
vault_link	Create, query, or delete graph edges
vault_context	Build a context window: search → graph traversal → ranked subgraph. Supports as_resource=true
vault_ingest	Ingest raw content or binary files (supports content_base64 for binary)
vault_compile	Compile unprocessed raw sources into wiki Markdown via MarkItDown
vault_feedback	Submit feedback on vault quality (status: OPEN; optional related_paths and outcome: useful / dead-end / corrected)
vault_list_feedbacks	List feedback note metadata (paths, tags, status; not full body)
vault_resource_read	Read a server-stored MCP resource by ID (fallback for clients without resources/read)
vault_trace	Trace a note's lineage: provenance, raw sources, and edges labeled extracted / inferred / manual
vault_path	Shortest paths between two notes — "what connects X to Y", every hop typed and origin-labeled
vault_impact	Walk everything downstream of a note (change-impact analysis)
vault_explain	Explain a note: summary, provenance, sources, links in/out, contradictions

Architecture

┌──────────────────────────────────────────────┐
│  MCP Client (Claude Desktop, Cursor, etc.)   │
└──────────┬───────────────────────────────────┘
           │  MCP (HTTP or stdio)
┌──────────▼───────────────────────────────────┐
│  PULSE8.ai Cortex  :8420                     │
│  ┌──────────────────────────────────────┐     │
│  │ Auth (API Key or Microsoft Entra ID) │     │
│  └──────────────┬───────────────────────┘     │
│  ┌─────────┐ ┌──┴───────┐ ┌──────────────┐   │
│  │ MCP     │ │ REST API │ │ Vault Watcher│   │
│  │ /mcp/   │ │ /api/v1/ │ │ (watchfiles) │   │
│  └────┬────┘ └────┬─────┘ └──────┬───────┘   │
│       └───────────┼──────────────┘           │
│            ┌──────▼──────┐                   │
│            │ Graph Engine│                   │
│            │ + Compiler  │                   │
│            └─────────────┘                   │
└──────────┬───────────────────────────────────┘
           │
┌──────────▼───────────────────────────────────┐
│  QMD  :3100                                  │
│  BM25 + vector search, auto-indexes on start │
└──────────┬───────────────────────────────────┘
           │
┌──────────▼───────────────────────────────────┐
│  Vault (bind-mounted volume)                 │
│  wiki/ raw/ agents/ sessions/ daily/ feedback/ │
│  .cortex/ (graph.json, index.md, log.md)     │
└──────────────────────────────────────────────┘

Vault layout

The vault is a plain directory of Markdown files organised by purpose. Cortex classifies each file into a typed node (NodeType) used by the graph engine and exposed in REST and MCP responses.

Folder	NodeType	Purpose
wiki/	note	Compiled, interlinked knowledge articles
raw/	raw_source	Unprocessed sources (PDF, DOCX, TXT, …) the compiler reads from
agents/	agent_def	Agent definitions
sessions/	session	Per-session notes / conversation transcripts
daily/	daily	Daily notes (Obsidian Daily Notes convention)
feedback/	feedback	Feedback on vault quality (status, related_paths)
.cortex/	(skipped)	Cortex internals — graph.json, index.md, log.md, manifests

How classification works

Order of precedence (first match wins):

1. *Frontmatter type:* — explicit override always wins (e.g. type: note in agents/foo.md resolves to NodeType.NOTE)
2. Folder prefix — files under raw/ agents/ sessions/ daily/ feedback/ inherit the folder's type with no filename suffix needed (e.g. daily/2026-06-10.md → daily)
3. Filename suffix (backward-compatible) — .agent.md, .session.md, .memory.md are still honored anywhere (e.g. wiki/legacy.agent.md → agent_def)
4. Default — NodeType.NOTE

In practice this means you can drop YYYY-MM-DD.md straight into daily/, or an unsuffixed planner.md into agents/, and the graph and API will classify them correctly without any renaming.

Daily activity log

Every vault_write, vault_ingest, and successful compile event (MCP and REST paths) is automatically mirrored into today's UTC daily note at daily/YYYY-MM-DD.md. The file is created on first event of the day and each subsequent event appends a ## [HH:MM] event | summary block plus a [[wiki-stem]] wikilink (so the watcher draws a LINKS_TO edge to the affected note). The format follows the Karpathy log.md greppable-prefix pattern — grep "^## \[" daily/2026-06-10.md gives a clean timeline of the day.

Writes targeting daily/, feedback/, or .cortex/ are deliberately not mirrored (would be self-referential noise). The hidden .cortex/log.md audit log is unaffected and continues to receive every operation.

Bulk ingest

For ingesting many files at once (dozens or hundreds of PDFs, papers, docs), use the one-click shell script instead of feeding them one at a time through MCP. It reads directly from a local directory (recursively, including subfolders) — no wire overhead, no running server required — deduplicates via SHA-256 hashing, compiles with bounded concurrency, and rebuilds the index once at the end. Subpaths are preserved under the vault raw folder (e.g. source/abcde/doc.html → raw/abcde/doc.html).

One-click script (recommended)

# Ingest all files from a directory
./scripts/bulk_ingest.sh ./my-papers/

# Dry-run to preview what would be ingested
./scripts/bulk_ingest.sh ./my-papers/ --dry-run

# Force re-ingest (bypass dedup manifest)
./scripts/bulk_ingest.sh ./my-papers/ --force

# Control LLM concurrency (default: 4)
./scripts/bulk_ingest.sh ./my-papers/ --concurrency 8

The script automatically loads your .env for the LLM key and vault path, prints a summary, then runs the full pipeline (copy, compile, reindex). No running Cortex server needed.

Python CLI (direct)

CORTEX_VAULT_PATH=./example_vault uv run cortex-bulk-ingest --source ./my-papers/

Inside Docker

# Set INGEST_DIR in .env or export it, then restart
export INGEST_DIR=/path/to/your/papers
docker compose up -d

# Run bulk ingest inside the container
docker exec pulse8-ai-cortex uv run cortex-bulk-ingest --source /ingest

Via REST API

For programmatic use without MCP (requires running Cortex server):

curl -X POST http://localhost:8420/api/v1/bulk-ingest \
  -H "Content-Type: application/json" \
  -H "x-api-key: your-secret-api-key" \
  -d '{"source_dir": "/ingest", "concurrency": 4}'

Deduplication

The dedup manifest is stored at .cortex/ingest-manifest.json. Files are matched by content hash, not filename — renaming a file won't cause re-ingestion, and the same content under a different name will be skipped.

Configuration

Copy the example and fill in your values:

cp .env.example .env