Project preprocessing hooksLink to Project preprocessing hooks

vaultspec-rag keeps vault records, conventional source code, and extracted documents in separate index domains. A project can contain other useful formats that the built-in source parsers should not interpret as code.

Preprocessing hooks let each project supply its own extraction logic and explicitly route the result to either the code or document domain. You register a command per file pattern. vaultspec-rag runs it, validates the output against a versioned schema, and indexes the extracted text first-class. The indexed chunks are searchable with anchors that deep-link back into the original document. vaultspec-rag does not infer ownership from directory names or ship client-specific file rules; it owns the contract and runs your extractor.

Examples use the uv run prefix, which runs the command inside a project environment. If you installed vaultspec-rag as a standalone tool, drop the prefix and call vaultspec-rag directly; see the installation guide for lane selection.

How it worksLink to How it works

Add a version-2 .vaultragpreprocess.toml to your project root (a sibling of .vaultragignore) mapping file patterns to extraction commands and explicit targets.
Approve the rules with vaultspec-rag preprocess approve. Rules never run until the person operating this machine has approved them, and any later edit to the file needs approval again.
During indexing, a file matching a rule is handed to your command, which prints one JSON document on stdout.
vaultspec-rag validates that JSON, turns it into searchable chunks (carrying your anchors and locators), and indexes them.
Search results for those chunks surface the source path and an anchor (for example report.pdf#page=12) instead of a line number.

Only the owning route admits a matched file. The extractor can turn binary or otherwise unsupported input into text without making it source code. A preprocess rule never re-includes files excluded by .gitignore or .vaultragignore; ignore always wins.

Configure rulesLink to Configure rules

A rule is code that runs as you. Once you approve a project’s rules, vaultspec-rag executes the command each rule names, with your account’s privileges and in no sandbox. Nothing runs until you approve, and editing the file withdraws the approval. Read Security posture before approving rules you did not write.

Create .vaultragpreprocess.toml at the project root:

version = 2

[[rule]]
pattern  = "*.pdf"                 # gitignore-style glob (same dialect as .vaultragignore)
command  = "my-pdf-extract {path}" # {path} is replaced with the file path
target = "document"                # code | document; required ownership
extractor_version = "1.0.0"        # required; bump when extraction semantics change
on_error = "skip"                  # skip | fail | passthrough  (default: skip)
timeout_s = 60                     # wall-clock bound for the command
max_source_bytes = 52428800         # optional per-rule input ceiling

[[rule]]
pattern  = "docs/**/*.xlsx"
command  = "python tools/xlsx_extract.py {path}"
target = "document"
extractor_version = "2.1.0"
priority = 10                      # lower priority sorts first; first matching rule wins; ties break by file order

Rule fields:

pattern (required) - one gitignore-style glob. Add more [[rule]] tables for more patterns.
target (required) - the sole owner of matching output: code or document. Ownership is explicit and stable; paths and extensions do not choose the domain.
extractor_version (required, non-empty string) - your extractor’s semantic version. Change it whenever output semantics change so signatures and cache entries are invalidated deterministically.
One of command or entry_point (required, exactly one):
- command is a command template. {path} is substituted with the source file path. The command is split with shell-style tokenization and run without a shell, so spaces and metacharacters in paths can’t inject. On Windows, use forward slashes in the interpreter or script path (a backslash path breaks the POSIX-style tokenizer even inside a TOML literal).
- entry_point is a "module:callable" reference. The callable (def my_callable(source_path: str) -> Mapping | BaseModel) runs out-of-process (the same isolation and timeout_s bound as command); the module must be importable in the service’s Python environment.
priority (optional, default 100) - lower priority sorts first; the first matching rule wins; ties break by file order.
on_error (optional, default skip):
- skip - drop the file from the index and report it (never a silent gap).
- fail - abort the whole index run (use when missing a document is unacceptable).
- passthrough - index the raw file unprocessed instead of failing.
timeout_s (optional) - kill the command after this many seconds and treat it as a failure per on_error. Must be a positive number. Without it, the command gets 120 seconds. Values above 1800 seconds are capped at 1800.
path_independent (optional, default false) - permit byte-identical inputs at different paths to share an extraction-cache entry. Enable it only when output never embeds the source path in text, anchors, locators, or metadata.
max_source_bytes (optional) - a positive per-rule input ceiling. The effective ceiling is the lower of this value and the document source limit of the active support profile, which vaultspec-rag status prints as the Document support line: 512 GiB under the default managed-service profile and 64 GiB under embedded-local. Those are whole-corpus figures applied per file, so unless you are extracting something extraordinary this rule’s own value is the one that binds.
[rule.options] (optional) - an opaque table forwarded to your extractor for its own use.

Both command and entry_point rules run out-of-process (a subprocess), so they share the same CPU-only isolation and timeout_s bound. An entry_point callable must be importable in the service’s environment and return a mapping (or pydantic model) shaped like the output schema.

Inspect, validate, and approve your configurationLink to Configure rules, Inspect, validate, and approve your configuration

Six commands cover authoring, approval, and debugging. Prefix each with uv run in a uv-managed environment:

uv run vaultspec-rag preprocess list            # show resolved rules, in precedence order
uv run vaultspec-rag preprocess check           # validate the config; non-zero exit on a bad config
uv run vaultspec-rag preprocess approve         # let the current rules of this project run
uv run vaultspec-rag preprocess revoke          # withdraw that approval
uv run vaultspec-rag preprocess run-one a.pdf   # trial the matching rule against one file
uv run vaultspec-rag preprocess status          # mode, approval, config presence, and rule count

preprocess list prints each resolved rule with its target, extractor version, cache-binding mode, pattern, priority, failure handling, timeout, and command, sorted in precedence order.
preprocess check strictly validates .vaultragpreprocess.toml and reports the first defect. Invalid or legacy targetless configuration is rejected before indexing can mutate a collection, sidecar, ledger, or cache.
preprocess approve records your approval of the rules exactly as they are now, for this project directory, and prints each rule’s command and the digest it approved. Approving an already approved project changes nothing. An invalid configuration cannot be approved. Run vaultspec-rag index afterwards to apply it.
preprocess revoke withdraws the approval, so the rules stop running from the next index run.
preprocess run-one <path> runs the matching rule against one file and prints the validated output, with no indexing side effect. It obeys the same gate as indexing: on an unapproved project it runs nothing and names the approval command, so approve after each edit while authoring. Routing and migration defects, plus an extractor abort under on_error = "fail", are structured errors with a non-zero exit. Other malformed rules use the non-strict loader and may appear as no match. When validating configuration, run preprocess check first.
preprocess status reports the effective execution mode, whether the current rules are approved, schema version, targets, extractor versions, rule count, and whether the kill switch or a missing approval prevents hooks from running. Its --json envelope carries more than that human summary shows, including the policy_digest an approval binds; the CLI reference names every field.

All six accept --json for scripting.

Invocation envelopeLink to Invocation envelope

Every extractor receives a versioned JSON envelope in the VAULTSPEC_PREPROCESS_INVOCATION environment variable. Rules run as a subprocess, so an extractor in any language reads that variable and parses the JSON below; nothing about this contract is Python-specific. A Python extractor may instead call vaultspec_rag.indexer._preprocess_schema.load_preprocess_invocation(), which validates the envelope for you - but the leading underscore is not decoration. That module is internal, nothing re-exports it, and it can move between releases, so an extractor you intend to keep is better off parsing the documented shape.

{
  "schema_version": 1,
  "source_paths": ["docs/report.pdf"],
  "options": {"layout": "pages"},
  "extractor_version": "1.0.0",
  "target": "document",
  "mode": "single"
}

For a batch rule, source_paths contains the bounded manifest members and mode is batch. The envelope is the authoritative place for rule options, target, and extractor version; command-line placeholders remain only the transport for source paths.

Output schemaLink to Output schema

Your command receives a source file path and prints one JSON object on stdout. This is the contract between your extractor and the indexer; invalid output is a per-file error, never a crash. The models are pydantic v2 with unknown fields forbidden, so a typo is a loud validation error rather than silent data loss.

{
  "schema_version": 1,                 // required; the schema major (currently 1)
  "preprocessor_id": "pdf-extract",    // required; your extractor's id
  "preprocessor_version": "1.2.0",     // required; your extractor's version
  "source_path": "docs/report.pdf",    // required; the path you were given
  // EXACTLY ONE of `units` or `text`:
  "units": [                           // (a) pre-chunked units
    {
      "text": "Quarterly revenue ...", // required; non-empty
      "title": "Q3 Results",           // optional
      "section": "Finance > Q3",       // optional
      "anchor": "docs/report.pdf#page=12", // optional anchor that deep-links into the source
      "locator": {"kind": "page", "value": 12}, // optional; see locator fields
      "metadata": {"author": "ACME"}   // optional; JSON values (scalars, arrays, or nested objects)
    }
  ],
  "text": "....",                      // (b) plain extracted text (indexer chunks it)
  "metadata": {}                       // optional document-level metadata
}

Document-level fields:

schema_version (required, integer) - the schema major. A schema_version newer than the running vaultspec-rag is rejected with an “upgrade vaultspec-rag” message.
preprocessor_id (required, non-empty string) - your extractor’s id, surfaced in preprocess run-one output.
preprocessor_version (required, non-empty string) - your extractor’s version.
source_path (required, non-empty string) - the path you were given.
units (one of units or text) - pre-chunked units you produced.
text (one of units or text) - plain extracted text the indexer runs through its normal text splitter.
metadata (optional) - document-level metadata, JSON values (scalars, arrays, or nested objects).

Unit fields (each entry in units):

text (required, non-empty string) - the unit’s body.
title (optional) - a heading for the unit.
section (optional) - a breadcrumb path within the document.
anchor (optional) - an anchor that deep-links into the source, surfaced verbatim in search results (for example report.pdf#page=12).
locator (optional) - a typed pointer into the source’s own addressing scheme, rendered as, for example, page 12 or sheet Summary.
metadata (optional) - per-unit metadata, JSON values (scalars, arrays, or nested objects).

Locator fields (the optional locator object):

kind (required) - one of byte, page, sheet, line, char, or none.
value (required) - an integer (page, line, byte, char) or a string (sheet name).
end (optional) - an integer or string marking the end of a range.

Rules:

Provide either units (you chunk) or text (the indexer chunks it), never both and never neither. When you provide units, it must be non-empty.

Batch hooksLink to Batch hooks

With one subprocess per file, interpreter startup dominates the cost of a cheap extractor. On one development machine, a bare python noop hook cost about 100 ms per file (about 220 ms through uv run), versus about 1 ms per file when one spawn handled 100 files. Your numbers will differ. On a first index or a clean rebuild that constant is paid for every matched file. A batch hook amortizes it: one subprocess processes many files at once.

Opt in per rule with batch = true. A batch rule’s command receives a manifest of source paths via a {paths} placeholder (not the per-file {path}) and emits a JSON array of per-file outputs:

[[rule]]
pattern  = "*.pdf"
command  = "python tools/pdf_batch.py {paths}"   # {paths} is the manifest file path
target = "document"
extractor_version = "1.0.0"
batch    = true
on_error = "skip"
timeout_s = 5                                     # per-file budget; see Bounds scale with the batch

Rules for a batch rule:

command only. batch = true is rejected on an entry_point rule (the entry-point form keeps per-file semantics), on a command missing {paths}, or on a command that also carries {path}. A non-batch command carrying {paths} is likewise rejected. Like any config defect, an invalid batch rule is dropped (or fails preprocess check in strict mode).

The manifest and response contract:

Manifest - vaultspec-rag replaces {paths} with the path of a temp file holding one absolute source path per line, UTF-8 encoded. Read it, process each path, and delete nothing (vaultspec-rag removes the manifest after the run).
Response - print one JSON array on stdout. Each element is a normal output object plus a "path" field naming the source file it belongs to (the absolute path from the manifest). vaultspec-rag maps each element back to its file by that "path".

[
  {
    "path": "/abs/docs/a.pdf",         // required; maps this element to its source file
    "schema_version": 1,
    "preprocessor_id": "pdf-batch",
    "preprocessor_version": "1.0",
    "source_path": "/abs/docs/a.pdf",
    "units": [ /* ... */ ]
  },
  { "path": "/abs/docs/b.pdf", "schema_version": 1, /* ... */ "text": "..." }
]

Bounds scale with the batch:

Timeout - the wall-clock budget is timeout_s * (files in the batch), capped at 600 s, so the per-file budget you declared scales with the work handed over.
Stdout cap scales the same way so the whole array fits; the per-file emitted-text cap (VAULTSPEC_RAG_PREPROCESS_MAX_EMITTED_BYTES) still applies unchanged to every element.

Failure handling is per file:

A file missing from the response array, or whose element fails schema validation or the emitted-text cap, is resolved through the rule’s on_error for that file alone (skip / passthrough / fail); the other files in the batch are unaffected.
A malformed envelope - non-JSON, not an array, a non-zero exit, or a timeout - fails the whole batch, and every file in it is resolved through on_error.
on_error = "fail" aborts the index run on the first affected file, exactly as it does per file.

on_error = "fail" has the same meaning for batch and single-file rules: the first affected file raises a preprocessing abort that propagates through the worker pool and stops the index run. skip and passthrough remain per-file outcomes, so unaffected members of a valid batch can still succeed.

Batch results are cached per file under the same cache key, so cache hits keep bypassing the hook entirely and a mixed hit/miss set shrinks the manifest to just the misses. vaultspec-rag groups matched files into manifests of at most 64 paths, each group a single spawn.

Cache and incremental indexingLink to Cache and incremental indexing

vaultspec-rag caches successful extraction output under the data directory. The key binds the source path and content hash to the output schema and canonical execution fingerprint, including the target, extractor version, command or entry point, and options. An unchanged file is not re-extracted on a full or resumed reindex. A changed input or execution fingerprint produces a new key. Only successful outputs are cached, so a transient extractor failure is never made sticky. Cross-path reuse requires the rule’s explicit path_independent = true declaration.

To force re-extraction of unchanged files after upgrading your extractor, bump extractor_version. Collection cleanup is intentionally separate from cache lifecycle: cleaning or rebuilding code, document, or combined does not erase extraction-cache entries. Their complete execution fingerprints make obsolete entries unreachable without coupling one domain’s cleanup to another domain’s extractor work.

From the CLI: vaultspec-rag index --rebuild --type document (or --type combined), or vaultspec-rag clean document followed by a fresh index. See rebuild from scratch for the full rebuild and clean surface.
From the Model Context Protocol (MCP): use the targeted document or combined reindex and clean tools.

The filesystem watcher routes a changed matched file (an edited .pdf, for example) through your extractor under the same watcher timing as code changes: an adaptive coalescing window, a freshness deadline, and a cooling delay after a successful run. See automatic convergence for the timing settings.

Failure visibilityLink to Failure visibility

Coverage gaps are the problem this feature exists to remove, so they’re never silent. Files skipped by an on_error = "skip" rule are counted and listed on every path:

IndexResult.preprocess_skipped and preprocess_failures on a full index,
the ~N suffix on the vaultspec-rag server jobs reindex summary,
the preprocess_skipped and preprocess_failures fields in vaultspec-rag index --json,
and a warning in the service log for every skip.

Success is equally visible. Extractors run in worker subprocesses whose own logging never reaches the service log, so the indexer counts every rule-fed file and surfaces the tally where the skips already live: IndexResult.preprocess_ok, the preprocess_ok field in vaultspec-rag index --json and on the reindex job record, and the files, preprocess_ok, and preprocess_skipped fields on the service log’s service.index completed event. A code-domain event also carries preprocess_rules, the number of preprocess rules routed in that run. A run whose rules matched nothing reports preprocess_ok=0 there - the discriminating signal when rule-fed content seems to be missing from the index.

For a non-interactive client of the resident service, two response fields carry the same visibility. A reindex job record from /jobs (and vaultspec-rag server jobs --json) carries preprocess_skipped and preprocess_failures, so the client sees exactly which files a hook failed to extract. The /reindex response also includes a preprocess pre-flight block reporting whether the root ships a config, its resolved rule count, the effective mode, and the hook state, mirroring the notice server start prints - so a client learns whether hooks will fire before the job runs. Its hooks field is active, disabled (the kill switch), unapproved, or none, and remediation carries the command an operator has to run. A client cannot approve a project; only the operator can.

Size limitsLink to Size limits

Input and output are both bounded. Source bytes are streamed and checked against the active support profile and optional max_source_bytes before an extractor launches. Emitted encoded bytes, unit counts, payloads, queue weight, subprocess output, timeout, and no-progress time are bounded independently. A limit failure remains a typed per-file outcome and cannot silently publish the file as converged. See the configuration reference for the operational ceilings.

Security postureLink to Security posture

A root’s .vaultragpreprocess.toml is code execution with your privileges. Its rules name arbitrary local commands that run under the account running the service - the same trust class as that repo’s make, npm install, or any of its build scripts - and anyone who can change the repository can change that file. So the file never authorises itself: a project’s rules run only after you approve them on this machine.

ApprovalLink to Security posture, Approval

vaultspec-rag preprocess approve records your approval of the project’s rules exactly as they are. Review them first with vaultspec-rag preprocess list.
Approval is bound to the project directory and to the exact bytes of the file. Any edit (a new command, a wider pattern, even a comment) returns the project to unapproved, and the same file in another clone or worktree needs its own approval.
Until then nothing runs. Indexing, the watcher, the service, and preprocess run-one all skip the rules and say so: status and preprocess status report configured but not approved, server start prints a notice, the /reindex response and index --dry-run --json carry the unapproved hook state, and every skipped file is listed as preprocessing awaiting approval; not extracted.
Content extracted before the project became unapproved is not all kept. Extracted documents stay searchable. Extracted content routed to the code domain survives the watcher’s per-file updates, but a full index or an unscoped incremental one removes it, and it returns once you approve and index again. The extraction cache means that costs embedding time, not extractor time.
Only the operator of this machine can approve. There is no approval over HTTP or the Model Context Protocol, no environment variable that approves, and no setting that trusts every project.
Approvals are stored in the service directory (~/.vaultspec-rag by default), never in the repository.

An unattended deployment approves as part of provisioning: run preprocess approve for each project after your own review step, and again whenever its rules change.

What approval does not doLink to Security posture, What approval does not do

Approval is consent, not containment. An approved hook runs with no sandbox, so its filesystem and network access are those of the account running the service: it can read and write what you can, and reach the network as you can. Approval also covers the rule file and not the programs its commands launch. If a rule runs python tools/xlsx_extract.py, a later change to that script runs without asking again. Approve a project’s rules only if you would build that project, and review .vaultragpreprocess.toml exactly as you would a build script or a CI job.

Two bounds still apply to every hook, and they earn their place at near-zero cost rather than as a security boundary:

Secrets - the child runs under a curated environment stripped of every VAULTSPEC_RAG_* knob and every credential (Qdrant API key, model-registry and cloud tokens, Git tokens), so a hook inherits none of the daemon’s secrets.
Process, time, and output bounds - the extractor runs out-of-process as a subprocess grandchild, which also keeps it off the GPU worker. Its working directory is the project root, so uv run, npm exec, and other project launchers resolve the project exactly as they do under preprocess run-one. A wall-clock timeout_s and stdout and stderr caps bound a misbehaving extractor in time and output.

vaultspec-rag preprocess status reports whether a root ships a config, its resolved rule count, whether those rules are approved, and the effective mode.

VAULTSPEC_RAG_PREPROCESS=off is the kill switch and wins over everything, approval included: no root’s rules run, ever. server start --no-preprocess disables them for that service. Note that index --no-preprocess applies to in-process indexing only: when a service is already running, index hands the work to it and the flag has no effect. See the configuration reference for the full variable and flag inventory.

One operational note: the index tracks the preprocess configuration it was built with, so changing whether a root’s hooks run - toggling off, or granting or losing approval - triggers an automatic rebuild on the next index run. That is correct, but expensive on a large corpus, so approve a project before its first index, and re-approve an edited policy before the next run picks the edit up.

Parser capability is not admissionLink to Parser capability is not admission

The indexer has plain-text and HTML-normalization capabilities that explicit routes may use for formats without an abstract syntax tree (AST) grammar. Those capabilities do not make every readable or supported extension source code. The source profile or an explicit target decides ownership first; only then does the owning pipeline select a parser or extractor. This separation is what keeps arbitrary XML, schemas, workbooks, generated data, and other project material out of code indexing by default.

Illustrative extractors (project-side, not shipped)Link to Illustrative extractors (project-side, not shipped)

These sketches show the schema generalizing across formats. They’re examples for your own tools/, not dependencies of vaultspec-rag. Licences are flagged because extractor choice affects your project’s licence posture.

PDF - pypdf (BSD-3-Clause):

import json, sys
from pypdf import PdfReader  # BSD-3-Clause

src = sys.argv[1]
reader = PdfReader(src)
units = [
    {
        "text": page.extract_text() or "",
        "anchor": f"{src}#page={i + 1}",
        "locator": {"kind": "page", "value": i + 1},
    }
    for i, page in enumerate(reader.pages)
]
print(
    json.dumps(
        {
            "schema_version": 1,
            "preprocessor_id": "pypdf",
            "preprocessor_version": "1.0",
            "source_path": src,
            "units": units,
        }
    )
)

PyMuPDF / fitz is faster but AGPL-3.0, whose terms propagate to your project’s license; prefer pypdf (BSD-3) or pdfplumber (MIT) for a licence-clean project.

XLSX - openpyxl (MIT): iterate worksheets, then rows; the sheet name is the locator ({"kind": "sheet", "value": ws.title}). Legacy .xls needs xlrd (BSD) or a conversion step.

DOCX - python-docx (MIT): iterate paragraphs; the locator is the paragraph index (Word has no render-time page numbers).

XML / XSD - stdlib xml.etree (PSF): walk elements, emit element text with a tag-path anchor; reach for lxml (BSD) only if you need XPath or source line numbers.

Project preprocessing hooks#Link to Project preprocessing hooks

How it works#Link to How it works

Configure rules#Link to Configure rules

Inspect, validate, and approve your configuration#Link to Configure rules, Inspect, validate, and approve your configuration

Invocation envelope#Link to Invocation envelope

Output schema#Link to Output schema

Batch hooks#Link to Batch hooks

Cache and incremental indexing#Link to Cache and incremental indexing

Failure visibility#Link to Failure visibility

Size limits#Link to Size limits

Security posture#Link to Security posture

Approval#Link to Security posture, Approval

What approval does not do#Link to Security posture, What approval does not do

Parser capability is not admission#Link to Parser capability is not admission

Illustrative extractors (project-side, not shipped)#Link to Illustrative extractors (project-side, not shipped)

See also#Link to See also