Ingestion · deep dive

Getting data in

Verity ingestion is a layered funnel: many doors, one structural choke point. Every door demands an explicit visibility decision and terminates at the same fail-closed gate in the Rust write path. This page walks every entry point with a working example, from the five-minute CLI wow to source-fidelity native connectors.

The BYOT doctrine: bring your own token

Verity ships zero OAuth apps. Bring-your-own-token is the only auth mode of the open-source core. Every connector quickstart reads the same way: create a key, private app, service account, or self-registered OAuth client in your own tenant, and paste it into Verity. The July 2026 coverage survey confirmed this is viable for 20/20 surveyed systems: no vendor-registered app is required for a customer to grant first-party access.

Three engineering consequences are absorbed once in the ingest SDK rather than per connector:

• Credential lifecycle, four shapes: static key/PAT, client-credentials minting, refresh-token rotation, and service-account JWT, plus expiry telemetry and a 401-invalidate-retry-once hook.
• Webhook receivability, three ways: a public HTTPS ingest endpoint; Socket-Mode delivery (Slack); or a delta/poll fallback so no source is ever push-blocked, including firewalled self-hosters.
• Credential wizards, not OAuth clients: verity-cli connect slack opens the app-from-manifest flow; verity-cli connect github uses a pasted fine-grained PAT once, then never stores it.

The layered funnel

Entry points, in adoption order. Each reuses the same push lane, HMAC scope handles, and fail-closed quarantine. The two lanes, convenience (snapshot-grade, admin-assigned visibility) and truth (source-fidelity ACLs, push freshness), are labeled everywhere and never blurred.

The one choke point

No SDK, sink, CLI flag, or endpoint can bypass the gate that governs all of them:

1 · verity-cli

The developer front door. One binary drives a running server over REST. It holds no database access and no enforcement logic of its own.

dev: the whole local plane, one command

cargo run --release -p verity-cli -- dev
# docker Postgres + the server + a "dev" tenant + an org-wide scope handle + a written config.
# Fully idempotent — re-run it anytime.

add --visibility: ingest with an explicit visibility

add accepts a file, a directory (recursive over .txt/.md/.json/.csv/.html, capped at 200 files), an http(s) URL, or stdin (-). --visibility is required by the argument parser and has no default. Verity never guesses who may see a memory. Omitting it is a usage error that names the invariant. Under the hood, add mints a short-lived scope whose principals are exactly your --visibility tokens and uploads under that handle.

verity-cli add ./docs --visibility 1                       # a directory, org-wide token
verity-cli add report.md --visibility 7,11 --entity account:acme  # narrow visibility + entity tag
echo "quick note" | verity-cli add - --visibility 1              # stdin

query: scoped hybrid recall

verity-cli query "what do we know about pricing?"   # uses the handle saved by `dev`
verity-cli query "renewal risks" -k 5 --json       # custom k, raw JSON hits

Every hit already passed your scope's visibility pre-filter in the index. There is no post-hoc filtering step to forget.

webhook mint: any JSON-POSTing system as a source

verity-cli webhook mint my-system --visibility 1
# The token in the returned URL is the credential, shown exactly once.
# Ingested memory carries source "webhook:my-system".

Watch what the gate refuses with verity-cli tail: it polls the quarantine and prints the raw payloads Verity declined to index permissively.

2 · Canonical envelope endpoints

The curl-in-60-seconds lane. Two shapes cover most structured ingestion.

POST /v1/episodes: an observation

Writes a durable observation under a scope handle. Visibility is inherited from the handle's principals; entities must be inside the handle's entity scope (omit to inherit the whole scope).

curl -s -X POST "$VERITY/v1/episodes" -H 'content-type: application/json' -d '{
  "scope_handle": "'"$HANDLE"'",
  "observation": "Acme confirmed the renewal moves to their Q4 board meeting.",
  "entities": ["account:acme"]
}'
# → { "episode_id": "0192..." }  — immediately searchable via /v1/recall

POST /v1/ingest/debezium: the CDC truth lane

Accepts a Debezium change envelope (or a JSON array of them) and turns it into a deterministic keyed L1 upsert. This endpoint is on the trusted connector plane: it is gated by the admin bearer token, not a scope handle. The ?tenant_id= query param names the tenant; ?pk= overrides the primary-key field (default id).

curl -s -X POST "$VERITY/v1/ingest/debezium?tenant_id=$TENANT" \
  -H 'content-type: application/json' \
  -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"payload":{
       "after":{"id":"opp-1","amount":84000,"stage":"negotiation"},
       "source":{"connector":"postgresql","db":"crm","table":"opportunities","ts_ms":1752000000000},
       "op":"u"}}'
# → { "facts_inserted":0, "facts_superseded":3, "facts_unchanged":0, "facts_retired":0 }

An op:"u"/"c" upserts each field of the row image; op:"d" retires the entity. Facts written this way carry mirrored provenance and record a freshness sample. The value becomes queryable at GET /v1/records/postgresql:crm.opportunities/opp-1/amount, measured at 31 ms POST-to-queryable on the dev profile.

3 · Minted scoped webhook URLs

Any system that can POST JSON (GitHub, Stripe, Zapier, n8n, cron jobs, internal services) becomes a push source with no connector and no OAuth. The visibility policy is bound at mint time into the URL; the blast radius is one URL, instantly revocable.

curl -s -X POST "$VERITY/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"tenant_id":"'"$TENANT"'","name":"demo-system","visibility":[11],
       "entity_scope":["account:acme"]}'
# → { "webhook_id":"0192...", "url":"/wh/n8sQ...raw-token" }

curl -s -X POST "$VERITY/wh/n8sQ...raw-token" -H 'content-type: application/json' \
  -d '{"content":"Acme signed the pilot agreement for the fall rollout.",
       "entities":["account:acme"]}'
# → 200 { "episode_id":"...", "chunks_indexed":1, "facts_written":0 }

curl -s -X POST "$VERITY/wh/n8sQ...raw-token" -H 'content-type: application/json' \
  -d '{"weird":"vendor shape we do not understand"}'
# → 202 { "quarantined": true }   (view it: verity-cli tail  /  GET /v1/admin/quarantine)

Facts and chunks from a webhook carry admin-assigned provenance: the visibility was an explicit admin decision at mint, not a mirrored source ACL. Revoke with DELETE /v1/webhooks/{id}; the URL stops resolving immediately.

4 · File drop

POST /v1/files is a multipart upload: scope_handle, an optional comma-separated entities, and the file part. The server splits the text into paragraph chunks and embeds each. Visibility is inherited from the scope handle the file is uploaded under.

curl -s -X POST "$VERITY/v1/files" \
  -F "scope_handle=$HANDLE" \
  -F "entities=account:acme" \
  -F "file=@handbook.md"
# → { "episode_id":"...", "chunks_indexed": 12 }

5 · Framework sinks

Six thin adapters plug into each framework's documented memory extension point, turning hundreds of community loaders into Verity sinks. Every one takes a required visibility_policy constructor argument with no default. Loaders strip source ACLs by construction, so this lane is always policy-based (admin-assigned provenance), and anything arriving without a policy hits quarantine. Bypass is impossible, not discouraged.

LlamaIndex

from verity_llamaindex import VerityVectorStore

store = VerityVectorStore(
    verity_url="http://127.0.0.1:7717",
    tenant_id=TENANT,
    visibility_policy=[1],          # REQUIRED — no default; missing → quarantine
    admin_token=ADMIN_TOKEN,
)
# every LlamaHub reader now writes into permission-aware Verity memory

LangChain

from verity_langchain import VerityVectorStore

store = VerityVectorStore(verity_url=URL, tenant_id=TENANT,
                          visibility_policy=[1], admin_token=ADMIN_TOKEN)
retriever = store.as_retriever()   # or verity_langchain.VerityRetriever

LangGraph

from verity_langgraph import VerityStore

store = VerityStore(verity_url=URL, tenant_id=TENANT,
                    visibility_policy=[1], admin_token=ADMIN_TOKEN)
# put() ingests; search() mints an entity-scoped handle and recalls; get() reads the brief

CrewAI

from verity_crewai import VerityStorage   # StorageBackend (CrewAI 1.0)

storage = VerityStorage(verity_url=URL, tenant_id=TENANT,
                        visibility_policy=[1], admin_token=ADMIN_TOKEN)

Google ADK

from verity_adk import VerityMemoryService

memory = VerityMemoryService(verity_url=URL, tenant_id=TENANT,
                             visibility_policy=[1], admin_token=ADMIN_TOKEN)
# add_session_to_memory(session) ingests; search_memory(...) recalls

OpenAI Agents SDK

from verity_openai_agents import VeritySession

session = VeritySession("conv-42", verity_url=URL, tenant_id=TENANT,
                        visibility_policy=[1], admin_token=ADMIN_TOKEN)
# add_items() ingests; get_items() recalls, ordered by item seq

6 · MCP write tools

Paste one MCP config into Claude Code or Cursor and the agent itself becomes the universal connector. Zero installs for users already in an agent. Four write tools reach the same choke point; visibility is inherited from the agent's scope handle, so there is no way to write something the caller couldn't already see.

memory_remember

Append a one-line observation (immutable episode + immediately searchable chunk). For durable knowledge learned during the task.

memory_ingest_text

Ingest a document-sized piece of text verbatim: pasted docs, transcripts, reports held in-context.

memory_ingest_file

Read a local UTF-8 text file (.txt/.md/.json/.csv/.html, ≤ 512 KB) and ingest it.

memory_ingest_url

Fetch a public http(s) URL (10 s timeout, 2 MB cap), reduce HTML to text, and ingest it.

// an agent's tool call — the scope_handle comes from memory_open_scope,
// which fixes identity from server config, never from arguments
{ "tool": "memory_ingest_text",
  "arguments": {
    "scope_handle": "vs_9f2...",
    "content": "Meeting notes: Acme wants net-30 terms and a Q4 close...",
    "entities": ["account:acme"] } }

See the API reference for all 14 MCP tools.

7 · Native connectors (the truth lane)

Native connectors give source-fidelity ACL envelopes and push freshness: the graduation path and the moat. Each is configured with a first-party credential you create in your own tenant. The Python ingest plane runs them; connectors are never on the read path.

HubSpot: Service Key, ACL tier C

Auth is a Service Key created in your own portal (Development → Keys → Service keys, ~2 min), read from HUBSPOT_SERVICE_KEY (HubSpot deprecated private apps in 2026; the legacy HUBSPOT_PRIVATE_APP_TOKEN still works; both are bearer tokens). HubSpot exposes no per-record ACL API, so it is ACL tier C: the connector cannot mint a faithful ACL envelope and instead requires an admin-assigned visibility_policy with no default. Every emitted event carries it; provenance is admin-assigned. The truth lane polls the CRM search API on hs_lastmodifieddate (contacts use lastmodifieddate, a documented quirk); the push lane maps UI-configured v3 webhook payloads.

export HUBSPOT_SERVICE_KEY=pat-na1-...
python -m verity_ingest.connectors.hubspot --once --visibility 1,2
python -m verity_ingest.connectors.hubspot --webhook-file payload.json --visibility 1,2

Google Drive: service account, ACL tier A (mirrored)

Auth is your own service account with domain-wide delegation configured in your admin console; the key path comes from GOOGLE_APPLICATION_CREDENTIALS. Drive is ACL tier A (permissions.list is best-in-class), so this is the mirroring proof. The connector polls changes.list, and for each changed file fetches metadata, then the ACL, then content only if the ACL resolves (Google Docs exported as text; text/* and JSON downloaded directly). ACL-before-content is the rule.

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
python -m verity_ingest.connectors.gdrive --once            # --dry-run prints request bodies

Because the ACL is mirrored, Drive documents carry mirrored provenance: a Verity scope inherits exactly the Drive permissions on each file. This is how a Drive ACL becomes an in-index pre-filter (see Permissions → ACL inheritance).

Gmail: service account, ACL tier A (mirrored participants)

The same BYOT service account with domain-wide delegation as Drive (GOOGLE_APPLICATION_CREDENTIALS + the gmail.readonly scope, impersonating the mailbox owner). Each message is keyed by its RFC822 Message-ID, so the same email seen in two connected mailboxes is indexed once. The ACL mirrors the message participants: From/To/Cc become user:<email> visibility, carrying mirrored provenance; a message whose participants don't resolve quarantines rather than indexing permissively.

python -m verity_ingest.connectors.gmail --backfill --subject you@corp.example   # --once to poll; --dry-run to preview

Salesforce: customer-created Connected App, ACL tier A (hardest)

Auth is a Connected App you create in your own org with the client-credentials flow (SF_MY_DOMAIN / SF_CLIENT_ID / SF_CLIENT_SECRET). Salesforce's client-credentials response carries no expires_in and no refresh token, so the token is cached until a 401 triggers the shared retry-once hook. The truth lane runs SOQL on LastModifiedDate for Account, Contact, and Opportunity.

Debezium / Postgres CDC: the built-in truth lane

Anything Debezium speaks (Postgres, MySQL, …) posts change envelopes to POST /v1/ingest/debezium (see above). ACL fidelity is mirrored when the schema declares principal columns via a manifest, or admin-assigned under a static policy. For a lightweight on-ramp, a copy-paste pg_net/Supabase trigger POSTs row changes to a minted webhook URL: CDC-lite into the same lane (31 ms POST-to-queryable).

Google Workspace directory sync: the Identity Plane connector

The content connectors above emit group:<email> ACLs but deliberately don't expand them. Nested-group closure is this connector's job. Same BYOT service account with domain-wide delegation (scopes admin.directory.user.readonly + admin.directory.group.readonly). It reconciles users and groups from the Admin SDK and writes direct membership edges into the SpiceDB ReBAC graph, nesting preserved. SpiceDB owns the transitive closure; the reconcile interval is the group-membership freshness bound. Proven live against a real Google Workspace directory: a doc shared with a group resolves to its nested members via open_scope, non-members provably dark. Run it supervised as a server-owned worker with verity-cli dev --directory.

python -m verity_ingest.connectors.gdirectory --once --subject admin@corp.example --domain corp.example

8 · Source manifests (connectors as config)

A source manifest turns any REST/webhook source into a reviewable YAML file: no connector code, no vendor OAuth. The manifest is data, not code: the Rust runtime (crates/verity-manifest) interprets it with hard evaluator limits and a fail-closed ACL gate; nothing in a manifest can execute or touch the network.

The shipped, fixture-verified example is examples/linear.yaml:

manifest_version: 1
source:
  name: linear                       # [a-z0-9_-]; becomes the L1 `source`
  tier: B                            # A | B | C — gate-enforced ACL contract
  webhook:
    signature:
      scheme: hmac_sha256            # failed signature → 401, no ingestion at all
      header: Linear-Signature
      secret_ref: secret://linear-webhook-secret
entities:
  - type: issue
    route:
      when: "type = 'Issue' and action in ['create','update']"
      operation: upsert              # upsert only in v1
    primary_key: "data.id"           # deterministic PK → idempotent replays
    valid_from: "data.updatedAt"     # dot-path or $now()
    map:
      title: "data.title"
      state: "data.state.name"
    content: "data.description"      # optional: text → a retrieval chunk
acl_policy:                          # absent ⇒ activation refused; runtime quarantines all
  mode: map                          # map | static | quarantine
  identity_namespace: source_native_id
  principals: "data.team.members[].id"
  approximation: true                  # mandatory for Tier B; note surfaces to the approver
  note: "Team membership approximates issue visibility."

acl_policy: three modes, tier contracts, human gate

The tier contract is gate-enforced: Tier A must be map with approximation:false; Tier B must be map with approximation:true and a human-readable note; Tier C must be static. Activating a manifest requires an explicit approved_by recorded in the audit log. An LLM may draft everything except the acl_policy. An unreviewed manifest can only quarantine, structurally.

# upload (draft) → activate (human gate) → bind to a webhook
curl -s -X POST "$VERITY/v1/manifests" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","yaml":"'"$(cat linear.yaml)"'"}'
curl -s -X POST "$VERITY/v1/manifests/$ID/activate" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","approved_by":"admin@corp.example"}'
curl -s -X POST "$VERITY/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
  -d '{"tenant_id":"'"$TENANT"'","name":"linear","visibility":[1],"manifest_id":"'"$ID"'"}'

ACL provenance & the quarantine rule

Every fact and chunk carries exactly one provenance tag, surfaced on every read:

• mirrored: from a real source ACL API (Drive permissions.list).
• approximated: from a proxy for the real ACL (team membership).
• admin-assigned: from an explicit policy at ingest (webhook, CLI, sink, static manifest).
• quarantined: no usable visibility could be derived; held for review.

Entry-point comparison

Pick a door by the trade-off you want. Freshness and ACL fidelity climb together toward the truth lane; effort climbs with them.