Connecting · per-system runbooks
Connecting your systems
Ingestion is the mechanism reference: every door and how the write path treats it. This page is the operator's counterpart: “I have HubSpot / Salesforce / a Google Drive / a Postgres database / a pile of docs / an internal system. Walk me through connecting it and getting my existing data in for the first time.” One runbook per system, grounded in the real connector source.
Just trying Verity? Do the file drop: verity-cli add ./dir --visibility 1, 60 seconds, no credential. Connecting a real system? Jump straight to yours: HubSpot, Salesforce, Google Drive, Postgres/MySQL, or any internal system. Every runbook is the same five steps.
The universal shape
Verity ships zero OAuth apps. Bring-your-own-token is the only auth mode of the open-source core: you create a first-party credential in your own tenant (a HubSpot private app, a Salesforce Connected App, a Google service account) and paste it into Verity. Your tokens never rest in a third-party cloud. Every runbook below reads the same five steps:
Common prerequisites
Every runbook assumes a running Verity server and a tenant. Create both once:
# local all-in-one: docker Postgres + server + a "dev" tenant + a scope handle
cargo run --release -p verity-cli -- dev
# or, against an existing server, mint a tenant on the trusted admin plane
export VERITY_URL=http://127.0.0.1:7717
export VERITY_ADMIN_TOKEN=... # the server's admin bearer, if configured
export TENANT=$(curl -s -X POST "$VERITY_URL/v1/admin/tenants" \
-H 'content-type: application/json' -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
-d '{"name":"acme-corp"}' | jq -r .tenant_id)The Python ingest plane (connectors) reads three shared env vars, plus each connector's own credential:
- VERITY_URL
- The server base URL the connector's sink POSTs to. Debezium-lane connectors (HubSpot, Salesforce) default to
http://127.0.0.1:7717; the Drive connector defaults tohttp://localhost:8080, so set it explicitly to your server. - VERITY_TENANT_ID
- The tenant UUID from above. Required by the Debezium sink; the Drive runner falls back to
"default". - VERITY_ADMIN_TOKEN
- Sent as the bearer token on the trusted connector plane (
/v1/ingest/debezium,/v1/ingest/documents,/v1/admin/principals) when the server requires it.
Install the ingest SDK: pip install verity-ingest (the Drive connector needs the extra: pip install 'verity-ingest[gdrive]' for google-auth).
HubSpot
The flagship connector. HubSpot exposes no per-record ACL API, so it is ACL tier C: the connector cannot mint a faithful ACL envelope and instead requires an admin-assigned visibility_policy with no default. Provenance on every emitted fact is admin-assigned.
1 · Create the credential
In your own HubSpot portal, go to Development → Keys → Service keys, create a Service Key, grant it the CRM read scopes you want mirrored (crm.objects.contacts.read, .companies.read, .deals.read), and copy it; it looks like pat-na1-.... This is BYOT: no vendor-hosted app, ~2 minutes. (HubSpot deprecated private apps in 2026. A legacy private-app token still works via HUBSPOT_PRIVATE_APP_TOKEN; both are used as a bearer token.)
2 · Configure
export HUBSPOT_SERVICE_KEY=pat-na1-...
export VERITY_URL=http://127.0.0.1:7717
export VERITY_TENANT_ID=$TENANT
export VERITY_ADMIN_TOKEN=... # if the ingest plane requires it3 · Run the initial backfill
--once runs one truth-lane poll cycle. With no saved cursor this is a full backfill: it searches contacts, companies, and deals on the last-modified property (hs_lastmodifieddate; contacts use lastmodifieddate, a documented HubSpot quirk) from epoch, ascending, paginated at 100/page, honoring the search API's 429 Retry-After. Each non-null property becomes one deterministic keyed L1 fact via POST /v1/ingest/debezium; the last-modified timestamp becomes valid_from. The cursor (max last-modified seen) is written to the state file at the end.
python -m verity_ingest.connectors.hubspot --once --visibility 1,2
# poll: 148 fact event(s), cursor -> 2026-07-09T18:04:57.000Z -> {written:148,...}
# cursor persists to .verity/hubspot_cursor (override: --state-file / $HUBSPOT_STATE_FILE)--visibility is required and has no default. It is the admin-assigned principal-token set every fact carries (HubSpot has no per-record ACL to mirror). Omitting it is a usage error that names the invariant; passing an empty set fails closed.4 · Verify
Facts land keyed (source="hubspot", table=<object>, id=<record id>, field); the L1 partition is hubspot:<object>. Point-read a field, then confirm scoping: a scope whose principals include your policy tokens sees it; one that does not sees nothing.
# mint a scope handle whose principals include a policy token (e.g. 1)
IN=$(curl -s -X POST "$VERITY_URL/v1/scopes" -H 'content-type: application/json' \
-d "{\"tenant_id\":\"$TENANT\",\"principals\":[1],\"entity_scope\":[]}" | jq -r .scope_handle)
# the record is queryable at its keyed L1 address
curl -s "$VERITY_URL/v1/records/hubspot:deals/<deal-id>/amount?scope_handle=$IN" | jq .value
# → "84000"
# a scope holding NO policy token sees nothing, fail closed
OUT=$(curl -s -X POST "$VERITY_URL/v1/scopes" -H 'content-type: application/json' \
-d "{\"tenant_id\":\"$TENANT\",\"principals\":[999],\"entity_scope\":[]}" | jq -r .scope_handle)
curl -s -o /dev/null -w '%{http_code}\n' "$VERITY_URL/v1/records/hubspot:deals/<deal-id>/amount?scope_handle=$OUT"
# → 404 (out-of-scope value is not visible to this principal set)5 · Ongoing sync
Two options. Re-run --once on a cron (the saved cursor makes it incremental), or push: HubSpot v3 webhook subscriptions are UI-configured under the private app and deliver to a Verity-minted webhook URL. Decode a recorded delivery with --webhook-file (only *.propertyChange events map to facts):
python -m verity_ingest.connectors.hubspot --webhook-file payload.json --visibility 1,2For a supervised schedule instead of cron, see Ongoing sync (Temporal).
Salesforce
Salesforce is ACL tier A (the *Share tables are readable) but the hardest reconstruction of the 20 surveyed systems. The connector does not reconstruct full effective visibility; enforcement uses the same fail-closed admin-assigned visibility_policy as HubSpot, and share rows ride along as approximated additive metadata.
1 · Create the credential
In your own org, create a Connected App and enable the OAuth client-credentials flow with a run-as integration user. This survived the post-Sept-2025 crackdown that made vendor-distributed apps harder; customer-created stays easy. Copy the consumer key and secret. Note your My Domain (e.g. acme → acme.my.salesforce.com). Salesforce's client-credentials response carries no expires_in and no refresh token, so the token is cached until a 401 (INVALID_SESSION_ID) triggers the shared retry-once hook.
2 · Configure
export SF_MY_DOMAIN=acme # or the full acme.my.salesforce.com host
export SF_CLIENT_ID=... # Connected App consumer key
export SF_CLIENT_SECRET=... # Connected App consumer secret
export VERITY_URL=http://127.0.0.1:7717
export VERITY_TENANT_ID=$TENANT
export VERITY_ADMIN_TOKEN=...3 · Run the initial backfill
One cycle runs SOQL through GET /services/data/v62.0/query for Account, Contact, and Opportunity with WHERE LastModifiedDate > <cursor> ORDER BY LastModifiedDate ASC. With no cursor that is a full backfill from epoch, following nextRecordsUrl (queryMore) pagination. Each non-null field becomes one keyed L1 fact. For Accounts changed in the window, the connector additionally fetches AccountShare rows and attaches best-effort share principals (user:005…, group:00G…) as approximated metadata. A failed share fetch logs a warning and never gates the facts.
python -m verity_ingest.connectors.salesforce --once --visibility 1,2
# poll: 312 fact event(s), cursor -> 2026-07-09T18:04:57.000+0000 -> {written:312,...}
# --no-shares skips the AccountShare fetch; cursor -> .verity/salesforce_cursor--visibility (admin-assigned), same fail-closed posture as HubSpot.4 · Verify
Records land at the salesforce:<sobject> partition, keyed by the Salesforce Id. Same scoped check as HubSpot. An in-policy scope sees it; an out-of-policy scope gets a 404:
curl -s "$VERITY_URL/v1/records/salesforce:Account/<account-id>/Name?scope_handle=$IN" | jq .value
# → "Acme Corporation"
curl -s "$VERITY_URL/v1/records/salesforce:Opportunity/<opp-id>/Amount?scope_handle=$IN" | jq .value
# → "84000.0" (out-of-policy scope → 404, fail closed)5 · Ongoing sync
Re-run --once on a cron or Temporal schedule; the saved cursor makes each cycle incremental (truncated to whole seconds, so a ≤1s window may replay safely into keyed upserts). The push lane, Salesforce CDC over the Pub/Sub API (gRPC), is a documented no-op in this poll-first connector; the truth lane reconciles everything. Live-org validation awaits a design partner (see Status & honesty).
Google Drive
Drive is ACL tier A and the mirroring proof: permissions.list is best-in-class, so documents carry mirrored provenance. A Verity scope inherits exactly the Drive permissions on each file. This is the one connector with no admin-assigned --visibility: visibility comes from the source ACL, resolved through the principal registry.
1 · Create the credential
In your own Google Cloud project, create a service account, download its JSON key, enable the Google Drive API in that project, and enable domain-wide delegation in your Workspace admin console for the read-only scope https://www.googleapis.com/auth/drive.readonly. (Without DWD, the service account itself must be granted access, e.g. added to shared drives; if the API isn't enabled, the first sync 403s with "API … has not been used in project … or is disabled"; enable it and retry.) No vendor OAuth app, ever.
2 · Configure
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
export VERITY_URL=http://127.0.0.1:7717 # the Drive runner otherwise defaults to :8080
export VERITY_TENANT_ID=$TENANT
export GDRIVE_DELEGATED_SUBJECT=admin@acme.example # the workspace user to impersonate (DWD)
export GDRIVE_ANYONE_MAPS_TO=org:everyone # optional; default: anyone-shared quarantines3 · Run the initial backfill
The first --once bootstraps: with no cursor, poll fetches a start page token via changes.getStartPageToken and returns with no events. History before the token is the reconciliation crawl's job, not the change feed's. The crawl (files.list over trashed = false) is what pulls your existing corpus: per file it fetches metadata, then the ACL via permissions.list, then, only if the ACL resolves, content (Google Docs exported as text/plain; text/* and JSON downloaded directly). Preview the exact request bodies first with --dry-run:
python -m verity_ingest.connectors.gdrive --once --dry-run # prints POST /v1/ingest/documents bodies
python -m verity_ingest.connectors.gdrive --once # deliver for real; cursor -> .verity/gdrive_cursor.json
# gdrive: delivered 91 request(s); cursor -> .verity/gdrive_cursor.jsonEach Drive permission maps to a canonical principal, resolved to an int token through POST /v1/admin/principals (the HttpRegistry), or a local --principal-map JSON for offline runs:
| Drive permission | Maps to |
|---|---|
type=user | user:<email> |
type=group | group:<email> (nested-group closure is the Identity Plane's job) |
type=domain | domain:<domain> |
type=anyone | quarantine unless GDRIVE_ANYONE_MAPS_TO is set |
| unknown / unmappable | quarantine (the whole envelope, fail closed) |
--visibility flag: visibility is the real Drive ACL. If every principal on a file is unmappable (or the ACL is unresolvable, e.g. an unmapped anyone-link), the document quarantines: the connector posts it with no visibility field and the server holds it. ACL-before-content means bytes are never pulled for an item that will quarantine.4 · Verify
Mirrored Drive docs are recall-searchable, pre-filtered by the mirrored ACL. A caller whose resolved principals intersect the file's Drive permissions gets hits; a caller who is not on the file gets zero, because the ACL became an in-index pre-filter (see Permissions → ACL inheritance).
# a scope for a principal on the file → the doc is recalled
curl -s -X POST "$VERITY_URL/v1/recall" -H 'content-type: application/json' \
-d "{\"scope_handle\":\"$IN\",\"text\":\"onboarding runbook\",\"k\":5}" | jq length
# → 3
# a scope for a principal NOT on the file → 0 hits (mirrored ACL excluded it in the index)
curl -s -X POST "$VERITY_URL/v1/recall" -H 'content-type: application/json' \
-d "{\"scope_handle\":\"$OUT\",\"text\":\"onboarding runbook\",\"k\":5}" | jq length
# → 05 · Ongoing sync
Run without --once to loop at --interval seconds (default 300); each cycle drives changes.list from the saved page-token cursor and re-emits changed files (permission drift shows up as re-emitted envelopes; the server diffs). The changes.watch push lane needs a public HTTPS endpoint and 7-day channel renewal and is a documented no-op today. For a supervised schedule, use Temporal below.
Gmail
Same BYOT service account and domain-wide delegation as Google Drive, with a different scope and command. In your Cloud project enable the Gmail API, and grant the SA the read-only scope https://www.googleapis.com/auth/gmail.readonly in your Workspace admin console (DWD), impersonating the mailbox owner (--subject). Each message is keyed by its RFC822 Message-ID, so the same email seen in two connected mailboxes indexes once; the ACL mirrors the participants: From/To/Cc become user:<email> visibility, mirrored provenance; a message whose participants don't resolve quarantines rather than indexing permissively.
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
python -m verity_ingest.connectors.gmail --backfill --subject you@acme.example # --once to poll · --dry-run to previewGoogle Workspace directory sync Identity Plane
The content connectors above emit group:<email> ACLs but deliberately don't expand them; nested-group closure is this connector's job. Same SA + DWD: enable the Admin SDK API and grant the read-only scopes admin.directory.user.readonly + admin.directory.group.readonly. It reconciles users and groups from the Admin SDK and writes direct membership edges into the SpiceDB ReBAC graph, nesting preserved. SpiceDB owns the transitive closure, and the reconcile interval is the group-membership freshness bound. Proven live against a real Google Workspace directory: a doc shared with a group resolves to its nested members via open_scope, non-members provably dark. Run it supervised as a server-owned worker with verity-cli dev --directory.
python -m verity_ingest.connectors.gdirectory --once --subject admin@acme.example --domain acme.exampleMicrosoft Entra ID directory sync Identity Plane
The Microsoft analog of the Google sync, for tenants whose groups live in Entra (Azure AD). BYOT app registration: in Entra, register a single-tenant app, add the application permissions User.Read.All + Group.Read.All + GroupMember.Read.All (read-only), and click Grant admin consent. The client secret lives in a 0600 file whose path is passed by env; the server never reads or logs it. It reconciles users, groups, and direct membership edges (SpiceDB owns the transitive closure), keyed on the immutable objectId, never UPN or mail. Freshness rides Graph delta, folded into a persisted full snapshot so a member removed by deleting the user object (a removal Graph's group-delta stream never reports) is still purged. Guests are excluded by a four-part gate (a B2B guest converted to Member is still excluded), and the guest-excluded tenant token is materialized explicitly. Proven live against a real (scratch) Entra tenant: nested closure, guest exclusion, and the delete-a-user edge purge. This is the identity plane the SharePoint/OneDrive content connector below resolves its group grants through.
export ENTRA_TENANT_ID=<directory-tenant-guid> ENTRA_CLIENT_ID=<app-client-id>
export ENTRA_CLIENT_SECRET_FILE=~/.verity/entra-client-secret # 0600; path only, never logged
python -m verity_ingest.connectors.entra_directory --once # --dry-run to preview · supervised via /v1/admin/planes/entra-directory/startSharePoint / OneDrive
The Microsoft content connector, riding the Entra identity plane above. SharePoint is ACL tier A for items whose permissions Graph returns in full: per-item grants carry mirrored provenance, resolved to Entra principals, so a Verity scope inherits the item's real SharePoint permissions including transitive group membership and sharing links. Proven live end to end on a real (scratch) tenant, including deletion-to-dark: delete a document at the source and it stops resolving on the next cycle, with a retire ledger to show for it. No --visibility: visibility is the source ACL.
1 · Create the credential
Reuse the same Entra app registration as the directory sync, with the per-site Sites.Selected posture: in the Azure portal grant the app access only to the specific sites you name, and provide the client secret in a 0600 file whose path is passed by env (the server never reads or logs it). The completeness canary is the proof that Graph returned a full ACL, not the scope name, so a drive without one fails closed rather than trusting a partial permission list.
2 · Configure
export ENTRA_TENANT_ID=contoso.onmicrosoft.com ENTRA_CLIENT_ID=<app-guid>
export ENTRA_CLIENT_SECRET_FILE=~/.verity/entra-client-secret # 0600; path only, never logged
export SHAREPOINT_SITE_IDS="contoso.sharepoint.com,<site-guid>" # Sites.Selected: the sites to crawl
export SHAREPOINT_TENANT_GUID=<tenant-guid> # unset: the everyone-except-external claim poisons
export SHAREPOINT_CANARIES_FILE=~/.verity/sp-canaries.json # {driveId:{item_id,expected_user_oid}}: G1 completeness
export VERITY_URL=http://127.0.0.1:7717 VERITY_API_KEY=$KEY VERITY_TENANT_ID=$TENANT3 · Run the initial backfill
--backfill runs the full per-drive delta crawl and stamps the reconcile SLA; run it first, then --once (or the poll loop) for steady state. As with Drive, ACL comes before content: a quarantined item never has its bytes fetched. A drive with no configured canary quarantines wholesale, because Graph returns a caller-filtered partial ACL to under-privileged callers with a 200, and an incomplete mirror we cannot vouch for must be held, not guessed.
python -m verity_ingest.connectors.sharepoint --backfill --dry-run # preview POST bodies
python -m verity_ingest.connectors.sharepoint --backfill # first: stamps the reconcile SLA
python -m verity_ingest.connectors.sharepoint --once # then: incremental delta4 · Verify
Same scoped check as Drive: a caller whose resolved Entra principals intersect the item's SharePoint permissions gets hits; a caller who is not on the item gets zero, because the mirrored ACL became an in-index pre-filter. This is the exact chain proven live: a PDF shared with a nested Entra group is recalled by a member and stays dark for a non-member.
5 · Ongoing sync
Run without --once to loop at --interval (default 300s). A detected deletion or a permission narrowing is parked and drained through POST /v1/admin/retire on the next cycle, so the boundary follows the source. Live-org site-group resolution and push subscriptions are the remaining lanes; see Status & honesty.
Slack
Slack is ACL tier B: a channel's membership is the audience. Each message is stored with a single group:slack-channel-<id> visibility, and membership is synced like a mini-directory, so a member who joins can recall the channel's history and someone who leaves loses it, enforced retroactively through the same permission graph. Fixture-tested and live-validated once. No --visibility: it comes from channel membership.
1 · Create the credential
One command, about three minutes. The wizard walks you through creating a Slack app from a manifest (Socket Mode on) and stores the tokens 0600 in ~/.verity/config.toml under [connectors.slack]:
verity-cli connect slackThe app needs the bot scopes channels:history, channels:read, channels:join, groups:history, groups:read, users:read, users:read.email. Invite the bot to the channels you want mirrored (it can self-join public channels; private channels need an invite). This is your consent surface: Verity remembers exactly the channels you add it to.
2 · Configure
The wizard already wrote the tokens, so only the Verity vars remain. To use an env token instead, set SLACK_BOT_TOKEN (xoxb-...), which overrides the config file.
export VERITY_URL=http://127.0.0.1:7717 VERITY_API_KEY=$KEY VERITY_TENANT_ID=$TENANT
python -m verity_ingest.connectors.slack --backfill # then --once to poll · --dry-run to preview3 · Verify
A caller who is a member of the channel recalls its messages; a caller who is not gets zero, because the channel-group token is an in-index pre-filter. Add or remove the bot, or move a member in and out, and the boundary follows on the next sync.
4 · Ongoing sync
Run without --once to loop at --interval (default 300s). Deletions and channel removals are parked and drained through POST /v1/admin/retire. DMs and private channels the bot is not in are out of scope by design; the Socket-Mode push lane is reserved for a later release, so this is the poll lane today.
Zoom
Zoom mirrors cloud-recording transcripts. Zoom exposes no per-recording audience, so this connector is honest about being a reconstruction: visibility is derived from the recording's share setting plus an operator-declared token, carried as admin-assigned, not a mirrored source ACL. It is fixture-verified and not yet validated against a live Zoom account (that needs a Pro+ seat with cloud recording and audio transcripts on). Fail-closed and over-hiding by construction.
1 · Create the credential
In your own Zoom account, create a Server-to-Server OAuth app (no marketplace review). Note the Account ID, Client ID, and Client Secret; the app needs cloud-recording read, past-participant read, and user:read:user:admin. The client secret lives in a 0600 file whose path is passed by env.
2 · Configure
export ZOOM_ACCOUNT_ID=<acct> ZOOM_CLIENT_ID=<client-id>
export ZOOM_CLIENT_SECRET_FILE=~/.verity/zoom-client-secret # 0600; path only, never logged
export ZOOM_USER_IDS="host1@acme.com,host2@acme.com" # whose cloud recordings to mirror
export ZOOM_INTERNAL_MAPS_TO=group:everyone@acme.com # where "share internally" maps; unset: those quarantine
export VERITY_URL=http://127.0.0.1:7717 VERITY_API_KEY=$KEY VERITY_TENANT_ID=$TENANT3 · Run the initial backfill
--backfill walks each host's recordings in month-sized windows (Zoom caps the list API at a one-month range) and stamps the reconcile SLA; then --once for incremental. The transcript VTT is parsed into speaker turns; the recording UUID is the stable document key.
python -m verity_ingest.connectors.zoom --backfill # then --once to poll · --dry-run to previewnone maps to the host alone, internally to ZOOM_INTERNAL_MAPS_TO plus the host (an admin-assigned policy), publicly quarantines, and anything unrecognized quarantines. Attendance-derived participant audiences are an explicit opt-in and labelled approximated.4 · Ongoing sync
Run without --once to loop at --interval (default 300s). A recording that disappears from the source list, or a trashed/deleted recording, is parked and drained through POST /v1/admin/retire. Live-account validation is the open item; see Status & honesty.
Postgres / MySQL
Two on-ramps for a relational database, both landing in the same deterministic keyed L1 path. Pick by whether you need history and ordering.
The pg_net trigger (CDC-lite, convenience)
One copy-paste trigger POSTs row changes to a minted webhook URL: no replication slot, no Kafka, no connector process. Works on Supabase (pg_net preinstalled) or any Postgres with pg_net installed. The full snippet is docs/snippets/pg-net-trigger.md; the shape:
Postgres → Verity in three steps
1Mint the webhook URL. Visibility is bound into the URL and can never be widened by a payload. The token in the URL is the credential, shown once.
verity-cli webhook mint pg:orders --visibility 12Enable pg_net and install the trigger, substituting the minted URL, your table, and its primary key. It emits Verity's native facts shape: one fact per column, keyed (source="pg:orders", entity_id=<pk>, field). INSERTs send every column; UPDATEs send only what changed.
create extension if not exists pg_net;
-- verity_row_to_facts() + AFTER INSERT OR UPDATE trigger, see the snippet file
perform net.http_post(
url := 'https://verity.example.com/wh/<token>',
body := jsonb_build_object('facts', facts),
headers := '{"Content-Type": "application/json"}'::jsonb);3Verify it lands. A changed value supersedes the old row bi-temporally (never UPDATE-in-place); valid_from tracks commit time.
verity-cli query "order 1042"net.http_post is async; gaps possible), sends no DELETEs, and has no backfill: triggers see changes from now on, not history. For existing rows, do a one-time UPDATE … SET id = id to fire the trigger across the table, or use the Debezium lane below. Provenance is admin-assigned (the mint-time visibility).The Debezium envelope (the truth lane)
Already running CDC, or need ordering, snapshots (backfill), and deletes? Point a Debezium Server HTTP sink (or a small Kafka consumer) at the admin ingest endpoint. Debezium's initial snapshot is your backfill; it accepts a single envelope or a JSON array, upserts row facts through the same bi-temporal path, and handles op:"d" by retiring the entity's facts.
curl -s -X POST "$VERITY_URL/v1/ingest/debezium?tenant_id=$TENANT&pk=id" \
-H 'content-type: application/json' \
-H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
-d '{"op":"u","source":{"connector":"postgresql","table":"orders","ts_ms":1752000000000},
"after":{"id":"1042","status":"paid","amount":99.00}}'
# → { "facts_inserted":2, "facts_superseded":0, "facts_unchanged":0, "facts_retired":0 }?pk= query param overrides the primary-key field (default id); the L1 partition is <connector>:<table> (e.g. postgresql:orders). Measured POST-to-queryable: 31 ms on the dev profile.Bulk documents
Loading an existing document corpus (a folder of markdown, a wiki export, PDFs already extracted to text) is the convenience lane: snapshot-grade, no per-object ACLs, so visibility is always an explicit admin decision (admin-assigned).
verity-cli add
add takes a file, a directory (recursive over .txt/.md/.json/.csv/.html, capped at 200 files), an http(s) URL, or stdin. --visibility is required by the parser and has no default: Verity never guesses who may see a memory. It mints a short-lived scope whose principals are exactly your tokens and uploads under it.
verity-cli add ./handbook --visibility 1 # a whole directory, org-wide
verity-cli add onboarding.md --visibility 7,11 --entity account:acme # narrow + entity tagFramework sinks
If your corpus already loads through a framework, six thin adapters turn its readers into Verity sinks: LlamaIndex, LangChain, LangGraph, CrewAI, Google ADK, OpenAI Agents. Every one takes a required visibility_policy with no default (loaders strip source ACLs by construction, so this lane is always policy-based; missing policy quarantines). Point hundreds of community loaders at permission-aware memory:
from verity_llamaindex import VerityVectorStore
store = VerityVectorStore(
verity_url="http://127.0.0.1:7717",
tenant_id=TENANT,
visibility_policy=[1], # REQUIRED, no default; missing → quarantine
admin_token=ADMIN_TOKEN,
)
# every LlamaHub reader now writes into permission-aware Verity memorySee Ingestion → framework sinks for all six adapters. For file uploads without a framework, POST /v1/files is a multipart upload under a scope handle (paragraph-chunked, embedded). Verify with a scoped verity-cli query.
Any internal system
A homegrown app, a cron job, an ops tool: anything that can POST JSON becomes a source in ~60 seconds, no connector code and no OAuth. The visibility policy is bound at mint time into the URL; the blast radius is one URL, instantly revocable.
Internal system → source in two calls
1Mint the URL (admin-gated). The raw token is returned once; only its sha256 persists.
curl -s -X POST "$VERITY_URL/v1/webhooks" -H "authorization: Bearer $VERITY_ADMIN_TOKEN" \
-H 'content-type: application/json' \
-d '{"tenant_id":"'"$TENANT"'","name":"ops-tool","visibility":[11],
"entity_scope":["account:acme"]}'
# → { "webhook_id":"0192...", "url":"/wh/n8sQ...raw-token" }2POST the native payload to the minted URL. No auth header; the path token is the credential. Accepts content/observation (text → a chunk), facts[] (keyed L1 upserts), entities[], and an optional visibility[] that may narrow but never widen the bound set.
curl -s -X POST "$VERITY_URL/wh/n8sQ...raw-token" -H 'content-type: application/json' \
-d '{"content":"Acme signed the pilot agreement for the fall rollout.",
"entities":["account:acme"]}'
# → 200 { "episode_id":"...", "chunks_indexed":1, "facts_written":0 }Provenance is admin-assigned (the visibility was an explicit admin decision at mint). Unknown shapes return 202 {"quarantined":true}: preserved for review, never permissively indexed. Revoke instantly with DELETE /v1/webhooks/{id}. For a reviewable, structured mapping over the same endpoint (with tier-enforced ACL policy), graduate to a source manifest.
Ongoing sync (Temporal)
Once a backfill has landed, promote the connector from ad-hoc --once runs to a supervised schedule. The orchestration plane runs a Temporal worker plus one Schedule per connector; the cursor lives in workflow state instead of a .verity/*_cursor file, and delivery stays at-least-once (the next cursor is returned only after the sink succeeds).
# which connectors sync, and how often, fail closed: EMPTY means nothing syncs
export VERITY_CONNECTORS=hubspot,salesforce,gdrive
export VERITY_SYNC_INTERVAL=300 # default poll seconds
export VERITY_SYNC_INTERVAL_HUBSPOT=60 # per-connector override
# the orchestration runners read each connector's OWN policy env, no default:
export HUBSPOT_VISIBILITY=1,2
export SALESFORCE_VISIBILITY=1,2
# 1) run the worker (scale horizontally by running more on the same task queue)
python -m verity_ingest.orchestration.worker
# 2) apply the schedules (dry-run without --apply; overlap policy SKIP)
python -m verity_ingest.orchestration.schedules --apply
# verity-sync-hubspot: created verity-sync-salesforce: created verity-sync-gdrive: createdverity-sync-<connector>) supervises one forever-looping workflow (connector-sync-<connector>); overlap policy SKIP guarantees never more than one sync chain per connector. A restarted chain resumes with cursor=None: HubSpot/Salesforce replay from epoch into keyed upserts; Drive re-arms the change feed and the reconciliation crawl covers the gap. Temporal needs TEMPORAL_ADDRESS/TEMPORAL_NAMESPACE (defaults localhost:7233 / default).System comparison
The one-glance table: credential, the initial-backfill command, freshness, and the ACL contract for each system.
| System | Credential | Initial backfill | Freshness | ACL tier | visibility_policy |
|---|---|---|---|---|---|
| HubSpot | Service Key (HUBSPOT_SERVICE_KEY) |
python -m verity_ingest.connectors.hubspot --once --visibility … |
poll + UI webhooks | C · admin-assigned | required, no default |
| Salesforce | Connected App, client-credentials (SF_MY_DOMAIN/SF_CLIENT_ID/SF_CLIENT_SECRET) |
python -m verity_ingest.connectors.salesforce --once --visibility … |
poll (Pub/Sub later) | A · admin-assigned + approximated shares | required, no default |
| Google Drive | service account + DWD (GOOGLE_APPLICATION_CREDENTIALS) |
python -m verity_ingest.connectors.gdrive --once |
poll (watch later) | A · mirrored | n/a · ACL is the source |
| SharePoint / OneDrive | Entra app, Sites.Selected (ENTRA_CLIENT_ID/ENTRA_CLIENT_SECRET_FILE) |
python -m verity_ingest.connectors.sharepoint --backfill |
poll (subscriptions later) | A · mirrored (site groups quarantine) | n/a · ACL is the source |
| Slack | verity-cli connect slack (app-from-manifest) |
python -m verity_ingest.connectors.slack --backfill |
poll (Socket Mode later) | B · mirrored (channel membership) | n/a · membership is the source |
| Zoom | S2S OAuth app (ZOOM_ACCOUNT_ID/ZOOM_CLIENT_ID/ZOOM_CLIENT_SECRET_FILE) |
python -m verity_ingest.connectors.zoom --backfill |
poll | reconstructed · admin-assigned | operator token (ZOOM_INTERNAL_MAPS_TO) |
| Postgres/MySQL (Debezium) | admin token + CDC pipeline | Debezium snapshot → POST /v1/ingest/debezium |
~31 ms to queryable | mirrored (manifest) / admin-assigned | manifest tier or static |
| Postgres (pg_net) | minted webhook URL | one-time UPDATE to fire the trigger (no native backfill) |
push (fire-and-forget) | admin-assigned | mint-time visibility |
| Bulk documents | scope handle / admin token | verity-cli add <file|dir|url> --visibility … |
manual / one-shot | admin-assigned | required, no default |
| Any internal system | minted webhook URL | replay history as POST /wh/{token} |
push (receipt-time) | admin-assigned | mint-time visibility |
Status & honesty
Verity is pre-1.0 and every claim on this site is measured, never quoted. Where a connector is fixture-verified vs live-validated:
- Fixture-verified. The HubSpot, Salesforce, and Google Drive connectors' deterministic mappings (search/query pages → facts,
permissions.list→ ACL envelope, webhook decode) run against recorded fixtures in CI. The request bodies for/v1/ingest/documentsand/v1/admin/principalsare pinned by fixture tests; full live-server integration lands as those endpoints finish. - Awaiting a design partner. Salesforce live-org validation, and the CDC Pub/Sub (gRPC) push lane, is not yet exercised against a real org. The share-derived principals are approximated additive metadata, not enforced visibility, until the identity crosswalk and the implicit-sharing gap are closed. We say so rather than imply live coverage.
- Forthcoming. A backfill-progress dashboard is not yet built; today the connectors emit best-effort heartbeats to
POST /v1/admin/connector-status(source, items synced, newest event time, cursor), readable atGET /v1/admin/connector-status. A heartbeat failure never fails a sync.