Document RAG for assistants
What this module does
It lets assistants answer using the tenant's own knowledge: the user links files already uploaded to the bucket to a specific assistant, the system indexes them, and a query skill retrieves only the relevant fragments in the chat instead of putting the whole document into every turn.
Delivery history: the first one covered the entitlement and retention base; the second added the HTTP endpoints to link and unlink files to a specific assistant's corpus; the third added the real indexing pipeline (text extraction, chunking, and the pending → ready/error transitions); this one adds the chat query skill (assignable like any other skill), citations (the file name) and the RAG switch in the public widget, per assistant. The API/SDK to manage those switches arrives in a later delivery (see "Next deliveries").
An entitlement separate from assistants and storage
Document RAG requires its own plan feature: rag_documental_enabled. Neither having assistants (ai_assistants) nor having object storage (object_storage_enabled) enables RAG on its own — it must be purchased separately (visible in GET /billing/catalog/system-options).
| Situation | Result |
|---|---|
Without rag_documental_enabled | Linking or managing the corpus and executing the query skill are rejected (authenticated chat and widget) |
With rag_documental_enabled | Management and querying allowed |
What happens when the entitlement is lost
The corpus is not deleted instantly. It is marked unusable and preserved:
- Entitlement loss → the corpus becomes
unusable(the exact moment is recorded). While it is like that it is neither queryable nor manageable, but the indexed data is still there. - Recovery before it is deleted → the corpus returns to
activeautomatically on the first access with the plan entitled again — there is no need to re-upload the bucket files. - 90 days without recovering the entitlement → an internal maintenance job (a daily purge, the same pattern as the click-event purge) deletes only the indexed data for that tenant. The original bucket files are never touched here. The corpus becomes
purged; recovering the entitlement after this restores nothing for free — it would need re-indexing.
No numeric cap of its own
There is no RAG-specific "number of files" or "indexed size" quota in this version — only the bucket's existing limits apply.
Corpus per assistant: linking and unlinking files
The corpus is per assistant, never a knowledge base shared at tenant level (see the spec's "Out of scope"). Every linked row points at a file that already exists in the bucket, with the same scope as the assistant: a tenant assistant can only link files from tenant space of its own company; a platform assistant only from platform space. There is no way to mix scopes or to link another company's file (isolation).
In NimoApps, corpus links and fragments are also limited to the authorized environment. Object change events include that environment, so a file with the same path in preview and production only updates the affected environment's index. The context comes from server authorization, not from a selector header sent by the client.
Tenant (OWNER/ADMIN; MEMBER/VIEWER → 403 through RolesGuard):
| Method | Path | Description |
|---|---|---|
GET | /ai/assistants/:id/corpus | Lists the assistant's corpus links |
POST | /ai/assistants/:id/corpus | Links a file from the company bucket ({ filePath }); idempotent if already linked |
DELETE | /ai/assistants/:id/corpus?filePath= | Unlinks the file; idempotent — 204 even if the link does not exist |
Platform (super admin only, resolved in the service — there is no company role to apply):
| Method | Path | Description |
|---|---|---|
GET | /ai/platform/assistants/:id/corpus | Lists the platform assistant's corpus links |
POST | /ai/platform/assistants/:id/corpus | Links a file from the platform bucket |
DELETE | /ai/platform/assistants/:id/corpus?filePath= | Unlinks the file — 204 |
Every link is created in the pending state (status column of rag_assistant_corpus_links) and indexing starts in the background — the POST response still returns pending immediately; the final state (ready/error) is read with the next GET or by forcing a manual reindex.
Entitlement rejection: linking or unlinking on a tenant assistant requires a live rag_documental_enabled; without it the API returns 422 RAG_NOT_ENTITLED before touching the bucket or the links table. Platform routes do not depend on any tenant's entitlement.
Indexing pipeline
RagCorpusIndexingService does all the extraction and chunking work. It requires no external vector engine (out of scope for the spec): fragments are stored as plain text in rag_corpus_chunks, ready for the query skill to do a simple keyword/substring search — enough for "relevant fragments".
Supported formats, by the filePath extension:
| Extension | Extractor |
|---|---|
.txt, .md, .markdown | Direct read as UTF-8 |
.pdf | pdf-parse (text, no OCR) |
.docx | mammoth (extractRawText) |
Any other extension (or a missing one) is rejected with an actionable reason in errorReason — it never fails silently.
Link lifecycle:
- On linking — the row is created as
pending;indexLink()starts in the background (fire-and-forget, it does not block the HTTP response). - Extraction + chunking — it reads the object from the bucket, computes a sha256 hash of the content, extracts text by extension, chunks it (~1000 characters, with overlap) and replaces the link's
rag_corpus_chunks. On success →ready,errorReason: null,indexedAtwith the timestamp. - Extraction failures — unsupported type, empty file, corrupt file or no text layer →
errorwith a readableerrorReason, visible to OWNER/ADMIN or super admin through the listGET. It does not block the rest of the corpus (the spec's edge case). - Content change in the bucket —
ObjectStorageServicepublishes a domain event (object-storage.object-written) after every objectPUT;RagCorpusReindexListenerlistens for it and, for each link pointing at that exact file (in any assistant), callsreindexIfChanged()— it only does real work when the content's sha256 hash changed since the lastready. - Deletion or inaccessible object — when a file or folder is deleted (
DELETE /object-storage/:space/objects),ObjectStorageServicepublishesobject-storage.object-deleted; the same listener invalidates (error, "file no longer exists or is not accessible") every link that pointed at that file or at any file under the deleted folder.indexLink()also invalidates lazily if the object no longer exists when it runs. - Move/rename —
POST /object-storage/:space/objects/movepublishes both events after a successful move:object-storage.object-deletedfor the source path (invalidating any link pointing there, just like a real deletion) andobject-storage.object-writtenfor the destination path (re-indexing any link already pointing at that new path). A move does not create or relink the link to the newfilePathautomatically — an explicitPOST /corpusis still needed to link the file at its new location.
Manual reindex — useful to force a synchronous reindex (for example after fixing a file left in error), without waiting for the automatic asynchronous cycle:
| Method | Path | Description |
|---|---|---|
POST | /ai/assistants/:id/corpus/reindex | Synchronously reindexes an already linked file (tenant, { filePath }) — returns the link with its final state |
POST | /ai/platform/assistants/:id/corpus/reindex | The same, for platform assistants (super admin) |
404 RAG_CORPUS_LINK_NOT_FOUND when the given file was never linked to that assistant.
RAG query skill in the chat
The "Query RAG documental corpus" skill (platform.rag_query, read classification, shareable) lives in the platform catalogue like any other skill: an OWNER/ADMIN assigns it to their tenant assistant from PUT /ai/assistants/:id/skills (or the super admin to a platform assistant) with exactly the same flow as any shareable skill — there is no parallel API to "enable RAG", it is one more skill in the allowlist.
When the model invokes it in a chat (authenticated or widget) while it is in the assistant's allowlist:
- It searches only in that specific assistant's corpus, and only among links in the
readystate — never another assistant's or another tenant's. - If there is no
readylink, it answers that no knowledge is available yet — it never invents content. - The search is a keyword/substring match over the already indexed text fragments (no vector engine, same as
RagCorpusIndexingService— see the spec's "Out of scope"). - If the model asks for the skill while it is not in the assistant's allowlist, it does not run — the same rule and the same rejection path as any other skill (
rejectedSkillsin the chat response). - Every generation using retrieved fragments goes through governance (quotas, rate limit, metering) exactly like any other chat turn — there is no bypass for using RAG.
- Without a live
rag_documental_enabledentitlement, executing the skill is rejected with422 RAG_NOT_ENTITLED, just like managing the corpus.
Source citations
Per assistant, you can enable answers that used RAG fragments to cite at least the file name of each source. When enabled and the skill returned fragments, the assistant's final answer includes a deterministic Sources: <file1>, <file2> line (it does not depend on the model "remembering" to cite). When disabled (the default), no citation is required.
RAG in the public widget
The public widget already offers only skills marked publicSafe, read-only (read) and with a serviceKey in the closed allowlist (billing.plans.list, rag.query) in the assistant's assignment. RAG adds an additional switch, specific to RAG and per assistant: even when the skill is in the authenticated allowlist, marked publicSafe and is rag.query, the anonymous widget does not execute it unless that switch is also on (it is off by default). When it is on, publicSafe + read + the general rule still apply — turning the RAG switch on does not make it public by itself. Widget AI usage (including the RAG query) is always billed to the assistant's owning tenant, like any other widget use.
Per-assistant RAG config — API
The widget and citation switches (widgetEnabled, citationsEnabled) are read and updated over HTTP with the same tenant/platform pattern as the rest of the corpus. Tenant writes are gated by the same management entitlement (assertManagementEntitled) as linking, unlinking and reindexing the corpus; the platform scope does not check a tenant entitlement.
Tenant (OWNER/ADMIN):
| Method | Route | Description |
|---|---|---|
GET | /ai/assistants/:id/rag-config | Current config of the tenant assistant; both switches false by default when there is no row |
PATCH | /ai/assistants/:id/rag-config | Updates widgetEnabled and/or citationsEnabled (partial patch, idempotent); 422 RAG_NOT_ENTITLED without the document entitlement |
Platform (super admin):
| Method | Route | Description |
|---|---|---|
GET | /ai/platform/assistants/:id/rag-config | Current config of the platform assistant |
PATCH | /ai/platform/assistants/:id/rag-config | Updates widgetEnabled and/or citationsEnabled (partial patch, idempotent) |
Both routes return 404 RAG_ASSISTANT_NOT_FOUND for an out-of-scope id (another tenant, or a crossed tenant/platform origin), like the rest of the corpus.
Next deliveries
- SDK parity.