> ## Documentation Index
> Fetch the complete documentation index at: https://docs.knowledgestack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingestion Pipeline

> How Knowledge Stack turns an uploaded file into searchable, embedded, citation-ready chunks.

When you `POST /v1/documents/ingest`, Knowledge Stack starts a durable Temporal workflow that prepares, converts, chunks, enriches, and embeds the file. The endpoint returns immediately with a `workflow_id` you can poll. Files become searchable as soon as the workflow completes.

<Info>
  Looking for the request/response schema? See [`/v1/documents/ingest`](/api-reference#tag/documents/POST/v1/documents/ingest) in the API Reference.
</Info>

## Ingest a document

<CodeGroup>
  ```bash kscli theme={null}
  kscli documents ingest \
    --file ./report.pdf \
    --path-part-id <folder-path-part-id> \
    --name "Q4 Report"
  ```

  ```python Python theme={null}
  with open("report.pdf", "rb") as f:
      doc = ks.documents.ingest(
          file=f,
          parent_path_part_id="<folder-path-part-id>",
          name="Q4 Report",
      )
  print(doc.workflow_id)
  ```

  ```typescript TypeScript theme={null}
  const doc = await ks.documents.ingest({
    file: fs.createReadStream("report.pdf"),
    parentPathPartId: "<folder-path-part-id>",
    name: "Q4 Report",
  });
  console.log(doc.workflowId);
  ```

  ```bash curl theme={null}
  curl -X POST $KS_BASE_URL/v1/documents/ingest \
    -H "Authorization: Bearer $KS_API_KEY" \
    -F "file=@report.pdf" \
    -F "parent_path_part_id=<folder-path-part-id>" \
    -F "name=Q4 Report"
  ```
</CodeGroup>

Response (truncated):

```json theme={null}
{
  "document_id": "01J...",
  "document_version_id": "01J...",
  "workflow_id": "ingest-01J...",
  "status": "QUEUED"
}
```

## Pipeline stages

| Step                | What happens                                                                                                                | Deep dive                                                                                                |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| **1. Preparation**  | Validate format/size, persist source file to S3, create `Document` + `DocumentVersion` rows, dewatermark PDFs.              | [PDF watermark](/ingestion/pdf-watermark), [S3 storage](/ingestion/s3-storage)                           |
| **2. Conversion**   | Route to a converter based on MIME type (Docling for PDFs, Excel pipeline for XLSX, etc.) and extract text + visual assets. | [Routing](/ingestion/routing), [Docling PDF](/ingestion/docling-pdf), [Excel](/ingestion/excel-pipeline) |
| **3. Chunking**     | Walk the document hierarchy, emit `Section` and `Chunk` rows with bounding boxes, page numbers, and roles.                  | [Chunk handling](/ingestion/chunk-handling)                                                              |
| **3.5. Enrichment** | Generate captions for images and structured summaries for tables via an LLM.                                                | [Chunk handling](/ingestion/chunk-handling)                                                              |
| **4. Embedding**    | Embed each chunk and upsert vectors into Qdrant; mark the workflow `COMPLETED`.                                             | [Qdrant](/infrastructure/qdrant), [Temporal workflow](/ingestion/temporal-workflow)                      |

## Supported formats

PDF, DOCX, PPTX, XLSX, Markdown, plaintext. Hard limits: **100 MB** per file, **150 pages** per document.

## Watch a workflow

Each ingest returns a `workflow_id`. Poll it via `/v1/workflows`:

| Method | Path                               | Description                                    |
| ------ | ---------------------------------- | ---------------------------------------------- |
| GET    | `/v1/workflows`                    | List recent workflows for your tenant          |
| GET    | `/v1/workflows/{id}`               | Live status, per-activity timing, error traces |
| POST   | `/v1/workflows/{id}?action=cancel` | Cancel (OWNER / ADMIN only)                    |
| POST   | `/v1/workflows/{id}?action=rerun`  | Rerun from scratch (OWNER / ADMIN only)        |

<CodeGroup>
  ```bash kscli theme={null}
  kscli workflows list --limit 5
  kscli workflows describe <workflow-id>
  ```

  ```python Python theme={null}
  wf = ks.workflows.describe(workflow_id="ingest-01J...")
  print(wf.status, wf.current_activity)
  ```
</CodeGroup>

## Reliability model

* **Durable** — each step runs as a Temporal activity with timeout + retry policy. The workflow survives worker crashes and infrastructure restarts.
* **Idempotent** — chunking clears prior content before re-creating; storage uploads overwrite at the same paths; identical content is deduplicated per-tenant.
* **Observable** — every step emits structured logs, metrics, and a span tree visible in the Temporal UI.

### Retry classification

| Class             | Examples                      | Behavior                                            |
| ----------------- | ----------------------------- | --------------------------------------------------- |
| **Retryable**     | 429, 502, 503, network errors | Up to 3 retries with exponential backoff (5s → 60s) |
| **Non-retryable** | 400, 401, 403, 404, 500       | Fail immediately; surface error on the workflow     |

### Per-step timeouts

| Step               | Timeout    |
| ------------------ | ---------- |
| Preparation        | 60s        |
| Conversion         | up to 2h   |
| Chunking           | 10m        |
| Enrichment         | 2m / chunk |
| Embedding          | 2m / batch |
| **Workflow total** | **30m**    |

## Specialized task queues

Each stage is dispatched to its own queue so heavy work doesn't block lighter work:

| Queue                 | Purpose                               |
| --------------------- | ------------------------------------- |
| `document-ingestion`  | Preparation, chunking, orchestration  |
| `document-conversion` | DOCX / PPTX / XLSX / MD conversion    |
| `pdf-conversion`      | VLM-backed PDF conversion (GPU-heavy) |
| `enrichment`          | Image captions + table summaries      |
| `embedding`           | Embedding generation + Qdrant upsert  |
| `vector-ops`          | Reindex, re-embed, vector maintenance |

## Re-embedding a folder

Switching embedding models? Trigger a folder-wide re-embed:

```bash theme={null}
curl -X POST $KS_BASE_URL/v1/folders/<folder_id>?action=reembed \
  -H "Authorization: Bearer $KS_API_KEY"
```

Recursive — applies to every document under the folder's subtree.

## Recipes

<CardGroup cols={2}>
  <Card title="Bulk ingest from S3" icon="cloud-arrow-up" href="https://github.com/knowledgestack/ks-cookbook/tree/main/recipes/bulk-ingest-s3">
    Stream a whole bucket through `/documents/ingest` with backpressure.
  </Card>

  <Card title="CI ingest pipeline" icon="github" href="https://github.com/knowledgestack/ks-cookbook/tree/main/recipes/ci-ingest">
    Re-ingest changed docs on every PR using `kscli`.
  </Card>
</CardGroup>