feat: initial RAG sync with .4nkaiignore (extension 0.3, server 0.2)

**Motivations:**
- Seed AnythingLLM workspace from cloned repo using gitignore-style filters

**Root causes:**
- N/A

**Correctifs:**
- N/A

**Evolutions:**
- Template 4nkaiignore.default; server copies after clone; extension uploads via POST /api/v1/document/upload
- New commands /workspace-sync; settings initialSync*; dependency ignore

**Pages affectées:**
- extensions/anythingllm-workspaces/*
- services/repos-devtools-server/*
- docs/features/initial-rag-sync-4nkaiignore.md

docs/features/initial-rag-sync-4nkaiignore.md

@@ -0,0 +1,35 @@
+# Synchronisation RAG initiale et `.4nkaiignore`
+
+**Author:** 4NK
+
+## Objectif
+
+À la **création du clone** (ou chargement sync), disposer d’un **workspace AnythingLLM** aligné sur le dépôt et importer une **première vague de fichiers** utiles au RAG, en excluant le bruit via un fichier **`.4nkaiignore`** (syntaxe **identique à `.gitignore`**).
+
+## Comportement
+
+1. **Serveur `repos-devtools-server`** : après `git clone` réussi, copie **`templates/4nkaiignore.default`** vers **`<repo>/.4nkaiignore`** si absent.
+2. **Extension 0.3.0** : après `/repos-clone-sync`, `/repos-load-sync`, ou sur **`/workspace-sync <nom>`**, si l’option **`anythingllm.initialSyncAfterClone`** n’est pas à `false` :
+   - assure **`.4nkaiignore`** depuis le template bundlé si toujours absent ;
+   - parcourt le dépôt, applique règles de base + `.4nkaiignore` ;
+   - envoie chaque fichier accepté via **`POST /api/v1/document/upload`** avec **`addToWorkspaces`** = slug du workspace.
+
+## Fichier type
+
+- **`extensions/anythingllm-workspaces/templates/4nkaiignore.default`**
+- **`services/repos-devtools-server/templates/4nkaiignore.default`** (même contenu ; à maintenir en parité).
+
+L’utilisateur renomme / copie en **`.4nkaiignore`** à la racine du projet et adapte les règles.
+
+## Prérequis AnythingLLM
+
+Le **collecteur / processeur de documents** doit être joignable par l’instance AnythingLLM ; sinon l’upload API échoue avec le message renvoyé par le serveur.
+
+## Modalités d’analyse
+
+- Compter les champs **`uploaded`**, **`skipped`**, **`errors`**, **`capped`**, **`dotfileCreated`** dans la section **Initial RAG sync** du panneau Dev tools.
+- Vérifier les logs AnythingLLM / collector en cas d’échec systématique des uploads.
+
+## Modalités de déploiement
+
+- Rebuild et redémarrage de **repos-devtools-server** ; repackaging / réinstallation de l’extension **0.3.0+**.

extensions/anythingllm-workspaces/README.md

@@ -1,102 +1,57 @@
 # AnythingLLM Workspaces (VS Code / Cursor)
 
-Extension that talks to the **AnythingLLM developer HTTP API** (list/create workspaces, open the UI in a browser). Optionally uses a **local repos HTTP service** (`repos-devtools-server`) to clone or open Git folders under a configured root, from a **Dev tools** webview panel.
+AnythingLLM **developer API** (workspaces, documents), optional **repos-devtools-server**, **Dev tools** webview, and **initial RAG upload** after clone/load using **`.4nkaiignore`** (same syntax as `.gitignore`).
-
-## Features
-
-| Area | What it does |
-|------|----------------|
-| AnythingLLM | List workspaces, open one in the browser, open the web UI. |
-| Workspace ensure | If no workspace matches a repo folder name, create it via `POST /api/v1/workspace/new`. |
-| Local repos API | Clone (`branch` **test** by default), list git folders, resolve paths — requires `repos-devtools-server`. |
 
 ## Requirements
 
-1. **AnythingLLM** reachable at a public base URL (example: `https://ia.enso.4nkweb.com/anythingllm`).
+- AnythingLLM with **API key** (**Settings → API Keys**). Do **not** use the nginx Bearer for `/ollama/` here.
-2. An **API key** from AnythingLLM: **Settings → API Keys**.
+- **`repos-devtools-server`** on the host that owns clones (default `http://127.0.0.1:37140`).
-
+- For **document upload**, AnythingLLM’s **document processor (collector)** must be online; otherwise `POST /api/v1/document/upload` returns an error.
-**Important:** Do **not** put the **nginx Bearer secret** used for `/ollama/` here (see `deploy/nginx/README-ia-enso.md`). AnythingLLM only accepts keys stored in its own app; a wrong value returns `403` and `{"error":"No valid api key found."}`.
-
-3. For **clone / repos-list / repos-load** commands: run **`repos-devtools-server`** on the machine that owns the clone directory (see `../../services/repos-devtools-server/README.md`). Default URL from the editor: `http://127.0.0.1:37140`. If Cursor connects over **SSH** to that host, `127.0.0.1` is the remote machine — no port forward needed. If the editor runs on another PC, set `anythingllm.reposApiBaseUrl` to a tunnel or the server’s reachable address.
-
-## Installation
-
-- **From source:** open `extensions/anythingllm-workspaces` in VS Code / Cursor, **Run → Start Debugging** (Extension Development Host).
-- **From VSIX:**  
-  `npm install && npm run compile && npx @vscode/vsce package`  
-  then **Extensions → … → Install from VSIX…** and pick `anythingllm-workspaces-*.vsix`.
-
-After install or upgrade, run **Developer: Reload Window** if commands are missing.
 
 ## Configuration
 
-Open **Settings**, search for **AnythingLLM**, or edit **User** `settings.json`:
-
 | Key | Description |
 |-----|-------------|
-| `anythingllm.baseUrl` | AnythingLLM public base URL, **no** trailing slash. |
+| `anythingllm.baseUrl` | AnythingLLM public URL (no trailing `/`). |
-| `anythingllm.apiKey` | API key from AnythingLLM (a leading `Bearer ` prefix is stripped if present). |
+| `anythingllm.apiKey` | API key. **User** settings. |
-| `anythingllm.reposApiBaseUrl` | `repos-devtools-server` base URL, no trailing slash (default `http://127.0.0.1:37140`). |
+| `anythingllm.reposApiBaseUrl` | repos-devtools-server URL. |
-| `anythingllm.reposApiToken` | Same secret as `REPOS_DEVTOOLS_TOKEN` on the server. |
+| `anythingllm.reposApiToken` | Same as `REPOS_DEVTOOLS_TOKEN`. |
+| `anythingllm.initialSyncAfterClone` | Default **on**: after `/repos-clone-sync`, `/repos-load-sync`, and `/workspace-sync`, upload filtered files. Set to `false` to disable. |
+| `anythingllm.initialSyncMaxFiles` | Max files per run (default `400`). |
+| `anythingllm.initialSyncMaxFileBytes` | Max bytes per file (default `5242880`). |
 
-Use **User** settings so secrets are not committed with a workspace.
+## Commands (palette)
 
-## Commands (Command Palette)
+- **AnythingLLM: List workspaces** — `GET /api/v1/workspaces`, open one in the browser.
+- **AnythingLLM: Open web UI**
+- **AnythingLLM: Dev tools panel** — webview for scripted commands.
 
-Open the palette: **Ctrl+Shift+P** (Windows / Linux) or **Cmd+Shift+P** (macOS).
+## `.4nkaiignore`
 
-| Title in palette | Command ID | Action |
+- **Template (reference):** `templates/4nkaiignore.default` in this extension (and the same file under `services/repos-devtools-server/templates/` for the clone server).
-|------------------|------------|--------|
+- **At repo root:** the file must be named **`.4nkaiignore`**.
-| **AnythingLLM: List workspaces** | `anythingllm.listWorkspaces` | Calls `GET /api/v1/workspaces`, pick a workspace, open it in the browser. |
+- **After `git clone` via the server:** if `.4nkaiignore` is missing, the server copies the template into the new repo (`fourNkAiIgnoreTemplateWrote` in the JSON response).
-| **AnythingLLM: Open web UI** | `anythingllm.openWebUi` | Opens `anythingllm.baseUrl` in the browser. |
+- **Before upload:** the extension creates `.4nkaiignore` from the bundled template only if it is still missing (e.g. repo cloned outside the server).
-| **AnythingLLM: Dev tools panel** | `anythingllm.openDevToolsPanel` | Opens the Dev tools webview (see below). |
 
-## Dev tools panel
+Filtering uses the **`ignore`** package (gitignore semantics). The extension always applies baseline rules (e.g. `.git/`, `node_modules/`) in addition to `.4nkaiignore`.
 
-### How to open it
+## Dev tools — command lines
-
-1. **Ctrl+Shift+P** / **Cmd+Shift+P**
-2. Type **AnythingLLM: Dev tools panel** (or `dev tools`, `anythingllm`).
-3. **Enter**
-
-A side editor tab opens with:
-
-- A **Commands** text area (one command per line)
-- **Run** — execute all non-empty lines in order
-- **Clear output**
-- **Response** — JSON or text from the server / API, or `ERROR: …`
-
-Settings are read **when you click Run**, so you can change `baseUrl` or tokens without reopening the panel.
-
-### Command lines
 
 | Line | Behaviour |
 |------|-----------|
-| `/repos-clone <git-url>` | `POST /repos-clone` — clone into `REPOS_DEVTOOLS_ROOT`, branch **`test`**. |
+| `/repos-clone-sync <url>` | Clone (branch `test`) → ensure workspace → **initial RAG upload** (if enabled) → open folder → browser. |
-| `/repos-clone-sync <url>` | Same as clone, then ensure an AnythingLLM workspace with the **same name as the repo folder**, **Open folder** in the editor, open that workspace in the browser. |
+| `/repos-load-sync <name>` | Open folder → ensure workspace → **initial RAG upload** → browser. |
-| `repos-list` or `/repos-list` | `GET /repos-list` — git repositories under the server root. |
+| `/workspace-sync <name>` | Resolve repo under `REPOS_DEVTOOLS_ROOT` → ensure workspace → **initial RAG upload** (no folder open). |
-| `/repos-load <folder-name>` | `POST /repos-load` — verify folder + **Open folder**. |
+| `/workspace-load <name>` | Ensure workspace → browser only (no file upload). |
-| `/repos-load-sync <name>` | Same as load + ensure workspace + browser. |
+| Other lines | See `help` in the panel. |
-| `/workspace-load <name>` | List workspaces; if none matches by **name** or **slug**, create via API; then open browser. |
-| `help` or `/help` | Print built-in help in **Response**. |
 
-**Workspace name:** Matching uses **exact** equality on AnythingLLM `name` or `slug` and the repo folder name you pass. Creation body: `{ "name": "<name>" }`.
+Upload uses **`POST /api/v1/document/upload`** with multipart field **`file`** and **`addToWorkspaces`** set to the workspace **slug**. Relative paths are flattened to a safe single-segment filename (`dir__file.ts`) to reduce name collisions.
 
-**API key vs browser user:** The extension only uses the **developer API key**. Multi-user behaviour is defined by AnythingLLM for that key.
+**JSON field `workspaceCreatedByApi`:** `true` only if this run called `POST /api/v1/workspace/new`; `false` if the workspace already existed.
 
 ## Ollama
 
-This extension does **not** call Ollama. For OpenAI-compatible URLs such as `https://ia.enso.4nkweb.com/ollama/v1`, configure the editor’s model provider and use the nginx Bearer as documented in `deploy/nginx/README-ia-enso.md`.
+Not used by this extension. Configure Cursor’s model URL for `/ollama/v1` separately.
-
-## Troubleshooting
-
-| Symptom | Check |
-|---------|--------|
-| `403` / `No valid api key found` | Use an AnythingLLM **Settings → API Keys** value, not the Ollama nginx Bearer. |
-| `401` on clone/list/load | `anythingllm.reposApiToken` must equal `REPOS_DEVTOOLS_TOKEN` on `repos-devtools-server`. |
-| `ECONNREFUSED` / fetch failed | Server running? Correct `anythingllm.reposApiBaseUrl`? |
-| Command palette has no AnythingLLM entries | Extension enabled? **Developer: Reload Window**. |
-| Clone fails | Remote must expose branch **`test`** (or change branch via the HTTP API body, not the one-line panel command). |
 
 ## Build
 
@@ -106,12 +61,10 @@ npm install
 npm run compile
 ```
 
-Package: `npx @vscode/vsce package --allow-missing-repository` (Node 20+ recommended for current `vsce`).
+## References
 
-## API reference
+- AnythingLLM document API: `POST /v1/document/upload` under `/api` (Mintplex-Labs anything-llm `server/endpoints/api/document/index.js`).
-
+- Local server: `services/repos-devtools-server/README.md`.
-- AnythingLLM (upstream): Mintplex-Labs **anything-llm** — `server/endpoints/api/workspace/index.js` (routes under `/api`, e.g. `GET /v1/workspaces`, `POST /v1/workspace/new`).
-- Local repos: `services/repos-devtools-server/README.md`.
 
 ## License
 

extensions/anythingllm-workspaces/package-lock.json

@@ -1,13 +1,16 @@
 {
   "name": "anythingllm-workspaces",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "anythingllm-workspaces",
-      "version": "0.1.0",
+      "version": "0.3.0",
       "license": "MIT",
+      "dependencies": {
+        "ignore": "^5.3.2"
+      },
       "devDependencies": {
         "@types/node": "^20.11.0",
         "@types/vscode": "^1.85.0",
@@ -32,6 +35,15 @@
       "integrity": "sha512-AGuxUEpU4F4mfuQjxPPaQVyuOMhs+VT/xRok1jiHVBubHK7lBRvCuOMZG0LKUwxncrPorJ5qq/uil3IdZBd5lA==",
       "dev": true
     },
+    "node_modules/ignore": {
+      "version": "5.3.2",
+      "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz",
+      "integrity": "sha512-hsBTNUqQTDwkWtcdYI2i06Y/nUBEsNEDJKjWdigLvegy8kDuJAS8uRlpkkcQpyEXL0Z/pjDy5HBmMjRCJ2gq+g==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 4"
+      }
+    },
     "node_modules/typescript": {
       "version": "5.9.3",
       "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",

extensions/anythingllm-workspaces/package.json

@@ -1,8 +1,8 @@
 {
   "name": "anythingllm-workspaces",
   "displayName": "AnythingLLM Workspaces (ia.enso)",
-  "description": "List AnythingLLM workspaces via your proxied instance (e.g. ia.enso.4nkweb.com/anythingllm).",
+  "description": "AnythingLLM API, repos devtools, initial RAG sync via .4nkaiignore.",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "publisher": "4nk",
   "license": "MIT",
   "engines": {
@@ -22,22 +22,41 @@
         "anythingllm.baseUrl": {
           "type": "string",
           "default": "https://ia.enso.4nkweb.com/anythingllm",
-          "markdownDescription": "Public base URL of AnythingLLM (nginx path `/anythingllm/`, no trailing slash required)."
+          "markdownDescription": "Public base URL of AnythingLLM (no trailing slash)."
         },
         "anythingllm.apiKey": {
           "type": "string",
           "default": "",
-          "markdownDescription": "AnythingLLM API key (UI: **Settings → API Keys**). Prefer **User** settings to avoid committing secrets."
+          "markdownDescription": "AnythingLLM API key (**Settings → API Keys**). **User** settings."
         },
         "anythingllm.reposApiBaseUrl": {
           "type": "string",
           "default": "http://127.0.0.1:37140",
-          "markdownDescription": "Base URL of **repos-devtools-server** (no trailing slash). Must match the machine where `/home/ncantu/code` (or `REPOS_DEVTOOLS_ROOT`) lives."
+          "markdownDescription": "repos-devtools-server base URL (no trailing slash)."
         },
         "anythingllm.reposApiToken": {
           "type": "string",
           "default": "",
-          "markdownDescription": "Bearer token shared with `REPOS_DEVTOOLS_TOKEN` on the repos-devtools-server. **User** settings only."
+          "markdownDescription": "Same as `REPOS_DEVTOOLS_TOKEN` on the server."
+        },
+        "anythingllm.initialSyncAfterClone": {
+          "type": "boolean",
+          "default": true,
+          "markdownDescription": "After `/repos-clone-sync` or `/repos-load-sync`, upload repo files to the workspace (filtered by `.4nkaiignore`). Requires AnythingLLM document processor (collector) online."
+        },
+        "anythingllm.initialSyncMaxFiles": {
+          "type": "number",
+          "default": 400,
+          "minimum": 1,
+          "maximum": 10000,
+          "markdownDescription": "Max files to upload per initial sync."
+        },
+        "anythingllm.initialSyncMaxFileBytes": {
+          "type": "number",
+          "default": 5242880,
+          "minimum": 1024,
+          "maximum": 104857600,
+          "markdownDescription": "Max size per file (bytes) for initial sync."
         }
       }
     },
@@ -61,6 +80,9 @@
     "watch": "tsc -watch -p ./",
     "vscode:prepublish": "npm run compile"
   },
+  "dependencies": {
+    "ignore": "^5.3.2"
+  },
   "devDependencies": {
     "@types/node": "^20.11.0",
     "@types/vscode": "^1.85.0",

extensions/anythingllm-workspaces/src/anythingllmClient.ts

@@ -55,6 +55,17 @@ const normalizeApiSecret = (raw: string): string => {
   return bearerPrefix.test(trimmed) ? trimmed.replace(bearerPrefix, "").trim() : trimmed;
 };
 
+const parseWorkspaceEnvelope = (payload: unknown): AnythingWorkspace => {
+  if (!isRecord(payload)) {
+    throw new Error("AnythingLLM API: expected object body");
+  }
+  const ws = payload.workspace;
+  if (!isWorkspace(ws)) {
+    throw new Error("AnythingLLM API: missing workspace in response");
+  }
+  return ws;
+};
+
 export const listWorkspaces = async (
   baseUrl: string,
   apiKey: string,
@@ -74,24 +85,11 @@ export const listWorkspaces = async (
   });
   const text = await response.text();
   if (!response.ok) {
-    throw new Error(
+    throw new Error(`AnythingLLM API ${response.status}: ${text.slice(0, 500)}`);
-      `AnythingLLM API ${response.status}: ${text.slice(0, 500)}`,
-    );
   }
   return parseListWorkspaces(parseJson(text));
 };
 
-const parseWorkspaceEnvelope = (payload: unknown): AnythingWorkspace => {
-  if (!isRecord(payload)) {
-    throw new Error("AnythingLLM API: expected object body");
-  }
-  const ws = payload.workspace;
-  if (!isWorkspace(ws)) {
-    throw new Error("AnythingLLM API: missing workspace in response");
-  }
-  return ws;
-};
-
 export const createWorkspace = async (
   baseUrl: string,
   apiKey: string,
@@ -118,9 +116,7 @@ export const createWorkspace = async (
   });
   const text = await response.text();
   if (!response.ok) {
-    throw new Error(
+    throw new Error(`AnythingLLM API ${response.status}: ${text.slice(0, 500)}`);
-      `AnythingLLM API ${response.status}: ${text.slice(0, 500)}`,
-    );
   }
   return parseWorkspaceEnvelope(parseJson(text));
 };

extensions/anythingllm-workspaces/src/anythingllmDocumentApi.ts

@@ -0,0 +1,56 @@
+import * as fs from "node:fs/promises";
+import { normalizeAnythingLlmBaseUrl } from "./anythingllmClient";
+
+const normalizeApiSecret = (raw: string): string => {
+  const trimmed = raw.trim();
+  const bearerPrefix = /^Bearer\s+/i;
+  return bearerPrefix.test(trimmed) ? trimmed.replace(bearerPrefix, "").trim() : trimmed;
+};
+
+const isRecord = (value: unknown): value is Record<string, unknown> =>
+  typeof value === "object" && value !== null && !Array.isArray(value);
+
+export const uploadLocalFileToWorkspace = async (
+  baseUrl: string,
+  apiKey: string,
+  workspaceSlug: string,
+  absoluteFilePath: string,
+  uploadFileName: string,
+): Promise<void> => {
+  const normalized = normalizeAnythingLlmBaseUrl(baseUrl);
+  const key = normalizeApiSecret(apiKey);
+  if (key.length === 0) {
+    throw new Error("anythingllm.apiKey is empty");
+  }
+  const buf = await fs.readFile(absoluteFilePath);
+  const body = new FormData();
+  body.append("file", new Blob([buf]), uploadFileName);
+  body.append("addToWorkspaces", workspaceSlug);
+  const url = `${normalized}/api/v1/document/upload`;
+  const response = await fetch(url, {
+    method: "POST",
+    headers: {
+      Authorization: `Bearer ${key}`,
+    },
+    body,
+  });
+  const text = await response.text();
+  let parsed: unknown;
+  try {
+    parsed = JSON.parse(text) as unknown;
+  } catch {
+    throw new Error(`document upload: non-JSON response ${response.status}: ${text.slice(0, 300)}`);
+  }
+  if (!response.ok) {
+    throw new Error(`document upload ${response.status}: ${text.slice(0, 500)}`);
+  }
+  if (!isRecord(parsed)) {
+    throw new Error("document upload: invalid JSON body");
+  }
+  const success = parsed.success;
+  const err = parsed.error;
+  if (success !== true) {
+    const msg = typeof err === "string" ? err : JSON.stringify(err);
+    throw new Error(`document upload failed: ${msg}`);
+  }
+};

extensions/anythingllm-workspaces/src/commandParser.ts

@@ -3,6 +3,7 @@ export type ParsedDevCommand =
   | { readonly kind: "repos-list" }
   | { readonly kind: "repos-load"; readonly name: string; readonly sync: boolean }
   | { readonly kind: "workspace-load"; readonly name: string }
+  | { readonly kind: "workspace-sync-repo"; readonly name: string }
   | { readonly kind: "help" }
   | { readonly kind: "unknown"; readonly raw: string };
 
@@ -32,6 +33,9 @@ export const parseDevCommandLine = (line: string): ParsedDevCommand => {
   if (cmd === "/workspace-load") {
     return { kind: "workspace-load", name: argRest };
   }
+  if (cmd === "/workspace-sync") {
+    return { kind: "workspace-sync-repo", name: argRest };
+  }
   if (cmd === "help" || cmd === "/help") {
     return { kind: "help" };
   }
@@ -41,12 +45,13 @@ export const parseDevCommandLine = (line: string): ParsedDevCommand => {
 export const devCommandsHelpText = (): string => {
   return [
     "Commands (one per line):",
-    "  /repos-clone <git-url>     — clone into REPOS_DEVTOOLS_ROOT, branch test (default)",
+    "  /repos-clone <git-url>      — clone (branch test)",
-    "  /repos-clone-sync <url>   — clone + ensure AnythingLLM workspace (same name) + open folder + browser",
+    "  /repos-clone-sync <url>    — clone + workspace + open folder + optional initial RAG upload (.4nkaiignore)",
-    "  repos-list                 — list cloned git repos under root",
+    "  repos-list                  — list git repos under REPOS_DEVTOOLS_ROOT",
-    "  /repos-load <name>        — verify repo; open folder in editor",
+    "  /repos-load <name>         — verify repo + open folder",
-    "  /repos-load-sync <name>   — open folder + ensure workspace + browser",
+    "  /repos-load-sync <name>    — open folder + workspace + optional initial RAG upload",
-    "  /workspace-load <name>    — ensure workspace by name (create via API if missing) + browser",
+    "  /workspace-load <name>     — ensure workspace + browser",
+    "  /workspace-sync <name>     — ensure workspace + initial RAG upload (repo must exist under root)",
     "  help                        — this list",
   ].join("\n");
 };

extensions/anythingllm-workspaces/src/config.ts

@@ -7,6 +7,9 @@ export interface DevToolsConfigSnapshot {
   readonly anythingApiKey: string;
   readonly reposApiBaseUrl: string;
   readonly reposApiToken: string;
+  readonly initialSyncAfterClone: boolean;
+  readonly initialSyncMaxFiles: number;
+  readonly initialSyncMaxFileBytes: number;
 }
 
 export const readAnythingConfig = (): { baseUrl: string; apiKey: string } => {
@@ -16,6 +19,14 @@ export const readAnythingConfig = (): { baseUrl: string; apiKey: string } => {
   return { baseUrl, apiKey };
 };
 
+const readPositiveInt = (cfg: vscode.WorkspaceConfiguration, key: string, fallback: number): number => {
+  const v = cfg.get(key);
+  if (typeof v === "number" && Number.isFinite(v) && v > 0) {
+    return Math.floor(v);
+  }
+  return fallback;
+};
+
 export const readDevToolsConfig = (): DevToolsConfigSnapshot => {
   const cfg = vscode.workspace.getConfiguration(CONFIG_SECTION);
   const { baseUrl, apiKey } = readAnythingConfig();
@@ -27,10 +38,16 @@ export const readDevToolsConfig = (): DevToolsConfigSnapshot => {
     typeof cfg.get("reposApiToken") === "string"
       ? (cfg.get("reposApiToken") as string)
       : "";
+  const initialSyncAfterClone = cfg.get("initialSyncAfterClone") !== false;
+  const initialSyncMaxFiles = readPositiveInt(cfg, "initialSyncMaxFiles", 400);
+  const initialSyncMaxFileBytes = readPositiveInt(cfg, "initialSyncMaxFileBytes", 5_242_880);
   return {
     anythingBaseUrl: baseUrl,
     anythingApiKey: apiKey,
     reposApiBaseUrl,
     reposApiToken,
+    initialSyncAfterClone,
+    initialSyncMaxFiles,
+    initialSyncMaxFileBytes,
   };
 };

extensions/anythingllm-workspaces/src/devToolsExecutor.ts

@@ -7,6 +7,7 @@ import {
 import { normalizeAnythingLlmBaseUrl } from "./anythingllmClient";
 import { reposApiClone, reposApiList, reposApiLoad } from "./reposApiClient";
 import { ensureWorkspaceForRepoName } from "./workspaceEnsure";
+import { runInitialRagImportFromRepo } from "./initialRagSync";
 
 const DEFAULT_BRANCH = "test";
 
@@ -26,6 +27,10 @@ export interface DevToolsRunnerContext {
   readonly anythingApiKey: string;
   readonly reposApiBaseUrl: string;
   readonly reposApiToken: string;
+  readonly initialSyncAfterClone: boolean;
+  readonly initialSyncMaxFiles: number;
+  readonly initialSyncMaxFileBytes: number;
+  readonly default4nkaiignoreTemplateFsPath: string;
   readonly openFolder: (fsPath: string) => Thenable<void>;
   readonly openAnythingWorkspaceInBrowser: (slug: string) => Thenable<void>;
 }
@@ -45,6 +50,27 @@ const assertAnythingConfig = (ctx: DevToolsRunnerContext): void => {
   }
 };
 
+const appendInitialRag = async (
+  ctx: DevToolsRunnerContext,
+  repoRoot: string,
+  workspaceSlug: string,
+): Promise<string> => {
+  if (!ctx.initialSyncAfterClone) {
+    return "";
+  }
+  assertAnythingConfig(ctx);
+  const res = await runInitialRagImportFromRepo({
+    baseUrl: ctx.anythingBaseUrl,
+    apiKey: ctx.anythingApiKey,
+    workspaceSlug,
+    repoRoot,
+    templateFsPath: ctx.default4nkaiignoreTemplateFsPath,
+    maxFiles: ctx.initialSyncMaxFiles,
+    maxFileBytes: ctx.initialSyncMaxFileBytes,
+  });
+  return `\n---\nInitial RAG sync: ${fmt(res)}`;
+};
+
 const runOne = async (
   cmd: ParsedDevCommand,
   ctx: DevToolsRunnerContext,
@@ -91,8 +117,9 @@ const runOne = async (
       out += `\n---\nAnythingLLM workspace: ${fmt({
         slug: ensured.workspace.slug,
         name: ensured.workspace.name,
-        created: ensured.created,
+        workspaceCreatedByApi: ensured.created,
       })}`;
+      out += await appendInitialRag(ctx, fsPath, ensured.workspace.slug);
       await ctx.openFolder(fsPath);
       await ctx.openAnythingWorkspaceInBrowser(ensured.workspace.slug);
     }
@@ -120,8 +147,9 @@ const runOne = async (
       out += `\n---\nAnythingLLM workspace: ${fmt({
         slug: ensured.workspace.slug,
         name: ensured.workspace.name,
-        created: ensured.created,
+        workspaceCreatedByApi: ensured.created,
       })}`;
+      out += await appendInitialRag(ctx, loaded.path, ensured.workspace.slug);
       await ctx.openAnythingWorkspaceInBrowser(ensured.workspace.slug);
     }
     return out;
@@ -140,9 +168,34 @@ const runOne = async (
     return fmt({
       slug: ensured.workspace.slug,
       name: ensured.workspace.name,
-      created: ensured.created,
+      workspaceCreatedByApi: ensured.created,
     });
   }
+  if (cmd.kind === "workspace-sync-repo") {
+    assertReposConfig(ctx);
+    assertAnythingConfig(ctx);
+    if (cmd.name.length === 0) {
+      throw new Error("/workspace-sync requires a repository folder name.");
+    }
+    const loaded = await reposApiLoad(
+      ctx.reposApiBaseUrl,
+      ctx.reposApiToken,
+      cmd.name,
+    );
+    const ensured = await ensureWorkspaceForRepoName(
+      ctx.anythingBaseUrl,
+      ctx.anythingApiKey,
+      loaded.name,
+    );
+    let out = fmt({
+      repoPath: loaded.path,
+      slug: ensured.workspace.slug,
+      name: ensured.workspace.name,
+      workspaceCreatedByApi: ensured.created,
+    });
+    out += await appendInitialRag(ctx, loaded.path, ensured.workspace.slug);
+    return out;
+  }
   return `Unhandled: ${JSON.stringify(cmd)}`;
 };
 

extensions/anythingllm-workspaces/src/devToolsPanel.ts

@@ -33,7 +33,7 @@ const buildHtml = (
 <body>
   <div class="row">
     <label for="cmd">Commands (one per line)</label>
-    <textarea id="cmd" spellcheck="false" placeholder="/repos-clone https://…&#10;repos-list"></textarea>
+    <textarea id="cmd" spellcheck="false" placeholder="/repos-clone-sync https://…&#10;/workspace-sync my-repo"></textarea>
   </div>
   <div class="row">
     <button type="button" id="run">Run</button>
@@ -67,6 +67,11 @@ export const showDevToolsPanel = (
   panel.webview.html = buildHtml(panel.webview, context.extensionUri);
 
   const openFolder = registerDevToolsOpenFolder(vscode);
+  const templateFsPath = vscode.Uri.joinPath(
+    context.extensionUri,
+    "templates",
+    "4nkaiignore.default",
+  ).fsPath;
 
   panel.webview.onDidReceiveMessage(
     (msg: unknown) => {
@@ -87,6 +92,10 @@ export const showDevToolsPanel = (
             anythingApiKey: use.anythingApiKey,
             reposApiBaseUrl: use.reposApiBaseUrl,
             reposApiToken: use.reposApiToken,
+            initialSyncAfterClone: use.initialSyncAfterClone,
+            initialSyncMaxFiles: use.initialSyncMaxFiles,
+            initialSyncMaxFileBytes: use.initialSyncMaxFileBytes,
+            default4nkaiignoreTemplateFsPath: templateFsPath,
             openFolder,
             openAnythingWorkspaceInBrowser: openBrowser,
           });

extensions/anythingllm-workspaces/src/initialRagSync.ts

@@ -0,0 +1,136 @@
+import ignore from "ignore";
+import * as fs from "node:fs/promises";
+import * as path from "node:path";
+import { uploadLocalFileToWorkspace } from "./anythingllmDocumentApi";
+
+const ALWAYS_IGNORE = [".git/", "node_modules/", "**/node_modules/"].join("\n");
+
+export interface InitialRagImportResult {
+  readonly uploaded: number;
+  readonly skipped: number;
+  readonly errors: readonly string[];
+  readonly dotfileCreated: boolean;
+  readonly capped: boolean;
+}
+
+export const ensureDot4nkaiignoreFromTemplate = async (
+  repoRoot: string,
+  templateFsPath: string,
+): Promise<{ created: boolean }> => {
+  const target = path.join(repoRoot, ".4nkaiignore");
+  try {
+    await fs.access(target);
+    return { created: false };
+  } catch {
+    const tmpl = await fs.readFile(templateFsPath, "utf8");
+    await fs.writeFile(target, tmpl, "utf8");
+    return { created: true };
+  }
+};
+
+const walkFiles = async (dir: string): Promise<string[]> => {
+  const out: string[] = [];
+  const scan = async (d: string): Promise<void> => {
+    const entries = await fs.readdir(d, { withFileTypes: true });
+    for (const e of entries) {
+      const p = path.join(d, e.name);
+      if (e.isSymbolicLink()) {
+        continue;
+      }
+      if (e.isDirectory()) {
+        await scan(p);
+        continue;
+      }
+      if (e.isFile()) {
+        out.push(p);
+      }
+    }
+  };
+  await scan(dir);
+  return out;
+};
+
+const toPosixRel = (root: string, abs: string): string => {
+  const rel = path.relative(root, abs);
+  return rel.split(path.sep).join("/");
+};
+
+const uploadNameForRel = (rel: string): string => {
+  return rel.split("/").join("__");
+};
+
+export const runInitialRagImportFromRepo = async (opts: {
+  readonly baseUrl: string;
+  readonly apiKey: string;
+  readonly workspaceSlug: string;
+  readonly repoRoot: string;
+  readonly templateFsPath: string;
+  readonly maxFiles: number;
+  readonly maxFileBytes: number;
+}): Promise<InitialRagImportResult> => {
+  const dot = await ensureDot4nkaiignoreFromTemplate(opts.repoRoot, opts.templateFsPath);
+  const ignorePath = path.join(opts.repoRoot, ".4nkaiignore");
+  let userRules = "";
+  try {
+    userRules = await fs.readFile(ignorePath, "utf8");
+  } catch {
+    userRules = "";
+  }
+  const ig = ignore();
+  ig.add(ALWAYS_IGNORE);
+  ig.add(userRules);
+
+  const absFiles = await walkFiles(opts.repoRoot);
+  const candidates: string[] = [];
+  for (const abs of absFiles) {
+    const rel = toPosixRel(opts.repoRoot, abs);
+    if (rel.length === 0 || rel.startsWith("..")) {
+      continue;
+    }
+    if (ig.ignores(rel)) {
+      continue;
+    }
+    candidates.push(abs);
+  }
+
+  let uploaded = 0;
+  let skipped = 0;
+  const errors: string[] = [];
+  let capped = false;
+
+  for (const abs of candidates) {
+    if (uploaded >= opts.maxFiles) {
+      capped = true;
+      skipped += 1;
+      continue;
+    }
+    const st = await fs.stat(abs);
+    if (st.size > opts.maxFileBytes) {
+      skipped += 1;
+      continue;
+    }
+    const rel = toPosixRel(opts.repoRoot, abs);
+    const uploadName = uploadNameForRel(rel);
+    try {
+      await uploadLocalFileToWorkspace(
+        opts.baseUrl,
+        opts.apiKey,
+        opts.workspaceSlug,
+        abs,
+        uploadName,
+      );
+      uploaded += 1;
+    } catch (e) {
+      const m = e instanceof Error ? e.message : String(e);
+      errors.push(`${rel}: ${m}`);
+    }
+  }
+
+  return {
+    uploaded,
+    skipped,
+    errors,
+    dotfileCreated: dot.created,
+    capped,
+  };
+};

extensions/anythingllm-workspaces/src/reposApiClient.ts

@@ -38,7 +38,9 @@ export const reposApiClone = async (
     /* keep text */
   }
   if (!res.ok) {
-    throw new Error(`repos API ${res.status}: ${typeof body === "string" ? body : JSON.stringify(body)}`);
+    throw new Error(
+      `repos API ${res.status}: ${typeof body === "string" ? body : JSON.stringify(body)}`,
+    );
   }
   return body;
 };
@@ -85,10 +87,10 @@ export const reposApiLoad = async (
     throw new Error("repos-load: invalid response");
   }
   const rec = body as Record<string, unknown>;
-  const path = rec.path;
+  const p = rec.path;
   const n = rec.name;
-  if (typeof path !== "string" || typeof n !== "string") {
+  if (typeof p !== "string" || typeof n !== "string") {
     throw new Error("repos-load: missing path or name");
   }
-  return { path, name: n };
+  return { path: p, name: n };
 };

extensions/anythingllm-workspaces/src/types.ts

@@ -1,14 +1,7 @@
-export interface AnythingThreadSummary {
-  readonly user_id: number | null;
-  readonly slug: string;
-  readonly name: string | null;
-}
-
 export interface AnythingWorkspace {
   readonly id: number;
   readonly name: string;
   readonly slug: string;
   readonly createdAt?: string;
   readonly lastUpdatedAt?: string;
-  readonly threads?: readonly AnythingThreadSummary[];
 }

extensions/anythingllm-workspaces/templates/4nkaiignore.default

@@ -0,0 +1,54 @@
+# .4nkaiignore — same rules as .gitignore (see gitignore(5))
+# Used by the AnythingLLM Workspaces extension to filter the initial document upload
+# after clone or /repos-load-sync. Copy or rename to `.4nkaiignore` at the repo root.
+
+# VCS
+.git/
+
+# Dependencies & build outputs
+node_modules/
+**/node_modules/
+dist/
+out/
+build/
+.next/
+.turbo/
+coverage/
+.nyc_output/
+target/
+
+# IDE / OS
+.idea/
+.vscode/
+.DS_Store
+Thumbs.db
+
+# Secrets & local env (never embed)
+.env
+.env.*
+!.env.example
+
+# Large or binary artifacts (remove a line if your project should embed that type)
+*.png
+*.jpg
+*.jpeg
+*.gif
+*.webp
+*.ico
+*.pdf
+*.zip
+*.tar
+*.gz
+*.7z
+*.wasm
+*.so
+*.dylib
+*.dll
+*.exe
+*.mp4
+*.mp3
+
+# Minified bundles (often redundant with sources)
+*.min.js
+*.min.css
+*.map

services/repos-devtools-server/README.md

@@ -1,47 +1,35 @@
 # repos-devtools-server
 
-Local HTTP API bound to **`127.0.0.1`** for git operations under **`REPOS_DEVTOOLS_ROOT`** (default `/home/ncantu/code`).
+Local HTTP API on **`127.0.0.1`** for git operations under **`REPOS_DEVTOOLS_ROOT`** (default `/home/ncantu/code`).
+
+After a successful **`POST /repos-clone`**, if the new repo has no **`.4nkaiignore`**, the server copies **`templates/4nkaiignore.default`** into the repository root. The response includes **`fourNkAiIgnoreTemplateWrote`: boolean**.
 
 ## Environment
 
 | Variable | Required | Description |
 |----------|----------|-------------|
-| `REPOS_DEVTOOLS_TOKEN` | yes | Shared secret; clients send `Authorization: Bearer <token>`. |
+| `REPOS_DEVTOOLS_TOKEN` | yes | `Authorization: Bearer <token>` on every request. |
-| `REPOS_DEVTOOLS_ROOT` | no | Absolute root for clones (default `/home/ncantu/code`). |
+| `REPOS_DEVTOOLS_ROOT` | no | Clone root (default `/home/ncantu/code`). |
 | `REPOS_DEVTOOLS_HOST` | no | Bind address (default `127.0.0.1`). |
 | `REPOS_DEVTOOLS_PORT` | no | Port (default `37140`). |
 
 ## Endpoints
 
-- `POST /repos-clone` — JSON `{ "url": "<git url>", "branch": "test" }` (`branch` optional, default `test`).
+- `POST /repos-clone` — JSON `{ "url": "<git>", "branch": "test" }` (`branch` optional).
-- `GET /repos-list` — Lists immediate subdirectories of the root that contain `.git`.
+- `GET /repos-list`
-- `POST /repos-load` — JSON `{ "name": "<folder name>" }` — Verifies the repo exists; returns absolute `path`.
+- `POST /repos-load` — JSON `{ "name": "<folder>" }`
-
-All endpoints require `Authorization: Bearer <REPOS_DEVTOOLS_TOKEN>`.
 
 ## Run
 
-### One-off (foreground)
-
 ```bash
-cd services/repos-devtools-server
 npm install
 npm run build
-# Create .env (gitignored) with REPOS_DEVTOOLS_TOKEN=... and REPOS_DEVTOOLS_ROOT=/home/ncantu/code
+export REPOS_DEVTOOLS_TOKEN='…'
-set -a && source .env && set +a && node dist/server.js
+npm start
 ```
 
-### systemd (user)
+After upgrading, reload **systemd** if used: `systemctl --user daemon-reload && systemctl --user restart repos-devtools-server.service`.
 
-Copy `systemd/user/repos-devtools-server.service` to `~/.config/systemd/user/`, create `.env` beside this README, then:
+## Templates
 
-```bash
+Keep **`templates/4nkaiignore.default`** aligned with `extensions/anythingllm-workspaces/templates/4nkaiignore.default` in the monorepo when you change defaults.
-systemctl --user daemon-reload
-systemctl --user enable --now repos-devtools-server.service
-```
-
-Use the same token in the VS Code / Cursor setting **`anythingllm.reposApiToken`**.
-
-## Integration
-
-The **AnythingLLM Workspaces** extension command **AnythingLLM: Dev tools panel** calls this API and the AnythingLLM HTTP API for workspace create/list.

services/repos-devtools-server/package.json

@@ -1,8 +1,8 @@
 {
   "name": "@4nk/repos-devtools-server",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "private": true,
-  "description": "Local HTTP API: git clone/list under REPOS_DEVTOOLS_ROOT (e.g. /home/ncantu/code).",
+  "description": "Local HTTP API: git clone/list under REPOS_DEVTOOLS_ROOT; writes default .4nkaiignore after clone.",
   "license": "MIT",
   "type": "module",
   "main": "dist/server.js",

services/repos-devtools-server/src/handlers.ts

@@ -9,6 +9,7 @@ import {
   repoDirForName,
   repoNameFromGitUrl,
 } from "./paths.js";
+import { copyDefault4nkaiignoreIfMissing } from "./write4nkaiignore.js";
 
 const json = (res: ServerResponse, status: number, body: unknown): void => {
   res.writeHead(status, { "Content-Type": "application/json; charset=utf-8" });
@@ -82,12 +83,27 @@ export const handleReposClone = async (
     });
     return;
   }
+  let fourNkAiIgnoreTemplateWrote = false;
+  try {
+    const c = await copyDefault4nkaiignoreIfMissing(dest);
+    fourNkAiIgnoreTemplateWrote = c.wrote;
+  } catch (e) {
+    const msg = e instanceof Error ? e.message : String(e);
+    json(res, 500, {
+      error: "clone ok but failed to write default .4nkaiignore template",
+      detail: msg,
+      name,
+      path: dest,
+    });
+    return;
+  }
   json(res, 200, {
     ok: true,
     name,
     path: dest,
     branch,
     url,
+    fourNkAiIgnoreTemplateWrote,
   });
 };
 

services/repos-devtools-server/src/paths.ts

@@ -16,8 +16,7 @@ export const assertSafeRepoName = (name: string): string => {
 
 export const getCodeRoot = (): string => {
   const raw = process.env.REPOS_DEVTOOLS_ROOT ?? "/home/ncantu/code";
-  const resolved = path.resolve(raw);
+  return path.resolve(raw);
-  return resolved;
 };
 
 export const repoDirForName = (codeRoot: string, name: string): string => {

services/repos-devtools-server/src/server.ts

@@ -1,5 +1,5 @@
 import * as http from "node:http";
-import { requireBearer, readExpectedToken } from "./auth.js";
+import { readExpectedToken, requireBearer } from "./auth.js";
 import {
   handleReposClone,
   handleReposList,

services/repos-devtools-server/src/write4nkaiignore.ts

@@ -0,0 +1,23 @@
+import * as fs from "node:fs/promises";
+import * as path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const templateFsPath = (): string => {
+  const here = path.dirname(fileURLToPath(import.meta.url));
+  return path.join(here, "..", "templates", "4nkaiignore.default");
+};
+
+export const copyDefault4nkaiignoreIfMissing = async (
+  repoRoot: string,
+): Promise<{ wrote: boolean }> => {
+  const target = path.join(repoRoot, ".4nkaiignore");
+  try {
+    await fs.access(target);
+    return { wrote: false };
+  } catch {
+    const src = templateFsPath();
+    const content = await fs.readFile(src, "utf8");
+    await fs.writeFile(target, content, "utf8");
+    return { wrote: true };
+  }
+};

services/repos-devtools-server/templates/4nkaiignore.default

@@ -0,0 +1,54 @@
+# .4nkaiignore — same rules as .gitignore (see gitignore(5))
+# Used to filter the initial document upload to AnythingLLM (extension).
+# Copy or rename to `.4nkaiignore` at the repo root and adjust per project.
+
+# VCS
+.git/
+
+# Dependencies & build outputs
+node_modules/
+**/node_modules/
+dist/
+out/
+build/
+.next/
+.turbo/
+coverage/
+.nyc_output/
+target/
+
+# IDE / OS
+.idea/
+.vscode/
+.DS_Store
+Thumbs.db
+
+# Secrets & local env (never embed)
+.env
+.env.*
+!.env.example
+
+# Large or binary artifacts (remove a line if your project should embed that type)
+*.png
+*.jpg
+*.jpeg
+*.gif
+*.webp
+*.ico
+*.pdf
+*.zip
+*.tar
+*.gz
+*.7z
+*.wasm
+*.so
+*.dylib
+*.dll
+*.exe
+*.mp4
+*.mp3
+
+# Minified bundles (often redundant with sources)
+*.min.js
+*.min.css
+*.map