smart_ide/services/ia_dev/git-issues/wiki-migrate-docs.sh
Nicolas Cantu 58cc2493e5 chore: consolidate ia_dev module, sync tooling, and harden gateways (0.0.5)
Initial state:
- ia_dev was historically referenced as ./ia_dev in docs and integrations, while the vendored module lives under services/ia_dev.
- AnythingLLM sync and hook installation had error masking / weak exit signaling.
- Proxy layers did not validate proxy path segments, allowing path normalization tricks.

Motivation:
- Make the IDE-oriented workflow usable (sync -> act -> deploy/preview) with explicit errors.
- Reduce security footguns in proxying and script automation.

Resolution:
- Standardize IA_DEV_ROOT usage and documentation to services/ia_dev.
- Add SSH remote data mirroring + optional AnythingLLM ingestion.
- Extend AnythingLLM pull sync to support upload-all/prefix and fail on upload errors.
- Harden smart-ide-sso-gateway and smart-ide-global-api proxying with safe-path checks and non-leaking error responses.
- Improve ia-dev-gateway runner validation and reduce sensitive path leakage.
- Add site scaffold tool (Vite/React) with OIDC + chat via sso-gateway -> orchestrator.

Root cause:
- Historical layout changes (submodule -> vendored tree) and missing central contracts for path resolution.
- Missing validation for proxy path traversal patterns.
- Overuse of silent fallbacks (|| true, exit 0 on partial failures) in automation scripts.

Impacted features:
- Project sync: git pull + AnythingLLM sync + remote data mirror ingestion.
- Site frontends: SSO gateway proxy and orchestrator intents (rag.query, chat.local).
- Agent execution: ia-dev-gateway script runner and SSE output.

Code modified:
- scripts/remote-data-ssh-sync.sh
- scripts/anythingllm-pull-sync/sync.mjs
- scripts/install-anythingllm-post-merge-hook.sh
- cron/git-pull-project-clones.sh
- services/smart-ide-sso-gateway/src/server.ts
- services/smart-ide-global-api/src/server.ts
- services/smart-ide-orchestrator/src/server.ts
- services/ia-dev-gateway/src/server.ts
- services/ia_dev/tools/site-generate.sh

Documentation modified:
- docs/** (architecture, API docs, ia_dev module + integration, scripts)

Configurations modified:
- config/services.local.env.example
- services/*/.env.example

Files in deploy modified:
- services/ia_dev/deploy/*

Files in logs impacted:
- logs/ia_dev.log (runtime only)
- .logs/* (runtime only)

Databases and other sources modified:
- None

Off-project modifications:
- None

Files in .smartIde modified:
- .smartIde/agents/*.md
- services/ia_dev/.smartIde/**

Files in .secrets modified:
- None

New patch version in VERSION:
- 0.0.5

CHANGELOG.md updated:
- yes
2026-04-04 18:36:43 +02:00

101 lines
3.0 KiB
Bash
Executable File

#!/usr/bin/env bash
#
# Migrate all docs/*.md (repo root) to Gitea wiki as pages.
# Mapping: docs/FILE.md → page "File" (stem with _ → -, first letter upper per segment).
# Requires GITEA_TOKEN or .secrets/git-issues/token.
# Usage: ./wiki-migrate-docs.sh [--dry-run] [file.md ...]
# --dry-run: print mapping and skip API calls.
# If file(s) given: migrate only those; else migrate all docs/*.md.
#
set -euo pipefail
GIT_ISSUES_DIR="${GIT_ISSUES_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}"
REPO_ROOT="${GIT_ISSUES_DIR}/.."
DOCS_DIR="${DOCS_DIR:-${REPO_ROOT}/docs}"
# shellcheck source=lib.sh
source "${GIT_ISSUES_DIR}/lib.sh"
REPO_PATH="/repos/${GITEA_REPO_OWNER}/${GITEA_REPO_NAME}"
GITEA_WIKI_REF="${GITEA_WIKI_REF:-master}"
WIKI_PAGE="${REPO_PATH}/wiki/page"
WIKI_NEW="${REPO_PATH}/wiki/new"
# docs/FILE.md → page name for wiki (stem: _ → -, title-case: First-Letter-Of-Each-Segment)
file_to_page_name() {
local base="$1"
local stem="${base%.md}"
echo "$stem" | tr '_' '-' | awk -F- '{
for(i=1;i<=NF;i++) {
s = $i; l = length(s)
if (l > 0) $i = toupper(substr(s,1,1)) tolower(substr(s,2))
}
}1' OFS='-'
}
dry_run=false
files=()
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) dry_run=true; shift ;;
*.md) files+=("$1"); shift ;;
*) log_err "Unknown option or not .md: $1"; exit 1 ;;
esac
done
if [[ ${#files[@]} -eq 0 ]]; then
while IFS= read -r -d '' f; do
files+=("$f")
done < <(find "$DOCS_DIR" -maxdepth 1 -name '*.md' -print0 | sort -z)
else
# Resolve args to full paths under DOCS_DIR
for i in "${!files[@]}"; do
u="${files[$i]}"
if [[ "$u" != */* ]] && [[ -f "${DOCS_DIR}/${u}" ]]; then
files[$i]="${DOCS_DIR}/${u}"
fi
done
fi
if [[ ${#files[@]} -eq 0 ]]; then
log_err "No .md files found in ${DOCS_DIR}"
exit 1
fi
if [[ "$dry_run" == true ]]; then
log_info "Dry run: would migrate ${#files[@]} file(s)"
for f in "${files[@]}"; do
base="$(basename "$f")"
page="$(file_to_page_name "$base")"
echo " $f$page"
done
exit 0
fi
load_gitea_token || exit 1
require_jq || exit 1
for f in "${files[@]}"; do
base="$(basename "$f")"
page="$(file_to_page_name "$base")"
if [[ ! -f "$f" ]]; then
log_err "Skip (not a file): $f"
continue
fi
content="$(cat "$f")"
content_b64="$(echo -n "$content" | base64 -w 0)"
body="$(jq -n --arg title "$page" --arg content "$content_b64" --arg msg "Migrate from docs/$base" \
'{ title: $title, content_base64: $content, message: $msg }')"
# Check if page exists (GET); if 200 use PATCH else POST
resp="$(gitea_api_get "${REPO_PATH}/wiki/page/${page}?ref=${GITEA_WIKI_REF}")"
if echo "$resp" | jq -e .title &>/dev/null; then
log_info "Update: $base$page"
patch_body="$(jq -n --arg content "$content_b64" --arg msg "Update from docs/$base" '{ content_base64: $content, message: $msg }')"
gitea_api_patch "${WIKI_PAGE}/${page}?ref=${GITEA_WIKI_REF}" "$patch_body" >/dev/null || true
else
log_info "Create: $base$page"
gitea_api_post "${WIKI_NEW}" "$body" >/dev/null || true
fi
done
log_info "Migration done: ${#files[@]} file(s)."