fix: saved-project URL restore, per-tab table filters, sidebar %; CI docs

2026-10-18 00:39:38 +02:00 · 2026-08-15 02:25:46 +03:00
1 parent f64e4c17d3
commit 18040e3c71
@@ -170,7 +170,7 @@ npm run dev
 <br />

 ```bash
-npm run build:cli
+npx tsc -b apps/cli     # builds the CLI + its workspace packages
 node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
 node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON

@@ -181,7 +181,7 @@ node apps/cli/dist/index.js analyze-logs access.log --format iis-w3c --verify-bo

 > **Sample logs** ship in [`samples/`](samples/) (`apache-access.txt`, `iis-access.txt`) so you can try the Log File Analyzer immediately — in the desktop app, open **Log Analyzer → Open Log Analyzer Window…** and import one.

-**CI / CD recipes** — ready-to-use [GitHub Actions](docs/ci/github-actions-example.yml) and [GitLab CI](docs/ci/gitlab-ci-example.yml) examples that crawl your site on a schedule, fail the build when broken-URL count exceeds a threshold, and upload the crawl as an artifact.
+**CI / CD recipes** — ready-to-use [GitHub Actions](docs/ci/github-actions-example.yml) and [GitLab CI](docs/ci/gitlab-ci-example.yml) examples that crawl your site on a schedule, fail the build when broken-URL count exceeds a threshold, and upload the crawl as an artifact. See the [CI/CD guide](docs/ci/README.md) for the exit-code contract and a regression-gate recipe.

 </details>

@@ -405,13 +405,9 @@ graph LR
 ## 📈 Status

 > [!NOTE]
-> **Active development.** All 32 analysis tabs are working — the core set (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP) plus the integration tabs (**PageSpeed, CrUX, Spelling, Search Console, GA4, AI, SEO Authority**).
+> **Active development — feature-complete for daily SEO audits.** All 32 analysis tabs, 150+ issue checks, JavaScript rendering (screenshots, LCP candidate, Mobile Usability), multi-window projects, the link visualiser, every Google integration (Search Console / GA4 / PageSpeed / CrUX), LanguageTool spelling, AI per-URL prompts, SEO Authority providers, Google Sheets + BigQuery export, encrypted projects, the MCP server with live crawl control, and the standalone Log File Analyzer all ship today.
 >
-> Also shipping: 150+ issue checks, **JavaScript rendering** (Playwright — post-JS DOM, screenshots, LCP candidate, Mobile Usability audit), **multi-window projects** (independent per-window crawls), near/exact-duplicate detection, hreflang validation, **internal PageRank link scoring**, project compare, the **link visualiser** (WebGL 3D force-directed crawl tree and link graph, plus 2D Crawl Tree and Directory Tree with click-to-collapse branches, hover summary cards, LCP and link-score overlays, and a click-to-trace Crawl Path Report), **Basic / Digest / Bearer + form-based & browser-driven SPA login**, **HTTP / SOCKS5 proxy**, the **MCP server with live crawl control**, all **Google integrations** (Search Console / URL Inspection / GA4 / PageSpeed / CrUX), **LanguageTool spelling & grammar**, **AI per-URL prompts** (OpenAI / Anthropic / Ollama), **SEO Authority providers** (Ahrefs / Majestic / Moz / Semrush), **Google Sheets + BigQuery export**, **encrypted project snapshots** (AES-256-GCM), **RAM-only storage mode**, scheduled crawls, robots.txt validator, URL rewriting, EN + TR UI, and the **standalone Log File Analyzer** (Apache / Nginx / IIS access logs → bot hits, crawl budget, crawl × log orphans, 40+ bot detection, CSV / Excel export, and a headless `analyze-logs` CLI).
->
-> Cross-platform installers ship for Windows (`.exe` + portable), macOS (`.dmg`), and Linux (`.AppImage` / `.deb` / `.rpm`), with Playwright Chromium bundled so JS rendering works on first launch. The virtualized 1M-row table streams the **first row in ~1 s**.
->
-> **Apple Silicon tuning:** long crawls hold a power assertion so macOS App Nap can't throttle them in the background, concurrency scales down automatically as the thermal state rises (so a fanless MacBook Air degrades gracefully instead of stalling), heavy aggregates never run on the main thread, and every large read is batch-streamed to stay clear of Electron's hard per-process heap ceiling. macOS also gets the full Windows-style `Cmd`-key shortcut set.
+> Cross-platform installers ship for Windows (`.exe` + portable), macOS (`.dmg`, arm64 + x64), and Linux (`.AppImage` / `.deb` / `.rpm`), with Playwright Chromium bundled so JS rendering works on first launch. The virtualized 1M-row table streams the first row in ~1 s, and long crawls are tuned for Apple Silicon (see [docs](docs/macos-apple-silicon-performance.md)).
 >
 > **Upcoming:** Plugin system, Light theme, Code-signing + auto-update.

@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/cli",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "type": "module",
  "bin": {
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/desktop",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "description": "FreeCrawl SEO â€” desktop application",
  "homepage": "https://github.com/kemalai/FreeCrawl-SEO-Tool",
@@ -1934,6 +1934,41 @@ function sweepStaleWorkingCopies(keep: ReadonlySet<string>): void {
  }
 }

+/**
+ * Rebuild the crawl config to restore into the UI when a project opens, so
+ * the URL bar and mode reflect the crawl the project holds.
+ *
+ * Prefers the full `lastCrawlConfig` that the desktop / MCP launch path
+ * persists (`launchCrawl`). When that key is absent — a project crawled by the
+ * headless CLI, which drives the engine directly and never writes it, or one
+ * saved by a build that predates `lastCrawlConfig` persistence — fall back to
+ * the seed the crawler engine ALWAYS records under `startUrl`. Without this
+ * fallback such a project reopens with a blank URL bar even though its tables
+ * are full, because the restore only ever consulted `lastCrawlConfig`.
+ *
+ * List / sitemap crawls store a fingerprint under `startUrl` ("list:…" /
+ * "sitemap:…"), not a display URL, so only an `http(s)` seed is adopted —
+ * those modes are seeded from a file / sitemap, not a single URL-bar value.
+ *
+ * Returns null when nothing is recoverable (a fresh scratch project); callers
+ * then leave the current config untouched.
+ */
+function restoreProjectConfig(db: ProjectDb): CrawlConfig | null {
+  const raw = db.getMeta('lastCrawlConfig');
+  if (raw) {
+    try {
+      return mergeCrawlConfig(JSON.parse(raw) as Partial<CrawlConfig>);
+    } catch {
+      /* malformed JSON — fall through to the engine seed */
+    }
+  }
+  const seed = db.getMeta('startUrl');
+  if (seed && /^https?:\/\//i.test(seed)) {
+    return mergeCrawlConfig({ startUrl: seed });
+  }
+  return null;
+}
+
 /**
 * Materialise a project document into a working database this session
 * can write to, then swap the DB + worker pools onto it.
@@ -2027,18 +2062,12 @@ async function openProjectAtPath(documentPath: string): Promise<void> {
  }
  // Rehydrate the crawl config from the project we just opened and push it
  // to the renderer so the UI reflects THIS project's saved settings
-  // (per-project settings). Null when the project has none saved yet — the
-  // renderer then keeps its current config.
-  let projectConfig: CrawlConfig | null = null;
-  try {
-    const raw = s.db.getMeta('lastCrawlConfig');
-    if (raw) {
-      projectConfig = mergeCrawlConfig(JSON.parse(raw) as Partial<CrawlConfig>);
-      s.lastCrawlConfig = projectConfig;
-    }
-  } catch {
-    /* malformed — leave config as-is */
-  }
+  // (per-project settings). Falls back to the engine's `startUrl` seed when
+  // `lastCrawlConfig` is absent (CLI-crawled or legacy project) so the URL
+  // bar isn't blank. Null when nothing is recoverable — the renderer then
+  // keeps its current config.
+  const projectConfig = restoreProjectConfig(s.db);
+  if (projectConfig) s.lastCrawlConfig = projectConfig;
  s.send(IPC.projectConfigChanged, projectConfig);
  fireDataChanged(s);
  // `fireDataChanged` marks the session dirty — but loading a document is
@@ -8039,10 +8068,10 @@ function registerIpc(): void {
  registerHandle(IPC.projectConfigGet, (): CrawlConfig | null => {
    if (!currentSession().currentProjectPath) return null;
    try {
-      const raw = getDb().getMeta('lastCrawlConfig');
-      return raw
-        ? mergeCrawlConfig(JSON.parse(raw) as Partial<CrawlConfig>)
-        : null;
+      // Same fallback as the open-project push: prefer `lastCrawlConfig`, else
+      // the engine's `startUrl` seed, so a CLI-crawled or legacy project still
+      // restores its URL bar on the renderer's cold-start pull.
+      return restoreProjectConfig(getDb());
    } catch {
      return null;
    }
@@ -12,8 +12,10 @@ interface Node {
  category?: UrlCategory;
  count?: number;
  /** Denominator for this node's "% of Total". Defaults to the tree-wide
-   *  total (internal URLs); image-level nodes override it with the total
-   *  image count so an image ÷ page ratio never shows a bogus %. */
+   *  total (internal URLs). Nodes whose count isn't internal-scoped override
+   *  it: image nodes use the total image count (so an image ÷ page ratio is
+   *  never bogus), and Response Codes / Security use internal+external (they
+   *  count external URLs, so an internal-only base would exceed 100%). */
  percentBase?: number;
  children?: Node[];
 }
@@ -273,6 +275,13 @@ const TreeNode = memo(function TreeNode({

 function buildTree(o: OverviewCounts | null): Node[] {
  if (!o) return [];
+  // Response Codes and Security deliberately count external URLs too (that's
+  // how a broken outbound link surfaces), so their numerators exceed the
+  // internal-only URL count. Their "% of Total" must therefore divide by the
+  // full crawled set (internal + external), not `totalInternalUrls` — else
+  // "All 6,657 / 6,555 = 109.30%" reads as a broken crawl. Every other group
+  // is internal-scoped and keeps the default (internal) base.
+  const allUrls = o.summary.totalInternalUrls + o.summary.totalExternalUrls;
  return [
    {
      key: 'summary',
@@ -333,37 +342,42 @@ function buildTree(o: OverviewCounts | null): Node[] {
      key: 'response-codes',
      label: 'Response Codes',
      children: [
-        { key: 'rc-all', label: 'All', count: o.responseCodes.all, category: 'all' },
+        { key: 'rc-all', label: 'All', count: o.responseCodes.all, category: 'all', percentBase: allUrls },
        {
          key: 'rc-blocked',
          label: 'Blocked by Robots',
          count: o.responseCodes.blockedRobots,
          category: 'status:blocked-robots',
+          percentBase: allUrls,
        },
        {
          key: 'rc-none',
          label: 'No Response',
          count: o.responseCodes.noResponse,
          category: 'status:no-response',
+          percentBase: allUrls,
        },
-        { key: 'rc-2xx', label: 'Success (2xx)', count: o.responseCodes.success2xx, category: 'status:2xx' },
+        { key: 'rc-2xx', label: 'Success (2xx)', count: o.responseCodes.success2xx, category: 'status:2xx', percentBase: allUrls },
        {
          key: 'rc-3xx',
          label: 'Redirection (3xx)',
          count: o.responseCodes.redirect3xx,
          category: 'status:3xx',
+          percentBase: allUrls,
        },
        {
          key: 'rc-4xx',
          label: 'Client Error (4xx)',
          count: o.responseCodes.clientError4xx,
          category: 'status:4xx',
+          percentBase: allUrls,
        },
        {
          key: 'rc-5xx',
          label: 'Server Error (5xx)',
          count: o.responseCodes.serverError5xx,
          category: 'status:5xx',
+          percentBase: allUrls,
        },
      ],
    },
@@ -376,8 +390,15 @@ function buildTree(o: OverviewCounts | null): Node[] {
          label: 'HTTPS URLs',
          count: o.security.https,
          category: 'security:https',
+          percentBase: allUrls,
+        },
+        {
+          key: 'sec-http',
+          label: 'HTTP URLs',
+          count: o.security.http,
+          category: 'security:http',
+          percentBase: allUrls,
        },
-        { key: 'sec-http', label: 'HTTP URLs', count: o.security.http, category: 'security:http' },
      ],
    },
    {
@@ -1,9 +1,11 @@
 import { create } from 'zustand';
 import {
  DEFAULT_CRAWL_CONFIG,
+  type AdvancedFilter,
  type CrawlConfig,
  type CrawlProgress,
  type CrawlSummary,
+  type CrawlUrlRow,
  type OverviewCounts,
  type UrlCategory,
 } from '@freecrawl/shared-types';
@@ -339,6 +341,22 @@ export const TAB_QUICK_FILTERS: Partial<Record<TabKey, TabQuickFilter[]>> = {
  ],
 };

+/**
+ * Ephemeral per-tab view state for the URLs table (search box, sort column,
+ * advanced filter). Kept per-tab because a sort/filter built against one
+ * tab's column vocabulary is meaningless on another. Persisted in the store
+ * (not prefs) so it survives leaving and returning to a tab within a session
+ * — including the full unmount that happens when the user visits a non-table
+ * view (Visualization, PageSpeed, …) — but does NOT outlive the app or leak
+ * across projects (cleared on project open).
+ */
+export interface TabViewState {
+  search: string;
+  sortBy: keyof CrawlUrlRow | undefined;
+  sortDir: 'asc' | 'desc';
+  filter: AdvancedFilter | null;
+}
+
 interface AppState {
  config: CrawlConfig;
  progress: CrawlProgress | null;
@@ -361,6 +379,11 @@ interface AppState {
  settingsTarget: SettingsTarget | null;
  recentUrls: string[];
  dataVersion: number;
+  /** Saved view state (search / sort / advanced filter) per table tab, so
+   *  returning to a tab restores what the user had set there. */
+  tabView: Partial<Record<TabKey, TabViewState>>;
+  /** Save a tab's current view state (called when the user leaves the tab). */
+  setTabView: (tab: TabKey, view: TabViewState) => void;
  setConfig: (patch: Partial<CrawlConfig>) => void;
  /** Replace the whole config from a project's saved settings WITHOUT
   *  writing it back to the global default prefs (per-project settings). */
@@ -480,6 +503,9 @@ export const useAppStore = create<AppState>((set) => ({
  settingsTarget: null,
  recentUrls: loadRecentUrls(),
  dataVersion: 0,
+  tabView: {},
+  setTabView: (tab, view) =>
+    set((s) => ({ tabView: { ...s.tabView, [tab]: view } })),
  setConfig: (patch) =>
    set((state) => {
      const next = { ...state.config, ...patch };
@@ -497,8 +523,10 @@ export const useAppStore = create<AppState>((set) => ({
    // sitemap, not blank in Spider mode while its tables hold the data.
    // Also the resume path after an interrupted crawl. Deliberately does
    // NOT touch global prefs — opening a project must not overwrite the
-    // user's default settings.
-    set({ config }),
+    // user's default settings. Saved per-tab view state is dropped too: a
+    // different project has its own tables, so a filter/sort from the last
+    // one must not carry over.
+    set({ config, tabView: {} }),
  setProgress: (p) => set({ progress: p }),
  setSummary: (s) => set({ summary: s }),
  markCrawlFinished: () =>
@@ -160,11 +160,30 @@ export function UrlsTab() {
        .filter((n) => n.trim().length > 0),
    [customExtractionRules],
  );
-  const [search, setSearch] = useState('');
-  const [sortBy, setSortBy] = useState<keyof CrawlUrlRow | undefined>(undefined);
-  const [sortDir, setSortDir] = useState<'asc' | 'desc'>('asc');
-  const [filter, setFilter] = useState<AdvancedFilter | null>(null);
+  const setTabView = useAppStore((s) => s.setTabView);
+  // Seed search / sort / filter from this tab's saved view state (if the user
+  // set one earlier this session and came back). The [activeTab] effect below
+  // restores on later tab switches and its cleanup saves the outgoing tab's
+  // state; these lazy initialisers cover the fresh mount after a full unmount
+  // (e.g. returning from Visualization / PageSpeed). Read once from the store
+  // at mount — the initialisers only run then.
+  const [search, setSearch] = useState(
+    () => useAppStore.getState().tabView[activeTab]?.search ?? '',
+  );
+  const [sortBy, setSortBy] = useState<keyof CrawlUrlRow | undefined>(
+    () => useAppStore.getState().tabView[activeTab]?.sortBy,
+  );
+  const [sortDir, setSortDir] = useState<'asc' | 'desc'>(
+    () => useAppStore.getState().tabView[activeTab]?.sortDir ?? 'asc',
+  );
+  const [filter, setFilter] = useState<AdvancedFilter | null>(
+    () => useAppStore.getState().tabView[activeTab]?.filter ?? null,
+  );
  const [filterDialogOpen, setFilterDialogOpen] = useState(false);
+  // Mirror the live view state so the [activeTab] effect's cleanup can save
+  // the latest values for the tab being left without re-subscribing.
+  const viewStateRef = useRef({ search, sortBy, sortDir, filter });
+  viewStateRef.current = { search, sortBy, sortDir, filter };
  const [exportDialogOpen, setExportDialogOpen] = useState(false);
  /** View mode toggle — list (flat virtual table), tree (URL-path
   *  hierarchy), or cluster (Duplicates-tab-only near-duplicate cluster
@@ -372,18 +391,19 @@ export function UrlsTab() {
    setPinnedLeftOrderState(loadPinnedLeft(activeTab));
    setColumnOrderState(loadColumnOrder(activeTab));
    setColumnPickerOpen(false);
-    // Scope ephemeral view state (search box / sort column / advanced
-    // filter) to the active tab — these are intentionally NOT persisted
-    // because each tab has a different column / category vocabulary
-    // (a sort by "depth" on Internal makes no sense on Response Codes,
-    // a filter clause on `images_missing_alt` is meaningless outside
-    // Images, etc). Without these resets the filter chips of one tab
-    // would silently apply to every other tab the user visits — which
-    // is exactly the bug surfaced in 2026-06-01.
-    setSearch('');
-    setSortBy(undefined);
-    setSortDir('asc');
-    setFilter(null);
+    // Restore this tab's saved view state (search box / sort column /
+    // advanced filter) rather than blank it. This state is scoped PER TAB —
+    // a sort by "depth" on Internal makes no sense on Response Codes, a filter
+    // clause on `images_missing_alt` is meaningless outside Images — so it
+    // must never leak across tabs (the 2026-06-01 bug). Per-tab keying is what
+    // prevents that leak; blank-on-switch also threw the state away when the
+    // user merely left and returned to the SAME tab, which is the regression
+    // being fixed here. The cleanup below saves the outgoing tab's state.
+    const saved = useAppStore.getState().tabView[activeTab];
+    setSearch(saved?.search ?? '');
+    setSortBy(saved?.sortBy);
+    setSortDir(saved?.sortDir ?? 'asc');
+    setFilter(saved?.filter ?? null);
    setFilterDialogOpen(false);
    // Pick up the persisted view-mode for the new tab. Cluster mode is
    // duplicates-tab-only — switching to any other tab while it was the
@@ -393,7 +413,13 @@ export function UrlsTab() {
    if (stored === 'tree') setViewMode('tree');
    else if (stored === 'cluster' && activeTab === 'duplicates') setViewMode('cluster');
    else setViewMode('list');
-  }, [activeTab]);
+    // Save the outgoing tab's latest view state on the way out (tab switch or
+    // full unmount into a non-table view). `activeTab` here is captured as the
+    // tab being left; the ref carries its current search / sort / filter.
+    return () => {
+      setTabView(activeTab, viewStateRef.current);
+    };
+  }, [activeTab, setTabView]);

  // Persist viewMode whenever the user toggles it.
  useEffect(() => {
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/mcp-server",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "type": "module",
  "bin": {
@@ -0,0 +1,100 @@
+# CI/CD Integration
+
+Run FreeCrawl's headless CLI as a quality gate in your pipeline — crawl a
+deploy, fail the build on broken links, validate the sitemap, and catch SEO
+regressions between releases.
+
+Two ready-to-adapt templates live next to this file:
+
+| File                                                         | Platform       |
+| ------------------------------------------------------------ | -------------- |
+| [`github-actions-example.yml`](./github-actions-example.yml) | GitHub Actions |
+| [`gitlab-ci-example.yml`](./gitlab-ci-example.yml)           | GitLab CI/CD   |
+
+Both crawl `TARGET_URL`, fail when 4xx/5xx URLs exceed `BROKEN_LINK_THRESHOLD`,
+validate the sitemap (advisory), and upload the crawl as an artifact.
+
+## How the CLI is built in CI
+
+The runner clones the source and builds only the CLI and the packages it
+imports:
+
+```bash
+npm ci --ignore-scripts        # skip the Playwright download — undici crawl needs no Chromium
+npx tsc -b apps/cli            # builds shared-types → db → core → cli via project references
+node apps/cli/dist/index.js <url> [options]
+```
+
+> `npm run build:cli` compiles **only** `apps/cli` and does **not** build the
+> workspace packages the CLI imports, so the crawl would fail to resolve
+> `@freecrawl/core` at runtime. Use `npx tsc -b apps/cli`, which follows the
+> project references and builds the dependency graph in order — while skipping
+> the heavy Electron desktop build. Drop `--ignore-scripts` only if you enable
+> JS rendering via a `--config` file (that path needs Chromium).
+
+## Exit-code contract
+
+Every command returns a CI-friendly exit code, so most gates need no scripting:
+
+| Command                    | `0`                | `1`                                                                   | `2`                      |
+| -------------------------- | ------------------ | --------------------------------------------------------------------- | ------------------------ |
+| `<url>` (crawl)            | clean — no 4xx/5xx | a 4xx/5xx was found, the seed was unreachable, or nothing was fetched | —                        |
+| `compare <before> <after>` | no differences     | at least one diff category is non-zero                                | a project failed to open |
+| `validate-sitemap <url>`   | valid              | validation warnings                                                   | fetch failure            |
+| `audit-robots <url>`       | URL allowed        | URL disallowed                                                        | invalid URL              |
+
+## Gate variants
+
+**Zero-tolerance (no script).** Let the crawl's own exit code fail the step —
+any 4xx/5xx fails the build:
+
+```yaml
+- run: node apps/cli/dist/index.js "$TARGET_URL" --max 5000
+```
+
+**Threshold (what the templates do).** Tolerate up to N broken URLs. Run the
+crawl with `--json` (append `|| true` so the CLI's exit code doesn't pre-empt
+the gate), then count 4xx/5xx from the summary's `byStatus` map:
+
+```bash
+node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --json > summary.json || true
+node -e "
+  const s = require('./summary.json');
+  const by = (s.summary && s.summary.byStatus) || {};
+  let n = 0;
+  for (const [code, count] of Object.entries(by)) if (Number(code) >= 400) n += count;
+  if (n > Number(process.env.BROKEN_LINK_THRESHOLD)) { console.error('too many broken:', n); process.exit(1); }
+"
+```
+
+Reading `byStatus` (a `{ '200': 1234, '404': 5, … }` map) is robust against the
+full dump's column layout and avoids parsing a multi-MB file just to count.
+
+## Regression gating with `compare`
+
+Fail a deploy when SEO metrics regress against a known-good baseline. Keep a
+baseline `.seoproject` (from a previous green run, restored from cache or an
+artifact store), crawl the new deploy into a fresh project, then diff:
+
+```bash
+node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --db new.seoproject
+node apps/cli/dist/index.js compare baseline.seoproject new.seoproject
+# exit 1 → a page changed status / title / meta / h1 / canonical / indexability
+```
+
+`compare` runs the same engine as the desktop Compare dialog and exits `1` on
+any non-zero diff category, so it drops straight into a pipeline gate. Promote
+`new.seoproject` to the new baseline when the pipeline passes.
+
+## Reproducible config
+
+Export your settings once from the desktop app (Settings → Export Settings) and
+commit the JSON, then feed it to every run so CI crawls exactly like your
+desktop audits:
+
+```bash
+node apps/cli/dist/index.js "$TARGET_URL" --config freecrawl.config.json --max 5000
+```
+
+Per-flag overrides still win over the file, so `--max` can differ per run while
+the shared config stays fixed.
@@ -1,13 +1,16 @@
 # FreeCrawl SEO — example GitHub Actions workflow
 #
-# Runs the headless CLI against your site on every push and on a daily
-# schedule, fails the build if more than N broken links are discovered,
-# and uploads the JSON / CSV crawl as a build artifact. Adapt
-# `TARGET_URL` and `BROKEN_LINK_THRESHOLD` to your site.
+# Crawls your site with the headless CLI on every push and on a daily
+# schedule, fails the build when broken links exceed a threshold, validates
+# the sitemap, and uploads the crawl as a build artifact. Adapt TARGET_URL
+# and the thresholds to your site.
 #
-# Save this file as `.github/workflows/freecrawl-audit.yml` in your
-# repo. No FreeCrawl install required on the runner — `npx` pulls the
-# CLI from the source clone.
+# Save this as `.github/workflows/freecrawl-audit.yml` in your own repo.
+# No FreeCrawl install is required on the runner — the workflow clones the
+# source and builds the CLI (plus its workspace packages) from it.
+#
+# See docs/ci/README.md for the CLI exit-code contract, a zero-tolerance
+# (no-script) gate variant, and how to add a regression-compare job.

 name: FreeCrawl SEO Audit

@@ -15,13 +18,13 @@ on:
  push:
    branches: [main]
  schedule:
-    - cron: "0 4 * * *" # 04:00 UTC daily
+    - cron: '0 4 * * *' # 04:00 UTC daily
  workflow_dispatch:

 env:
  TARGET_URL: https://example.com
-  MAX_URLS: "5000"
-  BROKEN_LINK_THRESHOLD: "10"
+  MAX_URLS: '5000'
+  BROKEN_LINK_THRESHOLD: '10'

 jobs:
  audit:
@@ -35,28 +38,49 @@ jobs:

      - uses: actions/setup-node@v4
        with:
-          node-version: "22"
+          node-version: '22'
          cache: npm
          cache-dependency-path: freecrawl/package-lock.json

      - name: Install + build CLI
        working-directory: freecrawl
        run: |
-          npm ci
-          npm run build:cli
+          # --ignore-scripts skips the Playwright browser download; the CLI's
+          # default crawl is undici-based and needs no Chromium. Drop the flag
+          # only if you enable JS rendering through a --config file.
+          npm ci --ignore-scripts
+          # Build the CLI and its workspace deps (shared-types → db → core) in
+          # dependency order via project references. The heavy Electron desktop
+          # build is intentionally skipped — `npm run build:cli` alone would
+          # NOT build the packages the CLI imports and the crawl would fail.
+          npx tsc -b apps/cli

      - name: Crawl ${{ env.TARGET_URL }}
        working-directory: freecrawl
        run: |
+          # --json prints a machine-readable summary to stdout (captured for
+          # the gate); --out writes the full per-URL dump for the artifact.
+          # `|| true` hands pass/fail to the threshold gate below — the CLI
+          # itself exits 1 on ANY 4xx/5xx (see the zero-tolerance variant in
+          # docs/ci/README.md).
          node apps/cli/dist/index.js "$TARGET_URL" \
            --max "$MAX_URLS" \
-            --out "$GITHUB_WORKSPACE/crawl.json"
+            --json \
+            --out "$GITHUB_WORKSPACE/crawl.json" \
+            > "$GITHUB_WORKSPACE/summary.json" || true
+          cat "$GITHUB_WORKSPACE/summary.json"

-      - name: Fail on too many broken links
+      - name: Gate on broken links
        run: |
+          # Count 4xx/5xx from the summary's byStatus map — robust against the
+          # full dump's column layout, and cheap (no multi-MB file parse).
          BROKEN=$(node -e "
-            const r = require('$GITHUB_WORKSPACE/crawl.json');
-            const n = r.filter(u => u.statusCode >= 400 && u.statusCode < 600).length;
+            const s = require('$GITHUB_WORKSPACE/summary.json');
+            const by = (s.summary && s.summary.byStatus) || {};
+            let n = 0;
+            for (const [code, count] of Object.entries(by)) {
+              if (Number(code) >= 400) n += count;
+            }
            console.log(n);
          ")
          echo "Broken-URL count: $BROKEN (threshold: $BROKEN_LINK_THRESHOLD)"
@@ -65,9 +89,16 @@ jobs:
            exit 1
          fi

+      - name: Validate sitemap (non-blocking)
+        working-directory: freecrawl
+        continue-on-error: true
+        run: node apps/cli/dist/index.js validate-sitemap "$TARGET_URL/sitemap.xml"
+
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: freecrawl-audit-${{ github.run_number }}
-          path: crawl.json
+          path: |
+            crawl.json
+            summary.json
          retention-days: 30
@@ -1,17 +1,19 @@
 # FreeCrawl SEO — example GitLab CI/CD pipeline
 #
-# Runs the headless FreeCrawl CLI against your site, fails the
-# pipeline if more than `$BROKEN_LINK_THRESHOLD` URLs are broken, and
-# stores the JSON crawl as a job artifact. Adapt `TARGET_URL` and the
-# threshold to your project.
+# Crawls your site with the headless FreeCrawl CLI, fails the pipeline when
+# broken links exceed BROKEN_LINK_THRESHOLD, validates the sitemap, and stores
+# the crawl as a job artifact. Adapt TARGET_URL and the thresholds.
 #
-# Save this file as `.gitlab-ci.yml` (or include it as a template).
-# Schedule daily runs via GitLab CI/CD → Schedules.
+# Save this as `.gitlab-ci.yml` (or include it as a template). Schedule daily
+# runs via GitLab → CI/CD → Schedules.
+#
+# See docs/ci/README.md for the CLI exit-code contract, a zero-tolerance
+# (no-script) gate variant, and how to add a regression-compare job.

 variables:
-  TARGET_URL: "https://example.com"
-  MAX_URLS: "5000"
-  BROKEN_LINK_THRESHOLD: "10"
+  TARGET_URL: 'https://example.com'
+  MAX_URLS: '5000'
+  BROKEN_LINK_THRESHOLD: '10'

 stages:
  - audit
@@ -24,26 +26,43 @@ freecrawl-audit:
    - apt-get update -qq && apt-get install -y -qq git ca-certificates
    - git clone --depth 1 https://github.com/kemalai/FreeCrawl-SEO-Tool.git freecrawl
    - cd freecrawl
-    - npm ci
-    - npm run build:cli
+    # --ignore-scripts skips the Playwright browser download (the default
+    # undici crawl needs no Chromium). `tsc -b apps/cli` builds the CLI AND
+    # its workspace deps (shared-types → db → core) — `npm run build:cli`
+    # alone would not, and the crawl would fail to resolve its imports.
+    - npm ci --ignore-scripts
+    - npx tsc -b apps/cli
  script:
-    - node apps/cli/dist/index.js "$TARGET_URL"
-        --max "$MAX_URLS"
-        --out "$CI_PROJECT_DIR/crawl.json"
+    # --json → machine-readable summary on stdout (captured for the gate);
+    # --out → full per-URL dump for the artifact. `|| true` lets the gate
+    # below decide pass/fail (the CLI exits 1 on ANY 4xx/5xx on its own).
    - >-
+      node apps/cli/dist/index.js "$TARGET_URL"
+      --max "$MAX_URLS" --json
+      --out "$CI_PROJECT_DIR/crawl.json"
+      > "$CI_PROJECT_DIR/summary.json" || true
+    - cat "$CI_PROJECT_DIR/summary.json"
+    # Count 4xx/5xx from the summary's byStatus map (robust + cheap).
+    - |
      BROKEN=$(node -e "
-        const r = require('$CI_PROJECT_DIR/crawl.json');
-        const n = r.filter(u => u.statusCode >= 400 && u.statusCode < 600).length;
+        const s = require('$CI_PROJECT_DIR/summary.json');
+        const by = (s.summary && s.summary.byStatus) || {};
+        let n = 0;
+        for (const [code, count] of Object.entries(by)) {
+          if (Number(code) >= 400) n += count;
+        }
        console.log(n);
      ")
-    - echo "Broken-URL count $BROKEN (threshold $BROKEN_LINK_THRESHOLD)"
-    - >-
+      echo "Broken-URL count $BROKEN (threshold $BROKEN_LINK_THRESHOLD)"
      if [ "$BROKEN" -gt "$BROKEN_LINK_THRESHOLD" ]; then
        echo "Broken-URL count $BROKEN exceeds threshold $BROKEN_LINK_THRESHOLD" >&2
        exit 1
      fi
+    # Sitemap validation is advisory — never fails the pipeline.
+    - node apps/cli/dist/index.js validate-sitemap "$TARGET_URL/sitemap.xml" || true
  artifacts:
    paths:
      - crawl.json
+      - summary.json
    expire_in: 30 days
    when: always
@@ -1,6 +1,6 @@
 {
  "name": "freecrawl-seo",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "description": "Open-source desktop SEO crawler â€” a free alternative to Screaming Frog",
  "license": "MIT",
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/core",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/db",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",
@@ -993,13 +993,39 @@ export class ProjectDb {
  deleteByDomain(domain: string): { urlsDeleted: number; linksDeleted: number } {
    const target = domain.trim().toLowerCase();
    if (!target) return { urlsDeleted: 0, linksDeleted: 0 };
-    return this.runInTransaction(() => {
-      // Snapshot the URL ids we'll delete so we can fan out into the
-      // dependent tables before the parent rows vanish.
-      const ids = (
-        this.db
-          .prepare(
-            `SELECT id FROM urls
+    // Foreign-key enforcement is suspended for the wipe, mirroring reset().
+    // A FreeCrawl project is usually one crawled site, so "Delete Domain
+    // Data" for that domain touches the same tables at the same magnitude
+    // reset() does (reset measured 17.3 s → 0.31 s with FKs off on a
+    // 6.6K-URL / 820K-link project), and the per-row parent/child checks on
+    // the `urls`/`images` deletes are the dominant cost.
+    //
+    // Safe under the same invariant as reset(): every table carrying a
+    // foreign key to a deleted row is emptied of those rows in the *same*
+    // transaction before the parent is deleted — links / headers /
+    // url_sources / image_usages are cleared for the target url_ids below
+    // (the only four tables with an FK to `urls`), and orphan `images` are
+    // removed with a `NOT IN image_usages` guard (the only FK to `images`).
+    // Nothing relies on ON DELETE CASCADE being reached, so dropping the
+    // check cannot leave a dangling child. The url-keyed audit tables have
+    // no FK at all and are wiped by host match regardless.
+    //
+    // `PRAGMA foreign_keys` is a no-op inside a transaction, so it is
+    // toggled outside one; the restore is in `finally`. Only applied at the
+    // top level (`txDepth === 0`) — every caller is top-level today; were
+    // this ever nested in an outer transaction the pragma would silently do
+    // nothing and the delete would simply revert to its slower FK-on path,
+    // still correct.
+    const wasTopLevel = this.txDepth === 0;
+    if (wasTopLevel) this.db.exec('PRAGMA foreign_keys = OFF');
+    try {
+      return this.runInTransaction(() => {
+        // Snapshot the URL ids we'll delete so we can fan out into the
+        // dependent tables before the parent rows vanish.
+        const ids = (
+          this.db
+            .prepare(
+              `SELECT id FROM urls
              WHERE LOWER(
                SUBSTR(
                  url,
@@ -1011,44 +1037,44 @@ export class ProjectDb {
                  END
                )
              ) = ?`,
-          )
-          .all(target) as { id: number }[]
-      ).map((r) => r.id);
+            )
+            .all(target) as { id: number }[]
+        ).map((r) => r.id);

-      if (ids.length === 0) return { urlsDeleted: 0, linksDeleted: 0 };
+        if (ids.length === 0) return { urlsDeleted: 0, linksDeleted: 0 };

-      // SQLite parameter limit: chunk the IN-list at 500 ids per delete.
-      const CHUNK = 500;
-      let linksDeleted = 0;
-      for (let i = 0; i < ids.length; i += CHUNK) {
-        const slice = ids.slice(i, i + CHUNK);
-        const placeholders = slice.map(() => '?').join(',');
-        // Links can reference these urls as `from_url_id` (page → link
-        // catalogue) or by url string in `to_url`. Wipe both directions.
-        const linkRes = this.db
-          .prepare(`DELETE FROM links WHERE from_url_id IN (${placeholders})`)
-          .run(...slice);
-        linksDeleted += Number(linkRes.changes);
-        this.db
-          .prepare(`DELETE FROM headers WHERE url_id IN (${placeholders})`)
-          .run(...slice);
-        this.db
-          .prepare(`DELETE FROM url_sources WHERE url_id IN (${placeholders})`)
-          .run(...slice);
-        this.db
-          .prepare(`DELETE FROM image_usages WHERE from_url_id IN (${placeholders})`)
-          .run(...slice);
-        this.db
-          .prepare(`DELETE FROM urls_issues WHERE url_id IN (${placeholders})`)
-          .run(...slice);
-      }
+        // SQLite parameter limit: chunk the IN-list at 500 ids per delete.
+        const CHUNK = 500;
+        let linksDeleted = 0;
+        for (let i = 0; i < ids.length; i += CHUNK) {
+          const slice = ids.slice(i, i + CHUNK);
+          const placeholders = slice.map(() => '?').join(',');
+          // Links can reference these urls as `from_url_id` (page → link
+          // catalogue) or by url string in `to_url`. Wipe both directions.
+          const linkRes = this.db
+            .prepare(`DELETE FROM links WHERE from_url_id IN (${placeholders})`)
+            .run(...slice);
+          linksDeleted += Number(linkRes.changes);
+          this.db
+            .prepare(`DELETE FROM headers WHERE url_id IN (${placeholders})`)
+            .run(...slice);
+          this.db
+            .prepare(`DELETE FROM url_sources WHERE url_id IN (${placeholders})`)
+            .run(...slice);
+          this.db
+            .prepare(`DELETE FROM image_usages WHERE from_url_id IN (${placeholders})`)
+            .run(...slice);
+          this.db
+            .prepare(`DELETE FROM urls_issues WHERE url_id IN (${placeholders})`)
+            .run(...slice);
+        }

-      // Wipe `to_url` references in `links` that pointed AT the deleted
-      // URLs by host string. This is a separate pass because `to_url`
-      // is a string column, not a foreign key.
-      this.db
-        .prepare(
-          `DELETE FROM links
+        // Wipe `to_url` references in `links` that pointed AT the deleted
+        // URLs by host string. This is a separate pass because `to_url`
+        // is a string column, not a foreign key.
+        this.db
+          .prepare(
+            `DELETE FROM links
            WHERE LOWER(
              SUBSTR(
                to_url,
@@ -1060,35 +1086,35 @@ export class ProjectDb {
                END
              )
            ) = ?`,
-        )
-        .run(target);
+          )
+          .run(target);

-      // Finally the parent URL rows.
-      let urlsDeleted = 0;
-      for (let i = 0; i < ids.length; i += CHUNK) {
-        const slice = ids.slice(i, i + CHUNK);
-        const placeholders = slice.map(() => '?').join(',');
-        const res = this.db
-          .prepare(`DELETE FROM urls WHERE id IN (${placeholders})`)
-          .run(...slice);
-        urlsDeleted += Number(res.changes);
-      }
+        // Finally the parent URL rows.
+        let urlsDeleted = 0;
+        for (let i = 0; i < ids.length; i += CHUNK) {
+          const slice = ids.slice(i, i + CHUNK);
+          const placeholders = slice.map(() => '?').join(',');
+          const res = this.db
+            .prepare(`DELETE FROM urls WHERE id IN (${placeholders})`)
+            .run(...slice);
+          urlsDeleted += Number(res.changes);
+        }

-      // Orphaned image rows (no remaining `image_usages` referencing
-      // them) are cleaned in a follow-up sweep.
-      this.db.exec(
-        `DELETE FROM images
+        // Orphaned image rows (no remaining `image_usages` referencing
+        // them) are cleaned in a follow-up sweep.
+        this.db.exec(
+          `DELETE FROM images
          WHERE id NOT IN (SELECT DISTINCT image_id FROM image_usages)`,
-      );
+        );

-      // The url-keyed audit tables (PageSpeed, CrUX, Search Console,
-      // Analytics, AI, SEO) have no FK to `urls`, so wipe them by the same
-      // host match — otherwise a GDPR delete leaves stale data that would
-      // re-attach if the URL is re-crawled later. Driven off
-      // `URL_KEYED_AUDIT_TABLES` so a newly-added audit table can never be
-      // silently missed here.
-      const hostMatchSql = (table: string): string =>
-        `DELETE FROM ${table}
+        // The url-keyed audit tables (PageSpeed, CrUX, Search Console,
+        // Analytics, AI, SEO) have no FK to `urls`, so wipe them by the same
+        // host match — otherwise a GDPR delete leaves stale data that would
+        // re-attach if the URL is re-crawled later. Driven off
+        // `URL_KEYED_AUDIT_TABLES` so a newly-added audit table can never be
+        // silently missed here.
+        const hostMatchSql = (table: string): string =>
+          `DELETE FROM ${table}
          WHERE LOWER(
            SUBSTR(
              url,
@@ -1100,12 +1126,15 @@ export class ProjectDb {
              END
            )
          ) = ?`;
-      for (const table of ProjectDb.URL_KEYED_AUDIT_TABLES) {
-        this.db.prepare(hostMatchSql(table)).run(target);
-      }
+        for (const table of ProjectDb.URL_KEYED_AUDIT_TABLES) {
+          this.db.prepare(hostMatchSql(table)).run(target);
+        }

-      return { urlsDeleted, linksDeleted };
-    });
+        return { urlsDeleted, linksDeleted };
+      });
+    } finally {
+      if (wasTopLevel) this.db.exec('PRAGMA foreign_keys = ON');
+    }
  }

  /**
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/shared-types",
-  "version": "0.9.4",
+  "version": "0.9.5",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",