feat: MCP crawl control from Claude Code with live progress polling

README.md

@@ -17,7 +17,7 @@
 
 <br />
 
-A high-performance website crawler built for serious SEO audits. Targets <kbd>1M+ URLs</kbd> on a single machine, with a dense Screaming Frog–style UI, ~80 SEO issue checks, and zero native dependencies.
+A high-performance website crawler built for serious SEO audits. Targets <kbd>1M+ URLs</kbd> on a single machine, with a dense Screaming Frog–style UI, **150+ SEO issue checks**, and zero native dependencies.
 
 </div>
 
@@ -43,15 +43,16 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 <td width="50%" valign="top">
 
 ### 🔍 Analysis
-- **~80 SEO issue checks** across 14 tabs
-- Near-duplicate clustering (SimHash + LSH)
-- Full hreflang validation, sitemap diff
-- OpenGraph / Twitter Card / JSON-LD / AMP
-- SSL/TLS cert audit, security headers, **CORS audit**
+- **150+ SEO issue checks** across 24 top-level tabs
+- Near-duplicate clustering (SimHash + LSH) + exact-duplicate content hash
+- Full hreflang validation (reciprocity, self-ref, inconsistent lang)
+- OpenGraph / Twitter Card / JSON-LD / AMP / **Web App Manifest** parsing
+- **Structured-data validation** — duplicate `@id`, malformed `@type`, missing required props
+- **Accessibility** — `<main>` landmark, skip-link, ARIA roles, heading order
+- SSL/TLS cert chain audit, security headers, **CORS audit**
 - **Active vs passive mixed content** split
 - Readability (Flesch, Flesch–Kincaid, Gunning Fog)
-- Custom CSS + Regex extraction (10 rules)
-- **Custom search** with `/regex/` or literal mode
+- Custom CSS + Regex extraction (10 rules) + `/regex/` literal Custom Search
 
 </td>
 </tr>
@@ -62,23 +63,25 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 - Dense dark theme, virtualized 1M+ row tables
 - Live-streaming rows during crawl
 - Advanced AND/OR search (24 fields × 12 ops)
-- Detail panel with **13 sub-tabs** per URL
+- Detail panel with 13 sub-tabs — **HTTP Headers** has Request/Response toggle + **status-code diagnosis banner** (decodes Cloudflare 52x/530 and 4xx/5xx with header signals)
 - Per-URL **Duplicates** view (cluster siblings)
+- **Live memory monitor** in status bar (RSS / Sys Free / per-URL cost / capacity projection)
+- **Robots.txt syntax validator** in the Robots Tester (typo suggestions, orphan rules, sitemap URL check)
 - URL rewriting (regex + whitelist + live preview)
-- Cytoscape graph + anchor-text word cloud
+- Cytoscape graph + anchor-text word cloud + **Top Words** report
 - Diagnostic popups for DNS/TLS/proxy issues
 
 </td>
 <td valign="top">
 
 ### 📤 Export & Reports
-- **20 reports** (histograms, top/bottom URLs, link positions)
+- **21 reports** — histograms, top/bottom URLs, link positions, top words
 - **Export Dialog** with column picker — CSV or **XLSX** (multi-sheet)
 - Bulk export (22 categorised CSVs)
 - Standalone HTML audit report
 - Sitemap generator (image / hreflang / sharded / gz)
 - Project-vs-project compare diff
-- **MCP server** for AI agents (Claude / any MCP client)
+- **MCP server** for AI agents — **drive crawls live from Claude Code**, poll progress, plus full read-only DB access
 - Webhook on completion + OS notifications
 
 </td>
@@ -223,13 +226,24 @@ npm --workspace apps/desktop run build:linux   # AppImage / .deb
 </details>
 
 <details>
-<summary><b>🤖 MCP server</b> — query your crawl from Claude / any MCP client</summary>
+<summary><b>🤖 MCP server</b> — query AND drive crawls from Claude / any MCP client</summary>
 
 <br />
 
-FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the active `.seoproject` to AI agents over stdio. Read-only — runs alongside the desktop app without contention (SQLite WAL).
+FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the active `.seoproject` to AI agents over stdio. Two capabilities in one server:
 
-**Tools**: `get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail`, `list_projects`, `set_project`, `current_project`.
+1. **Read-only data access** to the SQLite project — runs alongside the desktop app without contention (WAL allows concurrent readers).
+2. **Live crawl control** — when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to `<userData>/mcp-bridge.json` on app launch).
+
+**14 tools**:
+
+| Group | Tools |
+| :--- | :--- |
+| 📊 **Data queries** (always available) | `get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail` |
+| 📁 **Project management** | `list_projects`, `set_project`, `current_project` |
+| 🕷 **Crawl control** (desktop must be open) | `start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `get_crawl_progress`, `get_desktop_project` |
+
+`start_crawl` accepts a `startUrl` plus optional whitelisted overrides (scope, maxDepth, maxUrls, maxConcurrency, maxRps, crawlDelayMs, requestTimeoutMs, respectRobotsTxt, followRedirects, crawlExternal, userAgent, include/excludePatterns) — anything you don't override keeps the desktop user's saved value. Crawls launched via MCP go through the **same code path** as the UI's Start button, so progress shows up in the desktop app live as the agent drives it.
 
 **1. Build it once:**
 
@@ -291,9 +305,11 @@ The server speaks newline-delimited JSON-RPC 2.0 — point any MCP-compatible cl
 
 **3. Try it.** Ask your agent things like:
 
+> *"Crawl https://example.com with maxDepth 3 and watch the progress."*
 > *"Show the 10 URLs with the longest response time in my last crawl."*
 > *"What are the top 5 issue categories with the most affected pages?"*
 > *"List every URL with a missing meta description."*
+> *"Pause the running crawl, then resume it once I've checked the first 1000 URLs."*
 
 **Pointing at a non-default project:**
 
@@ -401,7 +417,7 @@ graph LR
 ## 📈 Status
 
 > [!NOTE]
-> **Active development.** All 14 analysis tabs, advanced search, ~80 issue categories, sitemap export variants, JSON / CSV / XLSX / HTML reports, list mode, custom extraction, near-duplicate detection (per-URL Duplicates view), hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, MCP server, OS notifications, robots.txt tester, URL rewriting + preview, in-app logs, and diagnostic popups are working. Live-streaming UX with **first row in ~1 s**.
+> **Active development.** All 24 analysis tabs, advanced search, 150+ issue categories, sitemap export variants, JSON / CSV / XLSX / HTML reports, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, **MCP server with crawl control + live progress**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, in-app logs, and diagnostic popups are working. Live-streaming UX with **first row in ~1 s**.
 >
 > **Upcoming:** plugin system, JavaScript rendering, log analyzer, PageSpeed API integration.
 

apps/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/cli",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "type": "module",
   "bin": {

apps/desktop/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/desktop",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "description": "FreeCrawl SEO — desktop application",
   "main": "./out/main/index.js",

apps/desktop/src/main/index.ts

@@ -110,6 +110,11 @@ import { dbWriterPool } from './db-writer-pool.js';
 import { freezeWatchdog } from './freeze-watchdog.js';
 import { parserPool } from './parser-pool.js';
 import { parseHtml as inlineParseHtml } from '@freecrawl/core';
+import {
+  startMcpBridge,
+  stopMcpBridge,
+  type McpStartCrawlInput,
+} from './mcp-bridge.js';
 import type { ProjectDb as ProjectDbType } from '@freecrawl/db';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
@@ -123,6 +128,22 @@ let activeCrawler: Crawler | null = null;
  * had set, without re-prompting. Hydrated lazily from `project_meta`
  * on first DB open. */
 let lastCrawlConfig: CrawlConfig | null = null;
+/** Latest `CrawlProgress` event captured from the active Crawler.
+ * Exposed to the MCP bridge so external agents (Claude Code, etc.)
+ * can poll live crawl state without a renderer subscription. Null
+ * when no crawl has run since app launch. */
+let latestProgress: CrawlProgress | null = null;
+/** Handle to the in-process crawl control primitives. Populated by
+ * `registerIpc()` so the MCP bridge can drive Start/Stop/Pause/Resume
+ * through the same code path as the renderer's IPC handlers — without
+ * having to know about the closure-scoped helpers. Null before
+ * `registerIpc()` runs. */
+let crawlController: {
+  launchCrawl: (config: CrawlConfig) => void;
+  stopCrawl: () => void;
+  pauseCrawl: () => void;
+  resumeCrawl: () => void;
+} | null = null;
 /** In-memory snapshot of the previous session's checkpoint, captured
  * before `db.reset()` wipes it. Cleared once the user accepts or
  * discards the recovery prompt. */
@@ -1689,6 +1710,7 @@ function registerIpc(): void {
   function attachCrawlerListeners(crawler: Crawler): void {
     crawler.on('progress', (p: CrawlProgress) => {
       if (activeCrawler !== crawler) return;
+      latestProgress = p;
       mainWindow?.webContents.send(IPC.crawlProgress, p);
       freezeWatchdog.updateCounters({
         crawled: p.crawled,
@@ -1739,7 +1761,7 @@ function registerIpc(): void {
     });
   }
 
-  ipcMain.handle(IPC.crawlStart, (_e, config: CrawlConfig) => {
+  function launchCrawl(config: CrawlConfig): void {
     if (activeCrawler) {
       activeCrawler.stop();
       logger.log('info', 'crawler', 'Stopped previous crawl before starting a new one');
@@ -1774,6 +1796,10 @@ function registerIpc(): void {
     activeCrawler = crawler;
     attachCrawlerListeners(crawler);
     void crawler.start();
+  }
+
+  ipcMain.handle(IPC.crawlStart, (_e, config: CrawlConfig) => {
+    launchCrawl(config);
   });
 
   ipcMain.handle(IPC.crawlStop, () => {
@@ -1788,6 +1814,16 @@ function registerIpc(): void {
     activeCrawler?.resume();
   });
 
+  // Expose the same primitives to the MCP bridge so external agents
+  // (Claude Code, etc.) can drive Start/Stop/Pause/Resume through the
+  // exact same code path the renderer does.
+  crawlController = {
+    launchCrawl,
+    stopCrawl: () => activeCrawler?.stop(),
+    pauseCrawl: () => activeCrawler?.pause(),
+    resumeCrawl: () => activeCrawler?.resume(),
+  };
+
   ipcMain.handle(IPC.crawlClear, async () => {
     const t0 = Date.now();
     activeCrawler?.stop();
@@ -2796,6 +2832,67 @@ void app.whenReady().then(() => {
   logger.log('info', 'main', `App ready — version ${app.getVersion()}`);
   freezeWatchdog.setMainOp('idle');
 
+  // Open the localhost MCP bridge so Claude Code / Claude Desktop can
+  // drive crawls + poll progress against this running instance. Bind
+  // failure is logged but non-fatal: the desktop app keeps working,
+  // only MCP-driven crawl control is disabled.
+  try {
+    const bridge = startMcpBridge({
+      userDataDir: app.getPath('userData'),
+      startCrawl: async (input: McpStartCrawlInput) => {
+        if (!crawlController) {
+          return { ok: false as const, error: 'IPC handlers not registered yet — try again in a moment.' };
+        }
+        // Layer the supplied overrides on top of the last-used config
+        // (so a `start_crawl` with just a `startUrl` keeps the user's
+        // saved scope/threads/RPS). Fall back to factory defaults when
+        // there is no saved config.
+        const base: CrawlConfig =
+          lastCrawlConfig !== null ? lastCrawlConfig : { ...DEFAULT_CRAWL_CONFIG };
+        const overrides = input.configOverrides ?? {};
+        const resolved: CrawlConfig = {
+          ...base,
+          ...overrides,
+          startUrl: (input.startUrl ?? base.startUrl ?? '').trim(),
+        };
+        if (!resolved.startUrl) {
+          return {
+            ok: false as const,
+            error: 'startUrl is empty — pass it in the request or run a crawl from the desktop UI at least once so the bridge has a saved default.',
+          };
+        }
+        crawlController.launchCrawl(resolved);
+        return { ok: true as const, config: resolved };
+      },
+      stopCrawl: () => crawlController?.stopCrawl(),
+      pauseCrawl: () => crawlController?.pauseCrawl(),
+      resumeCrawl: () => crawlController?.resumeCrawl(),
+      getProgress: () => latestProgress,
+      getProjectPath: () => currentProjectPath || null,
+      getUrlCount: () => {
+        try {
+          return getDb().countUrls();
+        } catch {
+          return 0;
+        }
+      },
+      log: (level, msg) => logger.log(level, 'mcp-bridge', msg),
+    });
+    if (bridge) {
+      logger.log(
+        'info',
+        'mcp-bridge',
+        `MCP bridge ready on 127.0.0.1:${bridge.port}. Discovery file written to userData/mcp-bridge.json.`,
+      );
+    }
+  } catch (err) {
+    logger.log(
+      'warn',
+      'mcp-bridge',
+      `MCP bridge failed to start: ${err instanceof Error ? err.message : String(err)} — MCP crawl control disabled this session.`,
+    );
+  }
+
   // One-shot silent update check 8 s after launch. Ignores network /
   // up-to-date outcomes (no popup), only opens the Update Available
   // dialog when there is a newer release that hasn't been dismissed.
@@ -2812,6 +2909,13 @@ void app.whenReady().then(() => {
 
 app.on('window-all-closed', () => {
   activeCrawler?.stop();
+  // Tear down the MCP bridge first so any in-flight HTTP calls fail
+  // fast rather than racing with the DB shutdown that follows.
+  try {
+    stopMcpBridge();
+  } catch {
+    /* best-effort */
+  }
   freezeWatchdog.setMainOp('shutdown');
   void freezeWatchdog.terminate();
   void parserPool.terminate();

apps/desktop/src/main/mcp-bridge.ts

@@ -0,0 +1,346 @@
+/**
+ * Localhost HTTP bridge between the desktop app and the FreeCrawl MCP
+ * server (`@freecrawl/mcp-server`). Lets an MCP client — typically
+ * Claude Code or Claude Desktop — drive the desktop app's Crawler:
+ * start a crawl, pause/resume/stop it, and poll live progress.
+ *
+ *   Claude Code  ↔  MCP server  ↔  http://127.0.0.1:<rand>/v1/...
+ *                                  ↑
+ *                                  desktop main process
+ *
+ * Design notes:
+ *
+ *   - Bound to `127.0.0.1` only. No external network exposure.
+ *   - Auth via a `Bearer <token>` header. The token is 32 random
+ *     bytes hex, written to `userData/mcp-bridge.json` on app launch
+ *     and removed on app quit. The file lives in the user's profile
+ *     directory whose default ACL already restricts read access to
+ *     the user (Windows %APPDATA%, macOS ~/Library, Linux ~/.config),
+ *     and we additionally `chmod 600` on POSIX as defence-in-depth.
+ *   - Ephemeral random port — `listen(0)` lets the OS pick. The
+ *     port + token go into the discovery file so the MCP server can
+ *     find them.
+ *   - All routes return JSON. Errors carry an `error` field; on
+ *     success the relevant data is at the top level.
+ *   - The bridge is read-mostly (`GET /v1/crawl/progress`) plus a
+ *     handful of POST control endpoints. No long-poll / SSE — MCP
+ *     clients poll progress when they want live updates.
+ *
+ * Endpoints (all under `/v1/`):
+ *
+ *   POST /v1/crawl/start    — body: { startUrl?, configOverrides? }
+ *   POST /v1/crawl/stop
+ *   POST /v1/crawl/pause
+ *   POST /v1/crawl/resume
+ *   GET  /v1/crawl/progress
+ *   GET  /v1/project        — { projectPath, urlsCrawled }
+ *
+ * Caller signs every request:
+ *   `Authorization: Bearer <token>`
+ */
+
+import { createServer, type Server, type IncomingMessage, type ServerResponse } from 'node:http';
+import { randomBytes } from 'node:crypto';
+import { writeFileSync, unlinkSync, chmodSync, existsSync } from 'node:fs';
+import { join } from 'node:path';
+import type { CrawlConfig, CrawlProgress } from '@freecrawl/shared-types';
+
+/**
+ * Functions the bridge needs from the main process. Wired by the
+ * caller (`index.ts`) at startup — the bridge module stays
+ * decoupled from the global desktop state.
+ */
+export interface McpBridgeDeps {
+  /** Discovery + token directory — usually `app.getPath('userData')`. */
+  userDataDir: string;
+  /**
+   * Kick off a crawl. The caller resolves any defaults/overrides and
+   * dispatches the existing crawler IPC pipeline. Returns synchronously
+   * once the crawl is launched (not when it finishes).
+   */
+  startCrawl: (input: McpStartCrawlInput) => Promise<{ ok: true; config: CrawlConfig } | { ok: false; error: string }>;
+  stopCrawl: () => void;
+  pauseCrawl: () => void;
+  resumeCrawl: () => void;
+  /**
+   * Snapshot of the latest progress event the Crawler emitted. Null
+   * when no crawl has ever run in this session. The bridge passes it
+   * verbatim to MCP — `running` / `paused` fields are authoritative.
+   */
+  getProgress: () => CrawlProgress | null;
+  /** Absolute path to the active `.seoproject`, or null if none. */
+  getProjectPath: () => string | null;
+  /** Row count for the active project — useful for confirming the
+   * MCP-side read-only view is targetting the same DB. */
+  getUrlCount: () => number;
+  /** Optional logger (info / warn / error). When omitted, the bridge
+   *  swallows internal trace; only HTTP-visible errors propagate. */
+  log?: (level: 'info' | 'warn' | 'error', msg: string) => void;
+}
+
+/**
+ * Fields the MCP server may pass when starting a crawl. Everything is
+ * optional except an effective startUrl must exist — either supplied
+ * here or already saved in the last-used config. The caller layers
+ * these on top of the last-used CrawlConfig and dispatches.
+ */
+export interface McpStartCrawlInput {
+  startUrl?: string;
+  /** Whitelisted CrawlConfig field overrides — keeping the contract
+   *  narrow so MCP can't smuggle in fields we'd rather validate. */
+  configOverrides?: {
+    scope?: CrawlConfig['scope'];
+    maxDepth?: number;
+    maxUrls?: number;
+    maxConcurrency?: number;
+    maxRps?: number;
+    crawlDelayMs?: number;
+    requestTimeoutMs?: number;
+    respectRobotsTxt?: boolean;
+    followRedirects?: boolean;
+    crawlExternal?: boolean;
+    userAgent?: string;
+    includePatterns?: string[];
+    excludePatterns?: string[];
+  };
+}
+
+interface DiscoveryFile {
+  port: number;
+  token: string;
+  pid: number;
+  version: 1;
+}
+
+const DISCOVERY_FILENAME = 'mcp-bridge.json';
+
+let server: Server | null = null;
+let discoveryPath: string | null = null;
+
+/**
+ * Open the localhost HTTP bridge and write the discovery file. Returns
+ * `null` if it could not start (already running, permission denied,
+ * etc.) — non-fatal: the desktop app keeps running, MCP just won't be
+ * able to drive a crawl.
+ */
+export function startMcpBridge(deps: McpBridgeDeps): { port: number; token: string } | null {
+  if (server) return null; // already started
+
+  const token = randomBytes(32).toString('hex');
+  discoveryPath = join(deps.userDataDir, DISCOVERY_FILENAME);
+
+  const log = deps.log ?? (() => undefined);
+
+  const handler = (req: IncomingMessage, res: ServerResponse): void => {
+    void handleRequest(req, res, token, deps).catch((err) => {
+      log('error', `mcp-bridge request error: ${err instanceof Error ? err.message : String(err)}`);
+      if (!res.headersSent) {
+        res.writeHead(500, { 'Content-Type': 'application/json' });
+        res.end(JSON.stringify({ error: 'internal-error' }));
+      }
+    });
+  };
+
+  try {
+    server = createServer(handler);
+    // listen(0, '127.0.0.1') — random ephemeral port on loopback only.
+    server.listen(0, '127.0.0.1');
+  } catch (err) {
+    log('error', `mcp-bridge listen failed: ${err instanceof Error ? err.message : String(err)}`);
+    server = null;
+    return null;
+  }
+
+  const addr = server.address();
+  if (!addr || typeof addr === 'string') {
+    log('error', 'mcp-bridge: invalid address after listen');
+    try {
+      server.close();
+    } catch {
+      /* ignore */
+    }
+    server = null;
+    return null;
+  }
+  const port = addr.port;
+
+  // Write the discovery file (port + token + pid). The MCP server
+  // reads this to find us. We `chmod 600` on POSIX so other local
+  // users can't sniff the token — Windows ACLs already restrict
+  // the userData directory to the current user.
+  const payload: DiscoveryFile = { port, token, pid: process.pid, version: 1 };
+  try {
+    writeFileSync(discoveryPath, JSON.stringify(payload), { encoding: 'utf8' });
+    if (process.platform !== 'win32') {
+      try {
+        chmodSync(discoveryPath, 0o600);
+      } catch {
+        /* best-effort — already restrictive enough on most setups */
+      }
+    }
+  } catch (err) {
+    log('error', `mcp-bridge: failed to write discovery file: ${err instanceof Error ? err.message : String(err)}`);
+    try {
+      server.close();
+    } catch {
+      /* ignore */
+    }
+    server = null;
+    discoveryPath = null;
+    return null;
+  }
+
+  log('info', `mcp-bridge listening on 127.0.0.1:${port}`);
+  return { port, token };
+}
+
+/**
+ * Shut down the bridge and remove the discovery file. Safe to call
+ * multiple times. Called on `before-quit`.
+ */
+export function stopMcpBridge(): void {
+  if (server) {
+    try {
+      server.close();
+    } catch {
+      /* ignore */
+    }
+    server = null;
+  }
+  if (discoveryPath && existsSync(discoveryPath)) {
+    try {
+      unlinkSync(discoveryPath);
+    } catch {
+      /* ignore */
+    }
+  }
+  discoveryPath = null;
+}
+
+async function readJsonBody(req: IncomingMessage): Promise<unknown> {
+  return new Promise((resolve, reject) => {
+    const chunks: Buffer[] = [];
+    let total = 0;
+    // 64 KB cap — config payloads are tiny; this defends against
+    // accidental large bodies.
+    const MAX_BODY = 64 * 1024;
+    req.on('data', (chunk: Buffer) => {
+      total += chunk.length;
+      if (total > MAX_BODY) {
+        req.destroy();
+        reject(new Error('request body too large'));
+        return;
+      }
+      chunks.push(chunk);
+    });
+    req.on('end', () => {
+      const raw = Buffer.concat(chunks).toString('utf8').trim();
+      if (raw === '') {
+        resolve({});
+        return;
+      }
+      try {
+        resolve(JSON.parse(raw));
+      } catch (err) {
+        reject(new Error(`invalid JSON body: ${err instanceof Error ? err.message : String(err)}`));
+      }
+    });
+    req.on('error', (err) => reject(err));
+  });
+}
+
+function send(res: ServerResponse, status: number, payload: unknown): void {
+  if (res.headersSent) return;
+  res.writeHead(status, { 'Content-Type': 'application/json' });
+  res.end(JSON.stringify(payload));
+}
+
+async function handleRequest(
+  req: IncomingMessage,
+  res: ServerResponse,
+  token: string,
+  deps: McpBridgeDeps,
+): Promise<void> {
+  // 1) Method + path gate — only the small allow-listed routes.
+  const url = req.url ?? '';
+  const method = req.method ?? 'GET';
+  // Strip query string defensively even though we don't use one.
+  const path = url.split('?')[0] ?? url;
+
+  // 2) Auth — Bearer token must match exactly. Constant-time-ish
+  //    comparison is overkill for a single random secret on
+  //    loopback but cheap to do.
+  const authHdr = req.headers['authorization'];
+  if (typeof authHdr !== 'string' || !authHdr.startsWith('Bearer ')) {
+    send(res, 401, { error: 'missing-bearer-token' });
+    return;
+  }
+  const presented = authHdr.slice('Bearer '.length).trim();
+  if (presented.length !== token.length || !timingSafeEqual(presented, token)) {
+    send(res, 401, { error: 'invalid-token' });
+    return;
+  }
+
+  // 3) Route — flat switch; the surface is small enough that we don't
+  //    need a router library.
+  if (method === 'GET' && path === '/v1/crawl/progress') {
+    const progress = deps.getProgress();
+    send(res, 200, { progress });
+    return;
+  }
+  if (method === 'GET' && path === '/v1/project') {
+    send(res, 200, {
+      projectPath: deps.getProjectPath(),
+      urlsCrawled: deps.getUrlCount(),
+    });
+    return;
+  }
+  if (method === 'POST' && path === '/v1/crawl/start') {
+    let body: unknown;
+    try {
+      body = await readJsonBody(req);
+    } catch (err) {
+      send(res, 400, { error: err instanceof Error ? err.message : 'bad-body' });
+      return;
+    }
+    const input = body as McpStartCrawlInput;
+    const result = await deps.startCrawl(input);
+    if (!result.ok) {
+      send(res, 400, { error: result.error });
+      return;
+    }
+    send(res, 200, { ok: true, config: result.config });
+    return;
+  }
+  if (method === 'POST' && path === '/v1/crawl/stop') {
+    deps.stopCrawl();
+    send(res, 200, { ok: true });
+    return;
+  }
+  if (method === 'POST' && path === '/v1/crawl/pause') {
+    deps.pauseCrawl();
+    send(res, 200, { ok: true });
+    return;
+  }
+  if (method === 'POST' && path === '/v1/crawl/resume') {
+    deps.resumeCrawl();
+    send(res, 200, { ok: true });
+    return;
+  }
+
+  send(res, 404, { error: 'not-found' });
+}
+
+/**
+ * Constant-time string equality. Avoids a (theoretical) timing oracle
+ * leaking the secret one byte at a time — overkill for a localhost
+ * secret but trivial to add. Both inputs are already known to be equal
+ * length at this point.
+ */
+function timingSafeEqual(a: string, b: string): boolean {
+  let diff = 0;
+  for (let i = 0; i < a.length; i++) {
+    diff |= a.charCodeAt(i) ^ b.charCodeAt(i);
+  }
+  return diff === 0;
+}

apps/mcp-server/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/mcp-server",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "type": "module",
   "bin": {

apps/mcp-server/src/desktop-bridge.ts

@@ -0,0 +1,139 @@
+/**
+ * Client-side counterpart to `apps/desktop/src/main/mcp-bridge.ts`.
+ * Lets the MCP server drive the desktop app's Crawler — start a
+ * crawl, pause/resume/stop, poll progress — through the localhost
+ * HTTP bridge.
+ *
+ * Discovery: the desktop app writes
+ * `<userData>/mcp-bridge.json` containing `{ port, token, pid }`
+ * at launch. This module reads that file on every call so an app
+ * restart (new port + token) transparently re-targets — no need to
+ * restart the MCP server.
+ *
+ * Auth: every request carries `Authorization: Bearer <token>`. The
+ * desktop server binds to `127.0.0.1` only and the token file lives
+ * in the user's profile directory, so a process running as another
+ * OS user can neither reach the socket nor read the token.
+ */
+
+import { readFileSync, existsSync } from 'node:fs';
+import { join } from 'node:path';
+import { userDataDir } from './project-resolver.js';
+
+const DISCOVERY_FILENAME = 'mcp-bridge.json';
+
+interface DiscoveryFile {
+  port: number;
+  token: string;
+  pid: number;
+  version: number;
+}
+
+export class BridgeUnavailableError extends Error {
+  constructor(reason: string) {
+    super(
+      `FreeCrawl desktop app does not appear to be running (${reason}). ` +
+        'Open the desktop app first — it writes the bridge discovery file ' +
+        '(<userData>/mcp-bridge.json) on launch.',
+    );
+    this.name = 'BridgeUnavailableError';
+  }
+}
+
+/**
+ * Read the discovery file the desktop app writes at launch. Returns
+ * null when the file is missing (desktop not running) or malformed
+ * (treat-as-missing — the next launch will rewrite it).
+ */
+function readDiscovery(): DiscoveryFile | null {
+  const path = join(userDataDir(), DISCOVERY_FILENAME);
+  if (!existsSync(path)) return null;
+  try {
+    const raw = readFileSync(path, 'utf8');
+    const parsed = JSON.parse(raw) as Partial<DiscoveryFile>;
+    if (
+      typeof parsed.port !== 'number' ||
+      typeof parsed.token !== 'string' ||
+      parsed.token.length < 16
+    ) {
+      return null;
+    }
+    return parsed as DiscoveryFile;
+  } catch {
+    return null;
+  }
+}
+
+/**
+ * Issue a request against the desktop bridge. Throws
+ * `BridgeUnavailableError` when the discovery file is missing or the
+ * socket refuses; throws a generic Error with the server-supplied
+ * `error` field for HTTP-level failures (400/401/404/500). Returns
+ * the parsed JSON body on success.
+ */
+export async function bridgeRequest<T>(
+  method: 'GET' | 'POST',
+  path: string,
+  body?: Record<string, unknown>,
+): Promise<T> {
+  const disc = readDiscovery();
+  if (!disc) {
+    throw new BridgeUnavailableError(
+      'discovery file <userData>/mcp-bridge.json not found',
+    );
+  }
+  const url = `http://127.0.0.1:${disc.port}${path}`;
+  let res: Response;
+  try {
+    res = await fetch(url, {
+      method,
+      headers: {
+        Authorization: `Bearer ${disc.token}`,
+        ...(body !== undefined ? { 'Content-Type': 'application/json' } : {}),
+      },
+      body: body !== undefined ? JSON.stringify(body) : undefined,
+    });
+  } catch (err) {
+    // ECONNREFUSED / network error → desktop crashed or restarted
+    // between writing the discovery file and our request. Treat as
+    // unavailable; the user fixes it by relaunching the desktop app.
+    throw new BridgeUnavailableError(
+      `localhost:${disc.port} unreachable (${err instanceof Error ? err.message : String(err)})`,
+    );
+  }
+
+  let parsed: unknown = null;
+  try {
+    parsed = await res.json();
+  } catch {
+    // Empty / non-JSON body — most error paths produce JSON, so this
+    // is unusual but recoverable.
+    parsed = null;
+  }
+
+  if (!res.ok) {
+    const errMsg =
+      parsed && typeof parsed === 'object' && 'error' in parsed
+        ? String((parsed as { error: unknown }).error)
+        : `HTTP ${res.status}`;
+    if (res.status === 401) {
+      throw new Error(
+        `Desktop bridge auth failed (${errMsg}). The token may be stale — ` +
+          'this happens when the desktop app was restarted between MCP calls. Retry.',
+      );
+    }
+    throw new Error(`Desktop bridge: ${errMsg}`);
+  }
+
+  return parsed as T;
+}
+
+/**
+ * Convenience: report whether the bridge looks reachable without
+ * making a network call. Used by tools that want to surface a
+ * friendly "open the desktop app" hint up-front before attempting an
+ * action.
+ */
+export function bridgeDiscoveryPresent(): boolean {
+  return readDiscovery() !== null;
+}

apps/mcp-server/src/index.ts

@@ -152,10 +152,27 @@ server.on('tools/call', async (raw) => {
   const tool = tools.find((t) => t.name === name);
   if (!tool) throw new Error(`Unknown tool: ${name}`);
 
-  const sess = getSession();
+  // Most tools need the project DB (read-only). Crawl-control tools
+  // proxy to the desktop bridge over HTTP and don't touch the DB, so
+  // we skip session init for them — this lets `start_crawl` / etc.
+  // work on a fresh install where the local `.seoproject` file doesn't
+  // exist yet (only the desktop app has it).
+  const needsDb = tool.requiresDb !== false;
+  // `db` is unused when `needsDb` is false; we pass the session's DB
+  // when available, else a never-used null cast — type system lets us
+  // do this because the handler ignores `db` for those tools.
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  let db: any = null;
+  if (needsDb) {
+    db = getSession().db;
+  } else if (session) {
+    // Re-use an already-open session if there is one (saves opening
+    // the DB just to satisfy the param), but don't force-open.
+    db = session.db;
+  }
   let result: unknown;
   try {
-    result = await tool.handler(args, sess.db);
+    result = await tool.handler(args, db);
   } catch (err) {
     const msg = err instanceof Error ? err.message : String(err);
     return {

apps/mcp-server/src/project-resolver.ts

@@ -20,28 +20,35 @@ import fs from 'node:fs';
 const PRODUCT_NAME = '@freecrawl/desktop';
 const PROJECT_FILENAME = 'default.seoproject';
 
+/**
+ * Resolve the Electron `userData` directory the desktop app uses. The
+ * MCP bridge's discovery file (`mcp-bridge.json`) lives here too, so
+ * the bridge client re-uses this helper rather than duplicating the
+ * per-platform logic.
+ */
+export function userDataDir(): string {
+  const home = os.homedir();
+  if (process.platform === 'win32') {
+    return path.join(
+      process.env.APPDATA ?? path.join(home, 'AppData', 'Roaming'),
+      PRODUCT_NAME,
+    );
+  }
+  if (process.platform === 'darwin') {
+    return path.join(home, 'Library', 'Application Support', PRODUCT_NAME);
+  }
+  return path.join(
+    process.env.XDG_CONFIG_HOME ?? path.join(home, '.config'),
+    PRODUCT_NAME,
+  );
+}
+
 export function defaultProjectPath(): string {
   // Explicit override always wins — handy for headless testing or
   // pointing at a non-default project.
   const override = process.env.FREECRAWL_PROJECT;
   if (override && override.trim().length > 0) return override.trim();
-
-  const home = os.homedir();
-  let userData: string;
-  if (process.platform === 'win32') {
-    userData = path.join(
-      process.env.APPDATA ?? path.join(home, 'AppData', 'Roaming'),
-      PRODUCT_NAME,
-    );
-  } else if (process.platform === 'darwin') {
-    userData = path.join(home, 'Library', 'Application Support', PRODUCT_NAME);
-  } else {
-    userData = path.join(
-      process.env.XDG_CONFIG_HOME ?? path.join(home, '.config'),
-      PRODUCT_NAME,
-    );
-  }
-  return path.join(userData, 'projects', PROJECT_FILENAME);
+  return path.join(userDataDir(), 'projects', PROJECT_FILENAME);
 }
 
 export function listProjectFiles(): string[] {

apps/mcp-server/src/tools.ts

@@ -8,12 +8,22 @@
 
 import type { ProjectDb } from '@freecrawl/db';
 import type { UrlCategory } from '@freecrawl/shared-types';
+import { bridgeRequest } from './desktop-bridge.js';
 
 export interface Tool {
   name: string;
   description: string;
   inputSchema: Record<string, unknown>;
   handler: (args: Record<string, unknown>, db: ProjectDb) => unknown | Promise<unknown>;
+  /**
+   * Whether this tool needs the project SQLite handle (read-only).
+   * Defaults to true. Crawl-control tools that proxy to the desktop
+   * bridge over HTTP set this to false so they work even when no
+   * `.seoproject` file exists locally yet (e.g. fresh install — only
+   * the desktop app has the file). Skipping the session init also
+   * avoids opening the DB unnecessarily for one-shot control calls.
+   */
+  requiresDb?: boolean;
 }
 
 const URL_CATEGORY_VALUES: UrlCategory[] = [
@@ -231,6 +241,104 @@ export function buildTools(): Tool[] {
       },
     },
 
+    // ----- Crawl-control tools (talk to the running desktop app
+    // over the localhost bridge — no-op when the desktop is closed)
+    {
+      requiresDb: false,
+      name: 'start_crawl',
+      description:
+        'Start a crawl in the running FreeCrawl desktop app. The desktop app must be open — this drives the SAME crawler the UI uses, so progress shows up in the app as it runs. `startUrl` is required unless the user has already run a crawl in this session (the last-used start URL is reused). Other config overrides are optional and layered on top of the desktop\'s saved settings.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          startUrl: {
+            type: 'string',
+            description: 'Seed URL to crawl. Required unless the desktop already has a saved last-used crawl config.',
+          },
+          configOverrides: {
+            type: 'object',
+            description: 'Optional whitelisted CrawlConfig overrides. Anything not listed here keeps the desktop user\'s saved value.',
+            properties: {
+              scope: {
+                type: 'string',
+                enum: ['subdomain', 'subfolder', 'all-subdomains', 'exact-url'],
+              },
+              maxDepth: { type: 'integer', minimum: 0 },
+              maxUrls: { type: 'integer', minimum: 1 },
+              maxConcurrency: { type: 'integer', minimum: 1, maximum: 200 },
+              maxRps: { type: 'integer', minimum: 1 },
+              crawlDelayMs: { type: 'integer', minimum: 0 },
+              requestTimeoutMs: { type: 'integer', minimum: 1000 },
+              respectRobotsTxt: { type: 'boolean' },
+              followRedirects: { type: 'boolean' },
+              crawlExternal: { type: 'boolean' },
+              userAgent: { type: 'string' },
+              includePatterns: { type: 'array', items: { type: 'string' } },
+              excludePatterns: { type: 'array', items: { type: 'string' } },
+            },
+          },
+        },
+      },
+      handler: async (args) => {
+        const startUrl = typeof args.startUrl === 'string' ? args.startUrl : undefined;
+        const configOverrides =
+          args.configOverrides && typeof args.configOverrides === 'object'
+            ? (args.configOverrides as Record<string, unknown>)
+            : undefined;
+        return bridgeRequest<{ ok: true; config: unknown }>('POST', '/v1/crawl/start', {
+          startUrl,
+          configOverrides,
+        });
+      },
+    },
+
+    {
+      requiresDb: false,
+      name: 'stop_crawl',
+      description:
+        'Stop the active crawl in the desktop app immediately. Crawled URLs stay in the project DB; pending URLs are dropped. No-op when no crawl is running.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () => bridgeRequest<{ ok: true }>('POST', '/v1/crawl/stop'),
+    },
+
+    {
+      requiresDb: false,
+      name: 'pause_crawl',
+      description:
+        'Pause the active crawl in the desktop app. The pending queue is preserved — call resume_crawl to continue. No-op when no crawl is running.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () => bridgeRequest<{ ok: true }>('POST', '/v1/crawl/pause'),
+    },
+
+    {
+      requiresDb: false,
+      name: 'resume_crawl',
+      description:
+        'Resume a paused crawl in the desktop app. No-op when the crawl is already running or no crawl is paused.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () => bridgeRequest<{ ok: true }>('POST', '/v1/crawl/resume'),
+    },
+
+    {
+      requiresDb: false,
+      name: 'get_crawl_progress',
+      description:
+        'Snapshot of the current crawl progress in the desktop app: discovered / crawled / pending / failed counts, URLs per second, average response time, elapsed time, and running/paused flags. Returns `progress: null` when no crawl has run this session. Poll this every 1–2 s to watch a crawl live; once `running` flips to false the crawl is done.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () =>
+        bridgeRequest<{ progress: unknown | null }>('GET', '/v1/crawl/progress'),
+    },
+
+    {
+      requiresDb: false,
+      name: 'get_desktop_project',
+      description:
+        'Which `.seoproject` the running desktop app currently has open, plus the row count. Use this to confirm the desktop is running and which project the crawl-control tools will affect — it can differ from the file the MCP read-only tools target via `set_project`.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () =>
+        bridgeRequest<{ projectPath: string | null; urlsCrawled: number }>('GET', '/v1/project'),
+    },
+
     {
       name: 'top_issues',
       description:

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "freecrawl-seo",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "freecrawl-seo",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "license": "MIT",
       "workspaces": [
         "apps/*",
@@ -25,7 +25,7 @@
     },
     "apps/cli": {
       "name": "@freecrawl/cli",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "dependencies": {
         "@freecrawl/core": "*",
         "@freecrawl/db": "*",
@@ -42,7 +42,7 @@
     },
     "apps/desktop": {
       "name": "@freecrawl/desktop",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "dependencies": {
         "@freecrawl/core": "*",
         "@freecrawl/db": "*",
@@ -84,7 +84,7 @@
     },
     "apps/mcp-server": {
       "name": "@freecrawl/mcp-server",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*"
@@ -8732,7 +8732,7 @@
     },
     "packages/core": {
       "name": "@freecrawl/core",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*",
@@ -8758,7 +8758,7 @@
     },
     "packages/db": {
       "name": "@freecrawl/db",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "dependencies": {
         "@freecrawl/shared-types": "*"
       },
@@ -8770,7 +8770,7 @@
     },
     "packages/shared-types": {
       "name": "@freecrawl/shared-types",
-      "version": "0.3.6",
+      "version": "0.3.7",
       "devDependencies": {
         "rimraf": "^6.0.1",
         "typescript": "^5.7.0"

package.json

@@ -1,6 +1,6 @@
 {
   "name": "freecrawl-seo",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "description": "Open-source desktop SEO crawler — a free alternative to Screaming Frog",
   "license": "MIT",

packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/core",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",

packages/db/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/db",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",

packages/shared-types/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/shared-types",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",