feat: SOCKS5/digest/form-login auth, XPath/JSONPath, PDF metadata, severity export

README.md

@@ -29,7 +29,7 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 
 ## ✨ What FreeCrawl Does
 
-FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 30 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), and a 10-rule custom CSS + regex extraction engine with a **live Preview dialog and JSON import / export** for sharing rule sets.
+FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 30 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), and a 10-rule custom extraction engine — **CSS selectors, XPath, regex, and JSONPath (for JSON API responses)** — with a **live Preview dialog and JSON import / export** for sharing rule sets.
 
 The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 16-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window, and a `.seoproject` file association for OS double-click.
 
@@ -330,7 +330,7 @@ npm --version     # 10+
 
 <br />
 
-- **Outbound HTTPS access** to the sites you crawl. Behind a corporate proxy? Set `HTTPS_PROXY=http://your-proxy:port` before launch — undici's `ProxyAgent` routes through it automatically.
+- **Outbound HTTPS access** to the sites you crawl. Behind a corporate proxy? Set `HTTPS_PROXY=http://your-proxy:port` before launch, or enter a proxy URL in **Settings → Network** — HTTP/HTTPS proxies route through undici's `ProxyAgent`, and **SOCKS5 / SOCKS4 proxies** (`socks5://…`, including the `h` / `4a` remote-DNS variants — handy for routing crawls through Tor / a SSH tunnel) are tunnelled automatically.
 - **TLS root certificates**. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set `NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt` — otherwise crawls fail with `UNABLE_TO_GET_ISSUER_CERT_LOCALLY`.
 
 </details>
@@ -388,7 +388,7 @@ graph LR
 ## 📈 Status
 
 > [!NOTE]
-> **Active development.** All 29 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **cross-source orphan detection**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
+> **Active development.** All 29 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, Cytoscape visualization, **Basic / Digest / Bearer + multi-step form-based login**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **cross-source orphan detection**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
 >
 > **Upcoming (V2):** Log file analyzer, Plugin system, Light theme, Multi-window, Code-signing + auto-update.
 

apps/desktop/package.json

@@ -43,6 +43,7 @@
     "react-dom": "^19.2.0",
     "react-i18next": "^17.0.8",
     "react-resizable-panels": "^2.1.9",
+    "undici": "^8.0.0",
     "zustand": "^5.0.2"
   },
   "devDependencies": {

apps/desktop/src/main/index.ts

@@ -217,6 +217,47 @@ let aiCancelRequested = false;
 /** Faz 7 — SEO provider run guard. */
 let seoRunActive = false;
 let seoCancelRequested = false;
+/**
+ * V2 Faz 0.5 Increment 5 — live snapshot of each slow batch-fetch run
+ * (PageSpeed / AI / SEO), polled by the MCP `get_fetch_progress` tool.
+ * The shared run-job helpers below update these whether the run was
+ * triggered from the renderer or the MCP bridge, so an agent can watch a
+ * user-started run too. The renderer keeps consuming its own `*Progress`
+ * IPC events — this snapshot is purely the MCP poll surface.
+ */
+interface FetchRunSnapshot {
+  running: boolean;
+  done: number;
+  total: number;
+  completed: number;
+  failed: number;
+  currentUrl: string | null;
+  cancelled: boolean;
+  startedAt: string | null;
+  finishedAt: string | null;
+  /** Provider id for ai/seo runs; null for pagespeed. */
+  provider: string | null;
+  /** Set when the run threw before finishing. */
+  error: string | null;
+}
+function idleFetchSnapshot(): FetchRunSnapshot {
+  return {
+    running: false,
+    done: 0,
+    total: 0,
+    completed: 0,
+    failed: 0,
+    currentUrl: null,
+    cancelled: false,
+    startedAt: null,
+    finishedAt: null,
+    provider: null,
+    error: null,
+  };
+}
+let pagespeedRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
+let aiRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
+let seoRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
 /** Faz 7 — GSC URL Inspection run guard. */
 let gscInspectRunActive = false;
 let gscInspectCancelRequested = false;
@@ -2362,86 +2403,132 @@ function registerIpc(): void {
     },
   );
 
+  // Shared PSI batch orchestrator — driven by both the renderer's
+  // `pagespeedRun` IPC (which awaits the result) and the MCP bridge's
+  // non-blocking `pagespeed-run` action (which fires-and-forgets and
+  // lets the agent poll `fetch-progress`). Updates `pagespeedRunSnapshot`
+  // throughout so the MCP poll surface reflects either trigger path.
+  async function runPagespeedJob(
+    input: PagespeedRunInput,
+  ): Promise<PagespeedRunResult> {
+    if (pagespeedRunActive) {
+      return { completed: 0, failed: 0, cancelled: true };
+    }
+    const urls = Array.from(
+      new Set(
+        (input.urls ?? []).filter(
+          (u): u is string => typeof u === 'string' && u.length > 0,
+        ),
+      ),
+    );
+    if (urls.length === 0) {
+      pagespeedRunSnapshot = {
+        ...idleFetchSnapshot(),
+        finishedAt: new Date().toISOString(),
+      };
+      return { completed: 0, failed: 0, cancelled: false };
+    }
+    const strategies: PagespeedStrategy[] =
+      input.strategy === 'both' ? ['mobile', 'desktop'] : [input.strategy];
+    const items: PagespeedBatchItem[] = [];
+    for (const url of urls) {
+      for (const strategy of strategies) items.push({ url, strategy });
+    }
+    const apiKeyRaw = resolveCredentials('pagespeed')['apiKey'];
+    const apiKey =
+      apiKeyRaw && apiKeyRaw.trim().length > 0 ? apiKeyRaw.trim() : undefined;
+
+    pagespeedRunActive = true;
+    pagespeedCancelRequested = false;
+    // Set synchronously (before the first await) so the MCP `pagespeed-run`
+    // action can read an accurate snapshot right after kicking us off.
+    pagespeedRunSnapshot = {
+      ...idleFetchSnapshot(),
+      running: true,
+      total: items.length,
+      startedAt: new Date().toISOString(),
+    };
+    // Pin writes to the project that was active when the run started —
+    // a PSI run can last minutes, and resolving `getDb()` per result
+    // would leak audits into a different project if the user opens
+    // one mid-run.
+    const runDb = getDb();
+    logger.log(
+      'info',
+      'pagespeed',
+      `run started — ${items.length} audit(s): ${urls.length} URL(s) × ${strategies.length} strategy${
+        apiKey ? ' (keyed)' : ' (keyless — low rate limit)'
+      }`,
+    );
+    try {
+      const result = await runPagespeedBatch({
+        items,
+        apiKey,
+        isCancelled: () => pagespeedCancelRequested,
+        onResult: (url, strategy, metrics) => {
+          try {
+            runDb.upsertPagespeedResult(url, strategy, metrics);
+          } catch (err) {
+            logger.log(
+              'error',
+              'pagespeed',
+              `DB write failed for ${url}: ${
+                err instanceof Error ? err.message : String(err)
+              }`,
+            );
+          }
+          if (metrics.status === 'ok') pagespeedRunSnapshot.completed++;
+          else pagespeedRunSnapshot.failed++;
+        },
+        onProgress: (done, total, currentUrl) => {
+          pagespeedRunSnapshot.done = done;
+          pagespeedRunSnapshot.total = total;
+          pagespeedRunSnapshot.currentUrl = currentUrl;
+          mainWindow?.webContents.send(IPC.pagespeedProgress, {
+            done,
+            total,
+            currentUrl,
+            running: true,
+          });
+        },
+      });
+      pagespeedRunSnapshot = {
+        ...pagespeedRunSnapshot,
+        running: false,
+        done: result.completed + result.failed,
+        completed: result.completed,
+        failed: result.failed,
+        cancelled: result.cancelled,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+      };
+      mainWindow?.webContents.send(IPC.pagespeedProgress, {
+        done: result.completed + result.failed,
+        total: items.length,
+        currentUrl: null,
+        running: false,
+      });
+      fireDataChanged();
+      return result;
+    } catch (err) {
+      pagespeedRunSnapshot = {
+        ...pagespeedRunSnapshot,
+        running: false,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+        error: err instanceof Error ? err.message : String(err),
+      };
+      throw err;
+    } finally {
+      pagespeedRunActive = false;
+      pagespeedCancelRequested = false;
+    }
+  }
+
   ipcMain.handle(
     IPC.pagespeedRun,
-    async (_e, input: PagespeedRunInput): Promise<PagespeedRunResult> => {
-      if (pagespeedRunActive) {
-        return { completed: 0, failed: 0, cancelled: true };
-      }
-      const urls = Array.from(
-        new Set(
-          (input.urls ?? []).filter(
-            (u): u is string => typeof u === 'string' && u.length > 0,
-          ),
-        ),
-      );
-      if (urls.length === 0) {
-        return { completed: 0, failed: 0, cancelled: false };
-      }
-      const strategies: PagespeedStrategy[] =
-        input.strategy === 'both' ? ['mobile', 'desktop'] : [input.strategy];
-      const items: PagespeedBatchItem[] = [];
-      for (const url of urls) {
-        for (const strategy of strategies) items.push({ url, strategy });
-      }
-      const apiKeyRaw = resolveCredentials('pagespeed')['apiKey'];
-      const apiKey =
-        apiKeyRaw && apiKeyRaw.trim().length > 0 ? apiKeyRaw.trim() : undefined;
-
-      pagespeedRunActive = true;
-      pagespeedCancelRequested = false;
-      // Pin writes to the project that was active when the run started —
-      // a PSI run can last minutes, and resolving `getDb()` per result
-      // would leak audits into a different project if the user opens
-      // one mid-run.
-      const runDb = getDb();
-      logger.log(
-        'info',
-        'pagespeed',
-        `run started — ${items.length} audit(s): ${urls.length} URL(s) × ${strategies.length} strategy${
-          apiKey ? ' (keyed)' : ' (keyless — low rate limit)'
-        }`,
-      );
-      try {
-        const result = await runPagespeedBatch({
-          items,
-          apiKey,
-          isCancelled: () => pagespeedCancelRequested,
-          onResult: (url, strategy, metrics) => {
-            try {
-              runDb.upsertPagespeedResult(url, strategy, metrics);
-            } catch (err) {
-              logger.log(
-                'error',
-                'pagespeed',
-                `DB write failed for ${url}: ${
-                  err instanceof Error ? err.message : String(err)
-                }`,
-              );
-            }
-          },
-          onProgress: (done, total, currentUrl) => {
-            mainWindow?.webContents.send(IPC.pagespeedProgress, {
-              done,
-              total,
-              currentUrl,
-              running: true,
-            });
-          },
-        });
-        mainWindow?.webContents.send(IPC.pagespeedProgress, {
-          done: result.completed + result.failed,
-          total: items.length,
-          currentUrl: null,
-          running: false,
-        });
-        fireDataChanged();
-        return result;
-      } finally {
-        pagespeedRunActive = false;
-        pagespeedCancelRequested = false;
-      }
-    },
+    async (_e, input: PagespeedRunInput): Promise<PagespeedRunResult> =>
+      runPagespeedJob(input),
   );
 
   ipcMain.handle(IPC.pagespeedCancel, () => {
@@ -2686,115 +2773,161 @@ function registerIpc(): void {
       .replace(/\{body\}/g, body);
   };
 
-  ipcMain.handle(
-    IPC.aiRun,
-    async (_e, input: AiRunInput): Promise<AiRunResult> => {
-      if (aiRunActive) {
-        return { completed: 0, failed: 0, cancelled: true };
-      }
-      const urls = Array.from(
-        new Set(
-          (input.urls ?? []).filter(
-            (u): u is string => typeof u === 'string' && u.length > 0,
-          ),
+  // Shared AI batch orchestrator — driven by the renderer's `aiRun` IPC
+  // (awaits) and the MCP bridge's non-blocking `ai-run` action (polls
+  // `fetch-progress`). Updates `aiRunSnapshot` throughout.
+  async function runAiJob(input: AiRunInput): Promise<AiRunResult> {
+    if (aiRunActive) {
+      return { completed: 0, failed: 0, cancelled: true };
+    }
+    const urls = Array.from(
+      new Set(
+        (input.urls ?? []).filter(
+          (u): u is string => typeof u === 'string' && u.length > 0,
         ),
-      );
-      if (urls.length === 0 || !input.prompt?.trim()) {
-        return { completed: 0, failed: 0, cancelled: false };
-      }
-      const runDb = getDb();
-      const model = resolveModel(input.provider, input.model);
-      const concurrency = defaultConcurrency(input.provider);
+      ),
+    );
+    if (urls.length === 0 || !input.prompt?.trim()) {
+      aiRunSnapshot = {
+        ...idleFetchSnapshot(),
+        provider: input.provider,
+        finishedAt: new Date().toISOString(),
+      };
+      return { completed: 0, failed: 0, cancelled: false };
+    }
+    const runDb = getDb();
+    const model = resolveModel(input.provider, input.model);
+    const concurrency = defaultConcurrency(input.provider);
 
-      // Build per-URL prompts up-front so a missing page doesn't get
-      // queued; URLs that aren't in the DB are skipped silently.
-      const items: AiBatchItem[] = [];
-      for (const url of urls) {
-        const ctx = runDb.getAiContext(url);
-        if (!ctx) continue;
-        items.push({
-          url,
-          prompt: substitutePrompt(input.prompt, ctx),
-        });
-      }
-      if (items.length === 0) {
-        return { completed: 0, failed: 0, cancelled: false };
-      }
+    // Build per-URL prompts up-front so a missing page doesn't get
+    // queued; URLs that aren't in the DB are skipped silently.
+    const items: AiBatchItem[] = [];
+    for (const url of urls) {
+      const ctx = runDb.getAiContext(url);
+      if (!ctx) continue;
+      items.push({
+        url,
+        prompt: substitutePrompt(input.prompt, ctx),
+      });
+    }
+    if (items.length === 0) {
+      aiRunSnapshot = {
+        ...idleFetchSnapshot(),
+        provider: input.provider,
+        finishedAt: new Date().toISOString(),
+      };
+      return { completed: 0, failed: 0, cancelled: false };
+    }
 
-      aiRunActive = true;
-      aiCancelRequested = false;
-      logger.log(
-        'info',
-        'ai',
-        `${input.provider}/${model} run started — ${items.length} prompt(s) (concurrency=${concurrency})`,
-      );
-      try {
-        const result = await runAiBatch({
-          provider: input.provider,
-          model,
-          concurrency,
-          items,
-          isCancelled: () => aiCancelRequested,
-          onResult: (outcome) => {
-            try {
-              const fetchedAt = new Date().toISOString();
-              if (outcome.ok && outcome.output) {
-                runDb.upsertAiResult(
-                  outcome.url,
-                  input.provider,
-                  outcome.output.model,
-                  outcome.output.text,
-                  outcome.output.tokensIn,
-                  outcome.output.tokensOut,
-                  'ok',
-                  null,
-                  fetchedAt,
-                );
-              } else {
-                runDb.upsertAiResult(
-                  outcome.url,
-                  input.provider,
-                  model,
-                  '',
-                  null,
-                  null,
-                  'error',
-                  outcome.error ?? 'Unknown error',
-                  fetchedAt,
-                );
-              }
-            } catch (err) {
-              logger.log(
+    aiRunActive = true;
+    aiCancelRequested = false;
+    aiRunSnapshot = {
+      ...idleFetchSnapshot(),
+      running: true,
+      total: items.length,
+      provider: input.provider,
+      startedAt: new Date().toISOString(),
+    };
+    logger.log(
+      'info',
+      'ai',
+      `${input.provider}/${model} run started — ${items.length} prompt(s) (concurrency=${concurrency})`,
+    );
+    try {
+      const result = await runAiBatch({
+        provider: input.provider,
+        model,
+        concurrency,
+        items,
+        isCancelled: () => aiCancelRequested,
+        onResult: (outcome) => {
+          try {
+            const fetchedAt = new Date().toISOString();
+            if (outcome.ok && outcome.output) {
+              runDb.upsertAiResult(
+                outcome.url,
+                input.provider,
+                outcome.output.model,
+                outcome.output.text,
+                outcome.output.tokensIn,
+                outcome.output.tokensOut,
+                'ok',
+                null,
+                fetchedAt,
+              );
+            } else {
+              runDb.upsertAiResult(
+                outcome.url,
+                input.provider,
+                model,
+                '',
+                null,
+                null,
                 'error',
-                'ai',
-                `DB write failed for ${outcome.url}: ${
-                  err instanceof Error ? err.message : String(err)
-                }`,
+                outcome.error ?? 'Unknown error',
+                fetchedAt,
               );
             }
-          },
-          onProgress: (done, total, currentUrl) => {
-            mainWindow?.webContents.send(IPC.aiProgress, {
-              done,
-              total,
-              currentUrl,
-              running: true,
-            });
-          },
-        });
-        mainWindow?.webContents.send(IPC.aiProgress, {
-          done: result.completed + result.failed,
-          total: items.length,
-          currentUrl: null,
-          running: false,
-        });
-        fireDataChanged();
-        return result;
-      } finally {
-        aiRunActive = false;
-        aiCancelRequested = false;
-      }
-    },
+          } catch (err) {
+            logger.log(
+              'error',
+              'ai',
+              `DB write failed for ${outcome.url}: ${
+                err instanceof Error ? err.message : String(err)
+              }`,
+            );
+          }
+          if (outcome.ok) aiRunSnapshot.completed++;
+          else aiRunSnapshot.failed++;
+        },
+        onProgress: (done, total, currentUrl) => {
+          aiRunSnapshot.done = done;
+          aiRunSnapshot.total = total;
+          aiRunSnapshot.currentUrl = currentUrl;
+          mainWindow?.webContents.send(IPC.aiProgress, {
+            done,
+            total,
+            currentUrl,
+            running: true,
+          });
+        },
+      });
+      aiRunSnapshot = {
+        ...aiRunSnapshot,
+        running: false,
+        done: result.completed + result.failed,
+        completed: result.completed,
+        failed: result.failed,
+        cancelled: result.cancelled,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+      };
+      mainWindow?.webContents.send(IPC.aiProgress, {
+        done: result.completed + result.failed,
+        total: items.length,
+        currentUrl: null,
+        running: false,
+      });
+      fireDataChanged();
+      return result;
+    } catch (err) {
+      aiRunSnapshot = {
+        ...aiRunSnapshot,
+        running: false,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+        error: err instanceof Error ? err.message : String(err),
+      };
+      throw err;
+    } finally {
+      aiRunActive = false;
+      aiCancelRequested = false;
+    }
+  }
+
+  ipcMain.handle(
+    IPC.aiRun,
+    async (_e, input: AiRunInput): Promise<AiRunResult> => runAiJob(input),
   );
 
   ipcMain.handle(IPC.aiCancel, () => {
@@ -2826,69 +2959,112 @@ function registerIpc(): void {
   // Moz / Semrush). Same shape as the AI run: pin the DB to the active
   // project, fan URLs through the provider's per-URL endpoint, persist
   // ok/error rows so partial outages are visible in the UI.
+  // Shared SEO-authority batch orchestrator — driven by the renderer's
+  // `seoRun` IPC (awaits) and the MCP bridge's non-blocking `seo-run`
+  // action (polls `fetch-progress`). Updates `seoRunSnapshot` throughout.
+  async function runSeoJob(input: SeoRunInput): Promise<SeoRunResult> {
+    if (seoRunActive) return { completed: 0, failed: 0, cancelled: true };
+    const urls = Array.from(
+      new Set(
+        (input.urls ?? []).filter(
+          (u): u is string => typeof u === 'string' && u.length > 0,
+        ),
+      ),
+    );
+    if (urls.length === 0) {
+      seoRunSnapshot = {
+        ...idleFetchSnapshot(),
+        provider: input.provider,
+        finishedAt: new Date().toISOString(),
+      };
+      return { completed: 0, failed: 0, cancelled: false };
+    }
+    const runDb = getDb();
+    seoRunActive = true;
+    seoCancelRequested = false;
+    seoRunSnapshot = {
+      ...idleFetchSnapshot(),
+      running: true,
+      total: urls.length,
+      provider: input.provider,
+      startedAt: new Date().toISOString(),
+    };
+    logger.log('info', 'seo', `${input.provider} run started — ${urls.length} URL(s)`);
+    try {
+      const result = await runSeoBatch({
+        provider: input.provider,
+        urls,
+        isCancelled: () => seoCancelRequested,
+        onResult: (outcome) => {
+          try {
+            runDb.upsertSeoResult(
+              outcome.url,
+              input.provider,
+              outcome.metrics,
+              outcome.ok ? 'ok' : 'error',
+              outcome.error,
+              new Date().toISOString(),
+            );
+          } catch (err) {
+            logger.log(
+              'error',
+              'seo',
+              `DB write failed for ${outcome.url}: ${
+                err instanceof Error ? err.message : String(err)
+              }`,
+            );
+          }
+          if (outcome.ok) seoRunSnapshot.completed++;
+          else seoRunSnapshot.failed++;
+        },
+        onProgress: (done, total, currentUrl) => {
+          seoRunSnapshot.done = done;
+          seoRunSnapshot.total = total;
+          seoRunSnapshot.currentUrl = currentUrl;
+          mainWindow?.webContents.send(IPC.seoProgress, {
+            done,
+            total,
+            currentUrl,
+            running: true,
+          });
+        },
+      });
+      seoRunSnapshot = {
+        ...seoRunSnapshot,
+        running: false,
+        done: result.completed + result.failed,
+        completed: result.completed,
+        failed: result.failed,
+        cancelled: result.cancelled,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+      };
+      mainWindow?.webContents.send(IPC.seoProgress, {
+        done: result.completed + result.failed,
+        total: urls.length,
+        currentUrl: null,
+        running: false,
+      });
+      fireDataChanged();
+      return result;
+    } catch (err) {
+      seoRunSnapshot = {
+        ...seoRunSnapshot,
+        running: false,
+        currentUrl: null,
+        finishedAt: new Date().toISOString(),
+        error: err instanceof Error ? err.message : String(err),
+      };
+      throw err;
+    } finally {
+      seoRunActive = false;
+      seoCancelRequested = false;
+    }
+  }
+
   ipcMain.handle(
     IPC.seoRun,
-    async (_e, input: SeoRunInput): Promise<SeoRunResult> => {
-      if (seoRunActive) return { completed: 0, failed: 0, cancelled: true };
-      const urls = Array.from(
-        new Set(
-          (input.urls ?? []).filter(
-            (u): u is string => typeof u === 'string' && u.length > 0,
-          ),
-        ),
-      );
-      if (urls.length === 0) return { completed: 0, failed: 0, cancelled: false };
-      const runDb = getDb();
-      seoRunActive = true;
-      seoCancelRequested = false;
-      logger.log('info', 'seo', `${input.provider} run started — ${urls.length} URL(s)`);
-      try {
-        const result = await runSeoBatch({
-          provider: input.provider,
-          urls,
-          isCancelled: () => seoCancelRequested,
-          onResult: (outcome) => {
-            try {
-              runDb.upsertSeoResult(
-                outcome.url,
-                input.provider,
-                outcome.metrics,
-                outcome.ok ? 'ok' : 'error',
-                outcome.error,
-                new Date().toISOString(),
-              );
-            } catch (err) {
-              logger.log(
-                'error',
-                'seo',
-                `DB write failed for ${outcome.url}: ${
-                  err instanceof Error ? err.message : String(err)
-                }`,
-              );
-            }
-          },
-          onProgress: (done, total, currentUrl) => {
-            mainWindow?.webContents.send(IPC.seoProgress, {
-              done,
-              total,
-              currentUrl,
-              running: true,
-            });
-          },
-        });
-        mainWindow?.webContents.send(IPC.seoProgress, {
-          done: result.completed + result.failed,
-          total: urls.length,
-          currentUrl: null,
-          running: false,
-        });
-        fireDataChanged();
-        return result;
-      } finally {
-        seoRunActive = false;
-        seoCancelRequested = false;
-      }
-    },
+    async (_e, input: SeoRunInput): Promise<SeoRunResult> => runSeoJob(input),
   );
   ipcMain.handle(IPC.seoCancel, () => {
     if (seoRunActive) {
@@ -3621,6 +3797,51 @@ function registerIpc(): void {
   // already requires an authenticated request, and a misshapen payload
   // throws which the bridge wraps into a 500 with the underlying error
   // message. The MCP tool definitions on the other side enforce schema.
+  // Resolve the target URL list for a batch-fetch action (pagespeed-run /
+  // ai-run / seo-run): an explicit `urls` array wins; otherwise pull from
+  // the active project by `category` (+ optional `search`), capped by
+  // `limit` (default 100, max 5000). `truncated` flags when the category
+  // matched more rows than the cap. Category resolution is preferred over
+  // a giant `urls` array because the bridge caps request bodies at 64 KB.
+  function resolveActionUrls(input: {
+    urls?: unknown;
+    category?: unknown;
+    search?: unknown;
+    limit?: unknown;
+  }): { urls: string[]; truncated: boolean } {
+    const explicit = Array.isArray(input.urls)
+      ? input.urls.filter(
+          (u): u is string => typeof u === 'string' && u.length > 0,
+        )
+      : [];
+    if (explicit.length > 0) {
+      return { urls: Array.from(new Set(explicit)), truncated: false };
+    }
+    const category =
+      typeof input.category === 'string'
+        ? (input.category as UrlCategory)
+        : 'all';
+    const search = typeof input.search === 'string' ? input.search : undefined;
+    const limit =
+      typeof input.limit === 'number' && Number.isFinite(input.limit)
+        ? Math.max(1, Math.min(5000, Math.floor(input.limit)))
+        : 100;
+    const { rows, total } = getDb().queryUrls({
+      category,
+      search,
+      limit,
+      offset: 0,
+    });
+    const urls = Array.from(
+      new Set(
+        rows
+          .map((r) => r.url)
+          .filter((u): u is string => typeof u === 'string' && u.length > 0),
+      ),
+    );
+    return { urls, truncated: total > urls.length };
+  }
+
   const mcpActions: Record<string, (input: unknown) => Promise<unknown>> = {
     'crawl-add-url': async (input) => {
       const url = (input as { url?: string }).url ?? '';
@@ -4081,6 +4302,222 @@ function registerIpc(): void {
       if (!id) throw new Error('integrationId is required (e.g. "gsc" or "ga4").');
       return getAuthState(id);
     },
+
+    // ---- Faz 0.5 Increment 5 — slow batch-fetch triggers (PSI / AI /
+    // SEO). Non-blocking: each *-run kicks off the shared run-job WITHOUT
+    // awaiting it (the job populates its snapshot synchronously before its
+    // first await, so the response below is accurate) and returns at once.
+    // The agent polls `fetch-progress` and may `fetch-cancel`. This
+    // mirrors crawl start / progress / stop — the MCP wire protocol can't
+    // stream, so the same poll model is reused.
+
+    'pagespeed-run': async (input) => {
+      const i = (input ?? {}) as {
+        urls?: unknown;
+        category?: unknown;
+        search?: unknown;
+        limit?: unknown;
+        strategy?: unknown;
+      };
+      if (pagespeedRunActive) {
+        return {
+          started: false,
+          reason:
+            'A PageSpeed run is already in progress — poll get_fetch_progress or cancel_fetch first.',
+          state: pagespeedRunSnapshot,
+        };
+      }
+      const { urls, truncated } = resolveActionUrls(i);
+      if (urls.length === 0) {
+        return {
+          started: false,
+          reason:
+            'No URLs matched — pass an explicit `urls` array or a `category` that has rows in this project.',
+          state: pagespeedRunSnapshot,
+        };
+      }
+      const strategy =
+        i.strategy === 'desktop'
+          ? ('desktop' as const)
+          : i.strategy === 'both'
+            ? ('both' as const)
+            : ('mobile' as const);
+      void runPagespeedJob({ urls, strategy }).catch((err) => {
+        logger.log(
+          'error',
+          'pagespeed',
+          `MCP-triggered run failed: ${err instanceof Error ? err.message : String(err)}`,
+        );
+      });
+      return {
+        started: pagespeedRunActive,
+        total: pagespeedRunSnapshot.total,
+        truncated,
+        state: pagespeedRunSnapshot,
+      };
+    },
+
+    'ai-run': async (input) => {
+      const i = (input ?? {}) as {
+        provider?: unknown;
+        model?: unknown;
+        prompt?: unknown;
+        urls?: unknown;
+        category?: unknown;
+        search?: unknown;
+        limit?: unknown;
+      };
+      const provider = typeof i.provider === 'string' ? i.provider : '';
+      if (
+        provider !== 'openai' &&
+        provider !== 'anthropic' &&
+        provider !== 'ollama'
+      ) {
+        throw new Error(
+          'ai_run requires `provider` to be one of: openai, anthropic, ollama.',
+        );
+      }
+      const prompt = typeof i.prompt === 'string' ? i.prompt : '';
+      if (prompt.trim() === '') {
+        throw new Error(
+          'ai_run requires a non-empty `prompt` (supports {url}/{title}/{description}/{h1}/{body} placeholders).',
+        );
+      }
+      if (aiRunActive) {
+        return {
+          started: false,
+          reason:
+            'An AI run is already in progress — poll get_fetch_progress or cancel_fetch first.',
+          state: aiRunSnapshot,
+        };
+      }
+      const { urls, truncated } = resolveActionUrls(i);
+      if (urls.length === 0) {
+        return {
+          started: false,
+          reason:
+            'No URLs matched — pass an explicit `urls` array or a `category` that has rows in this project.',
+          state: aiRunSnapshot,
+        };
+      }
+      const model = typeof i.model === 'string' ? i.model : undefined;
+      void runAiJob({ provider, model, prompt, urls }).catch((err) => {
+        logger.log(
+          'error',
+          'ai',
+          `MCP-triggered run failed: ${err instanceof Error ? err.message : String(err)}`,
+        );
+      });
+      // runAiJob skips URLs with no crawled page context; if that leaves
+      // nothing it returns synchronously without flipping the active flag.
+      if (!aiRunActive) {
+        return {
+          started: false,
+          reason:
+            'None of the matched URLs are crawled in this project, so there is no page context to build prompts from.',
+          state: aiRunSnapshot,
+        };
+      }
+      return {
+        started: true,
+        total: aiRunSnapshot.total,
+        truncated,
+        state: aiRunSnapshot,
+      };
+    },
+
+    'seo-run': async (input) => {
+      const i = (input ?? {}) as {
+        provider?: unknown;
+        urls?: unknown;
+        category?: unknown;
+        search?: unknown;
+        limit?: unknown;
+      };
+      const provider = typeof i.provider === 'string' ? i.provider : '';
+      if (
+        provider !== 'ahrefs' &&
+        provider !== 'majestic' &&
+        provider !== 'moz' &&
+        provider !== 'semrush'
+      ) {
+        throw new Error(
+          'seo_run requires `provider` to be one of: ahrefs, majestic, moz, semrush.',
+        );
+      }
+      if (seoRunActive) {
+        return {
+          started: false,
+          reason:
+            'An SEO run is already in progress — poll get_fetch_progress or cancel_fetch first.',
+          state: seoRunSnapshot,
+        };
+      }
+      const { urls, truncated } = resolveActionUrls(i);
+      if (urls.length === 0) {
+        return {
+          started: false,
+          reason:
+            'No URLs matched — pass an explicit `urls` array or a `category` that has rows in this project.',
+          state: seoRunSnapshot,
+        };
+      }
+      void runSeoJob({ provider, urls }).catch((err) => {
+        logger.log(
+          'error',
+          'seo',
+          `MCP-triggered run failed: ${err instanceof Error ? err.message : String(err)}`,
+        );
+      });
+      return {
+        started: seoRunActive,
+        total: seoRunSnapshot.total,
+        truncated,
+        state: seoRunSnapshot,
+      };
+    },
+
+    'fetch-progress': async (input) => {
+      const kind = (input as { kind?: unknown } | null)?.kind;
+      const all = {
+        pagespeed: pagespeedRunSnapshot,
+        ai: aiRunSnapshot,
+        seo: seoRunSnapshot,
+      };
+      if (kind === 'pagespeed' || kind === 'ai' || kind === 'seo') {
+        return { [kind]: all[kind] };
+      }
+      return all;
+    },
+
+    'fetch-cancel': async (input) => {
+      const kind = (input as { kind?: unknown } | null)?.kind;
+      const cancelled: Record<string, { wasRunning: boolean }> = {};
+      const cancelPagespeed = (): void => {
+        const was = pagespeedRunActive;
+        if (was) pagespeedCancelRequested = true;
+        cancelled.pagespeed = { wasRunning: was };
+      };
+      const cancelAi = (): void => {
+        const was = aiRunActive;
+        if (was) aiCancelRequested = true;
+        cancelled.ai = { wasRunning: was };
+      };
+      const cancelSeo = (): void => {
+        const was = seoRunActive;
+        if (was) seoCancelRequested = true;
+        cancelled.seo = { wasRunning: was };
+      };
+      if (kind === 'pagespeed') cancelPagespeed();
+      else if (kind === 'ai') cancelAi();
+      else if (kind === 'seo') cancelSeo();
+      else {
+        cancelPagespeed();
+        cancelAi();
+        cancelSeo();
+      }
+      return { ok: true, cancelled };
+    },
   };
 
   crawlController = {

apps/desktop/src/main/pagespeed.ts

@@ -18,6 +18,7 @@
  *     whole crawl.
  */
 import type { PagespeedMetrics, PagespeedStrategy } from '@freecrawl/shared-types';
+import { Agent, ProxyAgent, fetch as undiciFetch, type Dispatcher } from 'undici';
 import * as logger from './logger.js';
 
 const PSI_ENDPOINT =
@@ -32,6 +33,41 @@ const REQUEST_TIMEOUT_MS = 90_000;
 const CONCURRENCY_WITH_KEY = 4;
 const CONCURRENCY_NO_KEY = 1;
 
+/**
+ * Dedicated undici dispatcher for PSI requests.
+ *
+ * PSI runs a Lighthouse audit server-side and withholds the response
+ * headers for 10–30 s+ while it does. The crawler installs a *process-
+ * global* dispatcher (`setGlobalDispatcher`) with `headersTimeout:
+ * 10_000` — correct for crawling (a stuck origin can't freeze the pool)
+ * but fatal here: a plain `fetch` inherits it, so under batch
+ * concurrency Google's per-request response slips past 10 s and every
+ * audit is killed at the transport layer, surfacing as "fetch failed" /
+ * NET. (Single audits usually return just under 10 s, which is why they
+ * pass.) This dispatcher's header + body timeouts sit just above the
+ * 90 s `AbortSignal` that stays the real cap, so the audit gets the time
+ * it legitimately needs. HTTPS_PROXY / HTTP_PROXY are honoured to match
+ * the crawler so corporate-proxy users keep working.
+ */
+function createPsiDispatcher(): Dispatcher {
+  const timeouts = {
+    headersTimeout: REQUEST_TIMEOUT_MS + 5_000,
+    bodyTimeout: REQUEST_TIMEOUT_MS + 5_000,
+  };
+  const proxy =
+    process.env['HTTPS_PROXY'] ??
+    process.env['https_proxy'] ??
+    process.env['HTTP_PROXY'] ??
+    process.env['http_proxy'] ??
+    null;
+  if (proxy) {
+    return new ProxyAgent({ uri: proxy, ...timeouts });
+  }
+  return new Agent({ ...timeouts, connect: { autoSelectFamily: true } });
+}
+
+const psiDispatcher = createPsiDispatcher();
+
 function errorMetrics(fetchedAt: string, message: string): PagespeedMetrics {
   return {
     performance: null,
@@ -99,9 +135,12 @@ export async function fetchPagespeedAudit(
   if (apiKey) params.set('key', apiKey);
 
   try {
-    const res = await fetch(`${PSI_ENDPOINT}?${params.toString()}`, {
+    const res = await undiciFetch(`${PSI_ENDPOINT}?${params.toString()}`, {
       signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
       headers: { Accept: 'application/json' },
+      // Use PSI's own long-poll dispatcher, NOT the crawler's global one
+      // (whose 10 s headersTimeout strangles Lighthouse audits).
+      dispatcher: psiDispatcher,
     });
     const json = (await res.json().catch(() => null)) as Record<
       string,
@@ -149,13 +188,28 @@ export async function fetchPagespeedAudit(
   } catch (err) {
     // `AbortSignal.timeout` rejects with a DOMException, which is not
     // always an `instanceof Error` in Node — sniff `.name` directly.
+    // undici transport failures reject with a generic "fetch failed"
+    // whose real reason hides on `.cause` (e.g. UND_ERR_HEADERS_TIMEOUT);
+    // read it so the surfaced message reflects the actual failure instead
+    // of a bare, misleading "fetch failed".
     const name = (err as { name?: string } | null)?.name;
-    const message =
-      name === 'TimeoutError' || name === 'AbortError'
-        ? `PageSpeed audit timed out after ${REQUEST_TIMEOUT_MS / 1000}s`
-        : err instanceof Error
-          ? err.message
-          : String(err);
+    const cause = (err as { cause?: unknown } | null)?.cause as
+      | { code?: string; message?: string }
+      | undefined;
+    const code = cause?.code;
+    const isTimeout =
+      name === 'TimeoutError' ||
+      name === 'AbortError' ||
+      code === 'UND_ERR_HEADERS_TIMEOUT' ||
+      code === 'UND_ERR_BODY_TIMEOUT' ||
+      code === 'UND_ERR_CONNECT_TIMEOUT';
+    const message = isTimeout
+      ? `PageSpeed audit timed out after ${REQUEST_TIMEOUT_MS / 1000}s`
+      : err instanceof Error
+        ? cause?.message
+          ? `${err.message} (${cause.message})`
+          : err.message
+        : String(err);
     return errorMetrics(fetchedAt, message);
   }
 }

apps/desktop/src/renderer/src/components/BottomDetailPanel.tsx

@@ -1647,6 +1647,11 @@ function NameValueView({ row }: { row: CrawlUrlRow }) {
     ['Manifest Display Mode', row.manifestDisplay],
     ['Manifest Scope', row.manifestScope],
     ['Manifest Icon Count', row.manifestIconCount > 0 ? row.manifestIconCount : null],
+    ['PDF Title', row.pdfTitle],
+    ['PDF Author', row.pdfAuthor],
+    ['PDF Page Count', row.pdfPageCount],
+    ['PDF Created', row.pdfCreationDate],
+    ['PDF Producer', row.pdfProducer],
     ['RSS / Atom Feed', row.feedUrl],
     ['Title Tag Count', row.titleCount > 1 ? row.titleCount : null],
     ['Empty-Alt Images', row.imagesEmptyAlt > 0 ? row.imagesEmptyAlt : null],

apps/desktop/src/renderer/src/components/ExportDialog.tsx

@@ -177,6 +177,10 @@ export function ExportDialog({
 
   const [format, setFormat] = useState<Format>('xlsx');
   const [csvBom, setCsvBom] = useState<boolean>(true);
+  // V2 Faz 15 — when on, ignore the table tree and emit three rollups
+  // (Critical / Warning / Info) via the virtual `issues:severity-*`
+  // categories instead.
+  const [splitBySeverity, setSplitBySeverity] = useState<boolean>(false);
 
   // Picked leaves — childless tabs use the tab key, child rows use
   // `<tabKey>:<filename>`. Defaults to the active tab's first leaf so
@@ -218,6 +222,9 @@ export function ExportDialog({
    *  column picker (union of selected tabs' columns). */
   const pickedTabKeys = useMemo(() => {
     const set = new Set<TabKey>();
+    // Severity-split export emits standard URL rows, so offer the
+    // "internal" tab's columns even when no section leaf is picked.
+    if (splitBySeverity) set.add('internal');
     for (const node of tree) {
       if (node.children.length === 0) {
         if (pickedLeaves.has(leafId(node))) set.add(node.tabKey);
@@ -231,7 +238,7 @@ export function ExportDialog({
       }
     }
     return set;
-  }, [pickedLeaves, tree]);
+  }, [pickedLeaves, tree, splitBySeverity]);
 
   const offeredColumns = useMemo(() => {
     const seen = new Set<string>();
@@ -367,34 +374,73 @@ export function ExportDialog({
     setPickedColumns(new Set());
   };
 
+  const toggleSplitBySeverity = () => {
+    const entering = !splitBySeverity;
+    setSplitBySeverity(entering);
+    // Entering severity mode with only non-internal columns picked would
+    // leave the offered (internal) column set empty — seed it once.
+    if (entering) {
+      setPickedColumns((prev) => {
+        const specs = COLUMN_SPECS['internal'] ?? [];
+        if (specs.some((s) => prev.has(s.key as string))) return prev;
+        const next = new Set(prev);
+        for (const s of specs) next.add(s.key as string);
+        return next;
+      });
+    }
+  };
+
   const submit = async () => {
     if (orderedColumnList.length === 0) {
       setStatus(t('export.errorNoColumn', { defaultValue: 'Pick at least one column.' }));
       return;
     }
-    if (totalLeafSelections === 0) {
+    if (!splitBySeverity && totalLeafSelections === 0) {
       setStatus(t('export.errorNoTable', { defaultValue: 'Pick at least one table.' }));
       return;
     }
     setBusy(true);
     setStatus('');
     try {
-      const sections: ExportTabularSection[] = [];
-      for (const node of tree) {
-        if (node.children.length === 0) {
-          if (pickedLeaves.has(leafId(node))) {
-            sections.push({ label: node.label, category: node.category });
+      let sections: ExportTabularSection[];
+      if (splitBySeverity) {
+        // Three rollups via the virtual severity categories. Each yields
+        // the deduped set of URLs flagged by any issue of that tier.
+        sections = [
+          {
+            label: t('export.severityCritical', { defaultValue: 'Critical Issues' }),
+            category: 'issues:severity-critical',
+            filename: 'critical-issues',
+          },
+          {
+            label: t('export.severityWarning', { defaultValue: 'Warning Issues' }),
+            category: 'issues:severity-warning',
+            filename: 'warning-issues',
+          },
+          {
+            label: t('export.severityInfo', { defaultValue: 'Info Issues' }),
+            category: 'issues:severity-info',
+            filename: 'info-issues',
+          },
+        ];
+      } else {
+        sections = [];
+        for (const node of tree) {
+          if (node.children.length === 0) {
+            if (pickedLeaves.has(leafId(node))) {
+              sections.push({ label: node.label, category: node.category });
+            }
+            continue;
+          }
+          for (const child of node.children) {
+            if (!pickedLeaves.has(leafId(node, child))) continue;
+            sections.push({
+              label: `${node.label} — ${child.label}`,
+              category: child.category,
+              subdir: node.subdir,
+              filename: child.filename,
+            });
           }
-          continue;
-        }
-        for (const child of node.children) {
-          if (!pickedLeaves.has(leafId(node, child))) continue;
-          sections.push({
-            label: `${node.label} — ${child.label}`,
-            category: child.category,
-            subdir: node.subdir,
-            filename: child.filename,
-          });
         }
       }
       const result = await window.freecrawl.exportTabular({
@@ -430,7 +476,8 @@ export function ExportDialog({
     }
   };
 
-  const showFolderHint = format !== 'xlsx' && (totalLeafSelections > 1);
+  const showFolderHint =
+    format !== 'xlsx' && (totalLeafSelections > 1 || splitBySeverity);
 
   return (
     <div
@@ -702,6 +749,19 @@ export function ExportDialog({
                 </span>
               </label>
             )}
+
+            <label className="flex cursor-pointer items-center gap-2 text-[12px] text-surface-200">
+              <input
+                type="checkbox"
+                checked={splitBySeverity}
+                onChange={toggleSplitBySeverity}
+              />
+              <span>
+                {t('export.splitBySeverity', {
+                  defaultValue: 'Split by issue severity (Critical / Warning / Info)',
+                })}
+              </span>
+            </label>
           </div>
 
           {showFolderHint && (
@@ -714,7 +774,7 @@ export function ExportDialog({
             </div>
           )}
 
-          {format === 'xlsx' && totalLeafSelections > 1 && (
+          {format === 'xlsx' && (totalLeafSelections > 1 || splitBySeverity) && (
             <div className="mt-2 rounded border border-blue-700/40 bg-blue-900/20 px-2 py-1 text-[11px] text-blue-200">
               {t('export.xlsxNote', {
                 defaultValue:
@@ -723,6 +783,15 @@ export function ExportDialog({
             </div>
           )}
 
+          {splitBySeverity && (
+            <div className="mt-2 rounded border border-blue-700/40 bg-blue-900/20 px-2 py-1 text-[11px] text-blue-200">
+              {t('export.severityNote', {
+                defaultValue:
+                  'Severity split overrides the table selection: three rollups (Critical / Warning / Info), each listing every URL flagged by an issue of that tier, using the columns picked at right.',
+              })}
+            </div>
+          )}
+
           {status && (
             <div className="mt-3 truncate rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[11px] text-surface-200">
               {status}

apps/desktop/src/renderer/src/components/OverviewSidebar.tsx

@@ -1139,6 +1139,12 @@ function buildTree(o: OverviewCounts | null): Node[] {
               count: o.issues.http2NotSupported,
               category: 'issues:http2-not-supported',
             },
+            {
+              key: 'issues-http3-not-supported',
+              label: 'HTTP/3 Not Advertised',
+              count: o.issues.http3NotSupported,
+              category: 'issues:http3-not-supported',
+            },
             {
               key: 'issues-render-blocking',
               label: 'Render-Blocking Head (>5)',

apps/desktop/src/renderer/src/components/SettingsDialog.tsx

@@ -46,6 +46,7 @@ import type {
   CrawlConfig,
   CrawlMode,
   CustomExtractionRule,
+  FormLoginStep,
   HttpAuth,
   IntegrationDef,
   IntegrationsState,
@@ -116,6 +117,8 @@ interface FormState {
   webhookUrl: string;
   // auth + network
   auth: HttpAuth;
+  formLoginEnabled: boolean;
+  formLoginSteps: FormLoginStep[];
   proxyUrl: string;
   excludeExtensionsText: string;
   maxRedirects: string;
@@ -274,13 +277,13 @@ const SECTIONS: SectionDef[] = [
     key: 'auth',
     label: 'Authentication',
     icon: Shield,
-    keywords: 'auth authentication basic bearer token password http header',
+    keywords: 'auth authentication basic bearer digest token password http header form login session cookie csrf',
   },
   {
     key: 'network',
     label: 'Network',
     icon: Network,
-    keywords: 'network proxy https extension filter exclude redirect hop limit',
+    keywords: 'network proxy https socks socks5 extension filter exclude redirect hop limit',
   },
   {
     key: 'duplicates',
@@ -423,6 +426,12 @@ function configToForm(c: CrawlConfig): FormState {
     customExtractionRules: (c.customExtractionRules ?? []).map((r) => ({ ...r })),
     webhookUrl: c.webhookUrl ?? '',
     auth: { ...(c.auth ?? { type: 'none' }) },
+    formLoginEnabled: c.formLogin?.enabled ?? false,
+    formLoginSteps: (c.formLogin?.steps ?? []).map((s) => ({
+      ...s,
+      fields: (s.fields ?? []).map((f) => ({ ...f })),
+      captures: (s.captures ?? []).map((cp) => ({ ...cp })),
+    })),
     proxyUrl: c.proxyUrl ?? '',
     excludeExtensionsText: (c.excludeExtensions ?? []).join(', '),
     maxRedirects: String(c.maxRedirects ?? 10),
@@ -629,6 +638,17 @@ export function SettingsDialog({ open, onClose }: Props) {
         .slice(0, 10),
       webhookUrl: form.webhookUrl.trim(),
       auth: form.auth,
+      formLogin: {
+        enabled: form.formLoginEnabled,
+        steps: form.formLoginSteps
+          .map((s) => ({
+            url: s.url.trim(),
+            method: s.method,
+            fields: (s.fields ?? []).filter((f) => f.name.trim()),
+            captures: (s.captures ?? []).filter((c) => c.name.trim() && c.selector.trim()),
+          }))
+          .filter((s) => s.url),
+      },
       proxyUrl: form.proxyUrl.trim(),
       excludeExtensions: form.excludeExtensionsText
         .split(/[\s,]+/)
@@ -1941,29 +1961,45 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
                 className="rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
                 value={r.type}
                 onChange={(e) =>
-                  updateRule(i, { type: e.target.value as 'css' | 'regex' })
+                  updateRule(i, { type: e.target.value as CustomExtractionRule['type'] })
                 }
               >
                 <option value="css">{t('settingsPanels.customExtraction.cssSelector', { defaultValue: 'CSS Selector' })}</option>
+                <option value="xpath">{t('settingsPanels.customExtraction.xpath', { defaultValue: 'XPath' })}</option>
                 <option value="regex">{t('settingsPanels.customExtraction.regex', { defaultValue: 'Regex' })}</option>
+                <option value="jsonpath">{t('settingsPanels.customExtraction.jsonpath', { defaultValue: 'JSONPath (JSON response)' })}</option>
               </select>
             </label>
           </div>
 
           <label className="mb-2 flex flex-col gap-1">
             <FieldLabel
-              label={r.type === 'css'
-                ? t('settingsPanels.customExtraction.cssSelector', { defaultValue: 'CSS Selector' })
-                : t('settingsPanels.customExtraction.regexPattern', { defaultValue: 'Regex Pattern' })}
+              label={
+                r.type === 'css'
+                  ? t('settingsPanels.customExtraction.cssSelector', { defaultValue: 'CSS Selector' })
+                  : r.type === 'xpath'
+                    ? t('settingsPanels.customExtraction.xpathExpr', { defaultValue: 'XPath Expression' })
+                    : r.type === 'jsonpath'
+                      ? t('settingsPanels.customExtraction.jsonpathExpr', { defaultValue: 'JSONPath Expression' })
+                      : t('settingsPanels.customExtraction.regexPattern', { defaultValue: 'Regex Pattern' })
+              }
               info={
                 r.type === 'css'
                   ? 'Standard CSS selector — same syntax as `document.querySelectorAll`.'
-                  : 'JavaScript regex (no flags — /g is implicit). Use a capture group with `output=regex_group` to extract just part of the match.'
+                  : r.type === 'xpath'
+                    ? 'XPath 1.0 subset over the parsed DOM. End in `/@attr` or `/text()` to read an attribute / text node. Predicates: `[n]`, `[@class="x"]`, `[contains(@class,"x")]`, `[last()]`.'
+                    : r.type === 'jsonpath'
+                      ? 'JSONPath against a JSON response body (e.g. `application/json` APIs). Only runs on responses that parse as JSON — ignored on HTML pages.'
+                      : 'JavaScript regex (no flags — /g is implicit). Use a capture group with `output=regex_group` to extract just part of the match.'
               }
               example={
                 r.type === 'css'
                   ? '.price > .amount,  meta[property="og:image"],  .breadcrumb li:last-child'
-                  : 'sku-([A-Z0-9]+),  "price"\\s*:\\s*"([^"]+)"'
+                  : r.type === 'xpath'
+                    ? "//meta[@property='og:title']/@content,  //h1,  //div[@class='price']"
+                    : r.type === 'jsonpath'
+                      ? '$.products[*].price,  $..author,  $.data.items[0].title'
+                      : 'sku-([A-Z0-9]+),  "price"\\s*:\\s*"([^"]+)"'
               }
             />
             <input
@@ -1979,9 +2015,11 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
               <FieldLabel
                 label={t('settingsPanels.customExtraction.output', { defaultValue: 'Output' })}
                 info={
-                  r.type === 'css'
-                    ? 'What to read off each matched element.'
-                    : 'For regex: `regex_group` extracts capture group 1; otherwise the whole match is used.'
+                  r.type === 'css' || r.type === 'xpath'
+                    ? 'What to read off each matched element. Ignored for an XPath `/@attr` or `/text()` terminal — that value is used directly.'
+                    : r.type === 'jsonpath'
+                      ? 'JSONPath returns the matched JSON value as-is; choose `Count` to return the number of matches instead.'
+                      : 'For regex: `regex_group` extracts capture group 1; otherwise the whole match is used.'
                 }
                 example="text for visible content, attribute for href/src, count for occurrence count"
               />
@@ -1992,7 +2030,7 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
                   updateRule(i, { output: e.target.value as CustomExtractionRule['output'] })
                 }
               >
-                {r.type === 'css' ? (
+                {r.type === 'css' || r.type === 'xpath' ? (
                   <>
                     <option value="text">{t('settingsPanels.customExtraction.outputText', { defaultValue: 'Text' })}</option>
                     <option value="attribute">{t('settingsPanels.customExtraction.outputAttribute', { defaultValue: 'Attribute' })}</option>
@@ -2000,6 +2038,11 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
                     <option value="outer_html">{t('settingsPanels.customExtraction.outputOuterHtml', { defaultValue: 'Outer HTML' })}</option>
                     <option value="count">{t('settingsPanels.customExtraction.outputCount', { defaultValue: 'Count' })}</option>
                   </>
+                ) : r.type === 'jsonpath' ? (
+                  <>
+                    <option value="text">{t('settingsPanels.customExtraction.outputValue', { defaultValue: 'Value' })}</option>
+                    <option value="count">{t('settingsPanels.customExtraction.outputCount', { defaultValue: 'Count' })}</option>
+                  </>
                 ) : (
                   <>
                     <option value="regex_group">{t('settingsPanels.customExtraction.outputCaptureGroup', { defaultValue: 'Capture group 1' })}</option>
@@ -2009,7 +2052,7 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
                 )}
               </select>
             </label>
-            {r.type === 'css' && r.output === 'attribute' ? (
+            {(r.type === 'css' || r.type === 'xpath') && r.output === 'attribute' ? (
               <label className="flex flex-col gap-1">
                 <FieldLabel
                   label={t('settingsPanels.customExtraction.attribute', { defaultValue: 'Attribute' })}
@@ -2108,15 +2151,15 @@ function AuthPanel({ form, update }: PanelProps) {
   return (
     <>
       <p className="mb-3 text-[11px] text-surface-400">
-        {t('settingsPanels.auth.intro', { defaultValue: 'HTTP authentication applied on every request. Useful for staging environments behind Basic auth, or APIs that require a Bearer token. Digest is not supported (challenge-response state machine).' })}
+        {t('settingsPanels.auth.intro', { defaultValue: 'HTTP authentication applied on every request. Useful for staging environments behind Basic/Digest auth, or APIs that require a Bearer token. For sites behind an HTML login form, use Form Login below instead.' })}
       </p>
 
       <div className="mb-4 rounded border border-surface-800 bg-surface-950/40 p-3">
         <label className="mb-2 flex flex-col gap-1">
           <FieldLabel
             label={t('settingsPanels.auth.scheme', { defaultValue: 'Auth scheme' })}
-            info="`none` disables auth; `basic` adds `Authorization: Basic <base64>`; `bearer` adds `Authorization: Bearer <token>`."
-            example="basic for /staging behind nginx; bearer for protected APIs"
+            info="`none` disables auth; `basic` adds `Authorization: Basic <base64>`; `bearer` adds `Authorization: Bearer <token>`; `digest` performs the RFC 2617 challenge-response on the first 401."
+            example="basic/digest for /staging behind nginx; bearer for protected APIs"
           />
           <select
             className="rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
@@ -2127,17 +2170,18 @@ function AuthPanel({ form, update }: PanelProps) {
           >
             <option value="none">{t('settingsPanels.auth.schemeNone', { defaultValue: 'None' })}</option>
             <option value="basic">{t('settingsPanels.auth.schemeBasic', { defaultValue: 'Basic (username + password)' })}</option>
+            <option value="digest">{t('settingsPanels.auth.schemeDigest', { defaultValue: 'Digest (username + password)' })}</option>
             <option value="bearer">{t('settingsPanels.auth.schemeBearer', { defaultValue: 'Bearer (token)' })}</option>
           </select>
         </label>
 
-        {auth.type === 'basic' && (
+        {(auth.type === 'basic' || auth.type === 'digest') && (
           <>
             <Text
               label={t('settingsPanels.auth.username', { defaultValue: 'Username' })}
               value={auth.username ?? ''}
               onChange={(v) => setAuth({ username: v })}
-              info="Sent base64-encoded as the first half of the credential pair."
+              info="For Basic, sent base64-encoded; for Digest, hashed into the challenge response."
               example="staging-user"
             />
             <Text
@@ -2160,10 +2204,194 @@ function AuthPanel({ form, update }: PanelProps) {
           />
         )}
       </div>
+
+      <FormLoginEditor form={form} update={update} />
     </>
   );
 }
 
+function FormLoginEditor({ form, update }: PanelProps) {
+  const { t } = useTranslation();
+  const steps = form.formLoginSteps;
+  const setSteps = (next: FormLoginStep[]) => update('formLoginSteps', next);
+  const updateStep = (i: number, patch: Partial<FormLoginStep>) => {
+    const next = steps.slice();
+    next[i] = { ...next[i]!, ...patch };
+    setSteps(next);
+  };
+  const addStep = () =>
+    setSteps([...steps, { url: '', method: 'POST', fields: [], captures: [] }]);
+
+  return (
+    <div className="mb-4 rounded border border-surface-800 bg-surface-950/40 p-3">
+      <div className="mb-2 flex items-center justify-between">
+        <div className="text-[10px] font-semibold uppercase tracking-wider text-surface-400">
+          {t('settingsGroups.formLogin', { defaultValue: 'Form-Based Login' })}
+        </div>
+        <Bool
+          label={t('settingsPanels.formLogin.enabled', { defaultValue: 'Enabled' })}
+          checked={form.formLoginEnabled}
+          onChange={(v) => update('formLoginEnabled', v)}
+          info="Run the login steps once before the crawl, then replay the session cookies on every request."
+        />
+      </div>
+      <p className="mb-3 text-[10px] text-surface-500">
+        {t('settingsPanels.formLogin.intro', {
+          defaultValue:
+            'Steps run in order over one shared cookie jar. A GET step can capture a CSRF token from the page; a POST step submits credentials (and the captured token) as a form. Reference captured values with {{name}}.',
+        })}
+      </p>
+
+      {steps.length === 0 && (
+        <p className="mb-2 text-[11px] italic text-surface-500">
+          {t('settingsPanels.formLogin.empty', { defaultValue: 'No steps — add one to define the login flow.' })}
+        </p>
+      )}
+
+      {steps.map((step, i) => (
+        <div key={i} className="mb-3 rounded border border-surface-800 bg-surface-900/40 p-2">
+          <div className="mb-2 flex items-center justify-between">
+            <div className="text-[10px] font-semibold uppercase tracking-wider text-surface-500">
+              {t('settingsPanels.formLogin.stepN', { defaultValue: 'Step #{{n}}', n: i + 1 })}
+            </div>
+            <button
+              className="rounded p-1 text-surface-500 hover:bg-surface-800 hover:text-red-400"
+              onClick={() => setSteps(steps.filter((_, j) => j !== i))}
+              title={t('settingsPanels.formLogin.removeStep', { defaultValue: 'Remove step' })}
+            >
+              <Trash2 className="h-3.5 w-3.5" />
+            </button>
+          </div>
+
+          <div className="mb-2 flex gap-2">
+            <select
+              className="w-24 rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+              value={step.method}
+              onChange={(e) => updateStep(i, { method: e.target.value as FormLoginStep['method'] })}
+            >
+              <option value="POST">POST</option>
+              <option value="GET">GET</option>
+            </select>
+            <input
+              className="flex-1 rounded border border-surface-700 bg-surface-950 px-2 py-1 font-mono text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+              placeholder="https://example.com/login"
+              value={step.url}
+              spellCheck={false}
+              onChange={(e) => updateStep(i, { url: e.target.value })}
+            />
+          </div>
+
+          {step.method === 'POST' && (
+            <div className="mb-2">
+              <div className="mb-1 text-[10px] font-semibold uppercase tracking-wider text-surface-500">
+                {t('settingsPanels.formLogin.fields', { defaultValue: 'Form fields (name = value)' })}
+              </div>
+              {step.fields.map((f, fi) => (
+                <div key={fi} className="mb-1 flex items-center gap-2">
+                  <input
+                    className="w-32 rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+                    placeholder="username"
+                    value={f.name}
+                    onChange={(e) => {
+                      const next = step.fields.slice();
+                      next[fi] = { ...f, name: e.target.value };
+                      updateStep(i, { fields: next });
+                    }}
+                  />
+                  <input
+                    className="flex-1 rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+                    placeholder="admin  /  {{csrf}}"
+                    value={f.value}
+                    onChange={(e) => {
+                      const next = step.fields.slice();
+                      next[fi] = { ...f, value: e.target.value };
+                      updateStep(i, { fields: next });
+                    }}
+                  />
+                  <button
+                    className="rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-300 hover:border-red-500 hover:text-red-300"
+                    onClick={() => updateStep(i, { fields: step.fields.filter((_, j) => j !== fi) })}
+                  >
+                    <Trash2 className="h-3 w-3" />
+                  </button>
+                </div>
+              ))}
+              <button
+                className="mt-1 flex items-center gap-1 rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-200 hover:border-blue-500 hover:bg-surface-800"
+                onClick={() => updateStep(i, { fields: [...step.fields, { name: '', value: '' }] })}
+              >
+                <Plus className="h-3 w-3" /> {t('settingsPanels.formLogin.addField', { defaultValue: 'Add field' })}
+              </button>
+            </div>
+          )}
+
+          <div>
+            <div className="mb-1 text-[10px] font-semibold uppercase tracking-wider text-surface-500">
+              {t('settingsPanels.formLogin.captures', { defaultValue: 'Capture from response (name ← CSS selector)' })}
+            </div>
+            {step.captures.map((c, ci) => (
+              <div key={ci} className="mb-1 flex items-center gap-2">
+                <input
+                  className="w-28 rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+                  placeholder="csrf"
+                  value={c.name}
+                  onChange={(e) => {
+                    const next = step.captures.slice();
+                    next[ci] = { ...c, name: e.target.value };
+                    updateStep(i, { captures: next });
+                  }}
+                />
+                <input
+                  className="flex-1 rounded border border-surface-700 bg-surface-950 px-2 py-1 font-mono text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+                  placeholder='input[name="_csrf"]'
+                  spellCheck={false}
+                  value={c.selector}
+                  onChange={(e) => {
+                    const next = step.captures.slice();
+                    next[ci] = { ...c, selector: e.target.value };
+                    updateStep(i, { captures: next });
+                  }}
+                />
+                <input
+                  className="w-20 rounded border border-surface-700 bg-surface-950 px-2 py-1 text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+                  placeholder="value"
+                  value={c.attribute ?? ''}
+                  onChange={(e) => {
+                    const next = step.captures.slice();
+                    next[ci] = { ...c, attribute: e.target.value };
+                    updateStep(i, { captures: next });
+                  }}
+                />
+                <button
+                  className="rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-300 hover:border-red-500 hover:text-red-300"
+                  onClick={() => updateStep(i, { captures: step.captures.filter((_, j) => j !== ci) })}
+                >
+                  <Trash2 className="h-3 w-3" />
+                </button>
+              </div>
+            ))}
+            <button
+              className="mt-1 flex items-center gap-1 rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-200 hover:border-blue-500 hover:bg-surface-800"
+              onClick={() =>
+                updateStep(i, { captures: [...step.captures, { name: '', selector: '', attribute: 'value' }] })
+              }
+            >
+              <Plus className="h-3 w-3" /> {t('settingsPanels.formLogin.addCapture', { defaultValue: 'Add capture' })}
+            </button>
+          </div>
+        </div>
+      ))}
+
+      <button
+        className="flex items-center gap-1 rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-200 hover:border-blue-500 hover:bg-surface-800"
+        onClick={addStep}
+      >
+        <Plus className="h-3 w-3" /> {t('settingsPanels.formLogin.addStep', { defaultValue: 'Add login step' })}
+      </button>
+    </div>
+  );
+}
+
 function NetworkPanel({ form, update }: PanelProps) {
   const { t } = useTranslation();
   return (
@@ -2177,8 +2405,8 @@ function NetworkPanel({ form, update }: PanelProps) {
           label={t('settingsPanels.network.proxyUrl', { defaultValue: 'Proxy URL (overrides HTTPS_PROXY)' })}
           value={form.proxyUrl}
           onChange={(v) => update('proxyUrl', v)}
-          info="Same syntax as HTTPS_PROXY/HTTP_PROXY env vars. Leave empty to inherit env. Routes via undici's ProxyAgent."
-          example="http://user:pass@proxy.corp:8080, http://10.0.0.5:3128"
+          info="HTTP/HTTPS proxies route via undici's ProxyAgent; SOCKS proxies (socks5://, socks5h://, socks4://, socks4a://) tunnel via the socks client. The `h`/`4a` variants resolve DNS at the proxy. Leave empty to inherit HTTPS_PROXY/HTTP_PROXY env vars."
+          example="http://user:pass@proxy.corp:8080, socks5h://127.0.0.1:9050"
         />
       </div>
 

apps/desktop/src/renderer/src/i18n/labels.ts

@@ -157,6 +157,7 @@ const TR_LABELS: Record<string, string> = {
   'Compression Missing': 'Sıkıştırma Eksik',
   'Keep-Alive Disabled': 'Keep-Alive Devre Dışı',
   'HTTP/2 Not Advertised': 'HTTP/2 İlan Edilmedi',
+  'HTTP/3 Not Advertised': 'HTTP/3 İlan Edilmedi',
   'Too Many Requests (>100 subresources)': 'Çok Fazla İstek (>100 alt kaynak)',
 
   // URL issues
@@ -496,6 +497,11 @@ const TR_LABELS: Record<string, string> = {
   'Content Kind': 'İçerik Sınıfı',
   'TTFB (ms)': 'TTFB (ms)',
   'HTTP Protocol': 'HTTP Protokolü',
+  'PDF Title': 'PDF Başlığı',
+  'PDF Author': 'PDF Yazarı',
+  'PDF Page Count': 'PDF Sayfa Sayısı',
+  'PDF Created': 'PDF Oluşturulma Tarihi',
+  'PDF Producer': 'PDF Üretici',
   Server: 'Sunucu',
   'Query String Length': 'Sorgu Dizesi Uzunluğu',
   'Render-Blocking (head)': 'Render-Engelleyici (head)',

apps/desktop/src/renderer/src/i18n/locales/en.json

@@ -199,6 +199,11 @@
     "multiCsvNote": "Multiple tables + CSV → you'll be asked to pick a folder; one file is written per table.",
     "folderNote": "Multiple sections + {{fmt}} → you'll be asked to pick a folder; one file is written per section, nested under sub-folders where applicable.",
     "xlsxNote": "Excel output → single .xlsx workbook with one sheet per section.",
+    "splitBySeverity": "Split by issue severity (Critical / Warning / Info)",
+    "severityNote": "Severity split overrides the table selection: three rollups (Critical / Warning / Info), each listing every URL flagged by an issue of that tier, using the columns picked at right.",
+    "severityCritical": "Critical Issues",
+    "severityWarning": "Warning Issues",
+    "severityInfo": "Info Issues",
     "summary": "{{n}} section(s) · {{c}} column(s)",
     "emptyCols": "Pick a table on the left to load its columns.",
     "errorNoColumn": "Pick at least one column.",
@@ -343,6 +348,7 @@
     "urlStructureThresholds": "URL Structure Thresholds",
     "linkFollowBehaviour": "Link Follow Behaviour",
     "savedProxyProfiles": "Saved Proxy Profiles",
+    "formLogin": "Form-Based Login",
     "ruleN": "Rule #{{n}}"
   },
   "settingsPanels": {
@@ -417,17 +423,22 @@
       "intro3": "for literal case-insensitive substring."
     },
     "customExtraction": {
-      "intro1": "Up to 10 custom extraction rules. Each runs against every crawled HTML page; results are stored on the URL row and visible in the URL Details panel under",
+      "intro1": "Up to 10 custom extraction rules. CSS / XPath / regex run on every crawled HTML page; JSONPath runs on JSON responses. Results are stored on the URL row and visible in the URL Details panel under",
       "extractionLabel": "Extraction",
       "empty": "No rules — click \"Add Rule\" to start.",
       "removeRule": "Remove rule",
       "name": "Name",
       "type": "Type",
       "cssSelector": "CSS Selector",
+      "xpath": "XPath",
+      "jsonpath": "JSONPath (JSON response)",
+      "xpathExpr": "XPath Expression",
+      "jsonpathExpr": "JSONPath Expression",
       "regex": "Regex",
       "regexPattern": "Regex Pattern",
       "output": "Output",
       "outputText": "Text",
+      "outputValue": "Value",
       "outputAttribute": "Attribute",
       "outputInnerHtml": "Inner HTML",
       "outputOuterHtml": "Outer HTML",
@@ -484,15 +495,28 @@
       "url": "Webhook URL"
     },
     "auth": {
-      "intro": "HTTP authentication applied on every request. Useful for staging environments behind Basic auth, or APIs that require a Bearer token. Digest is not supported (challenge-response state machine).",
+      "intro": "HTTP authentication applied on every request. Useful for staging environments behind Basic/Digest auth, or APIs that require a Bearer token. For sites behind an HTML login form, use Form Login below instead.",
       "scheme": "Auth scheme",
       "schemeNone": "None",
       "schemeBasic": "Basic (username + password)",
+      "schemeDigest": "Digest (username + password)",
       "schemeBearer": "Bearer (token)",
       "username": "Username",
       "password": "Password",
       "token": "Token"
     },
+    "formLogin": {
+      "enabled": "Enabled",
+      "intro": "Steps run in order over one shared cookie jar. A GET step can capture a CSRF token from the page; a POST step submits credentials (and the captured token) as a form. Reference captured values with {{name}}.",
+      "empty": "No steps — add one to define the login flow.",
+      "stepN": "Step #{{n}}",
+      "removeStep": "Remove step",
+      "fields": "Form fields (name = value)",
+      "addField": "Add field",
+      "captures": "Capture from response (name ← CSS selector)",
+      "addCapture": "Add capture",
+      "addStep": "Add login step"
+    },
     "network": {
       "intro": "Network-level controls: proxy override, file-extension exclusion, redirect hop cap.",
       "proxyUrl": "Proxy URL (overrides HTTPS_PROXY)",

apps/desktop/src/renderer/src/i18n/locales/tr.json

@@ -199,6 +199,11 @@
     "multiCsvNote": "Birden çok tablo + CSV → bir klasör seçmen istenecek; tablo başına bir dosya yazılır.",
     "folderNote": "Birden çok bölüm + {{fmt}} → bir klasör seçmen istenecek; her bölüm için bir dosya, uygun yerlerde alt-klasörler altında yazılır.",
     "xlsxNote": "Excel çıktı → tek .xlsx çalışma kitabı, bölüm başına bir sayfa.",
+    "splitBySeverity": "Önem derecesine göre böl (Kritik / Uyarı / Bilgi)",
+    "severityNote": "Önem derecesi bölünmesi tablo seçimini geçersiz kılar: üç derleme (Kritik / Uyarı / Bilgi), her biri o derecedeki bir sorunla işaretlenmiş tüm URL'leri sağdaki seçili kolonlarla listeler.",
+    "severityCritical": "Kritik Sorunlar",
+    "severityWarning": "Uyarı Sorunları",
+    "severityInfo": "Bilgi Sorunları",
     "summary": "{{n}} bölüm · {{c}} kolon",
     "emptyCols": "Solda bir tablo seç — kolonları buraya yüklensin.",
     "errorNoColumn": "En az bir kolon seç.",
@@ -342,6 +347,7 @@
     "urlStructureThresholds": "URL Yapı Eşikleri",
     "linkFollowBehaviour": "Link Takip Davranışı",
     "savedProxyProfiles": "Kayıtlı Proxy Profilleri",
+    "formLogin": "Form Tabanlı Giriş",
     "ruleN": "Kural #{{n}}"
   },
   "settingsPanels": {
@@ -416,17 +422,22 @@
       "intro3": "için harf duyarsız literal substring."
     },
     "customExtraction": {
-      "intro1": "10'a kadar özel çıkarım kuralı. Her biri her crawl edilmiş HTML sayfasına karşı çalışır; sonuçlar URL satırında saklanır ve URL Detayları panelinde",
+      "intro1": "10'a kadar özel çıkarım kuralı. CSS / XPath / regex her crawl edilmiş HTML sayfasında, JSONPath ise JSON yanıtlarında çalışır. Sonuçlar URL satırında saklanır ve URL Detayları panelinde",
       "extractionLabel": "Extraction",
       "empty": "Kural yok — başlamak için \"Add Rule\" tıkla.",
       "removeRule": "Kuralı kaldır",
       "name": "İsim",
       "type": "Tür",
       "cssSelector": "CSS Seçici",
+      "xpath": "XPath",
+      "jsonpath": "JSONPath (JSON yanıtı)",
+      "xpathExpr": "XPath İfadesi",
+      "jsonpathExpr": "JSONPath İfadesi",
       "regex": "Regex",
       "regexPattern": "Regex Kalıbı",
       "output": "Çıktı",
       "outputText": "Metin",
+      "outputValue": "Değer",
       "outputAttribute": "Öznitelik",
       "outputInnerHtml": "Inner HTML",
       "outputOuterHtml": "Outer HTML",
@@ -483,15 +494,28 @@
       "url": "Webhook URL"
     },
     "auth": {
-      "intro": "Her istekte uygulanan HTTP kimlik doğrulaması. Basic auth arkasındaki staging ortamları veya Bearer token gerektiren API'ler için faydalı. Digest desteklenmiyor (meydan okuma-yanıt durum makinesi).",
+      "intro": "Her istekte uygulanan HTTP kimlik doğrulaması. Basic/Digest auth arkasındaki staging ortamları veya Bearer token gerektiren API'ler için faydalı. HTML giriş formu arkasındaki siteler için aşağıdaki Form Login'i kullanın.",
       "scheme": "Auth şeması",
       "schemeNone": "Yok",
       "schemeBasic": "Basic (kullanıcı adı + şifre)",
+      "schemeDigest": "Digest (kullanıcı adı + şifre)",
       "schemeBearer": "Bearer (token)",
       "username": "Kullanıcı adı",
       "password": "Şifre",
       "token": "Token"
     },
+    "formLogin": {
+      "enabled": "Etkin",
+      "intro": "Adımlar tek bir paylaşılan çerez kavanozu üzerinde sırayla çalışır. GET adımı sayfadan bir CSRF token'ı yakalayabilir; POST adımı kimlik bilgilerini (ve yakalanan token'ı) form olarak gönderir. Yakalanan değerlere {{name}} ile başvurun.",
+      "empty": "Adım yok — giriş akışını tanımlamak için bir tane ekleyin.",
+      "stepN": "Adım #{{n}}",
+      "removeStep": "Adımı kaldır",
+      "fields": "Form alanları (ad = değer)",
+      "addField": "Alan ekle",
+      "captures": "Yanıttan yakala (ad ← CSS seçici)",
+      "addCapture": "Yakalama ekle",
+      "addStep": "Giriş adımı ekle"
+    },
     "network": {
       "intro": "Ağ seviyesi kontroller: proxy override, dosya-uzantısı hariç tutma, yönlendirme hop limiti.",
       "proxyUrl": "Proxy URL (HTTPS_PROXY'yi geçersiz kılar)",

apps/mcp-server/src/tools.ts

@@ -1484,7 +1484,7 @@ export function buildTools(): Tool[] {
       requiresDb: false,
       name: 'extraction_preview',
       description:
-        'Test custom extraction rules against a live URL fetch — same primitive as Settings → Custom Extraction "Preview". Returns per-rule values (or compile errors) plus the response\'s status / content-type / byte size / fetch ms. The agent can iterate on selectors without running a full crawl.',
+        'Test custom extraction rules against a live URL fetch — same primitive as Settings → Custom Extraction "Preview". Rule types: `css` (cheerio selector), `xpath` (XPath 1.0 subset over the DOM), `regex` (raw HTML), `jsonpath` (against a JSON response body). Returns per-rule values (or compile errors) plus the response\'s status / content-type / byte size / fetch ms. The agent can iterate on selectors without running a full crawl.',
       inputSchema: {
         type: 'object',
         properties: {
@@ -1495,7 +1495,7 @@ export function buildTools(): Tool[] {
               type: 'object',
               properties: {
                 name: { type: 'string' },
-                type: { type: 'string', enum: ['css', 'regex'] },
+                type: { type: 'string', enum: ['css', 'xpath', 'regex', 'jsonpath'] },
                 selector: { type: 'string' },
                 attribute: { type: 'string' },
                 output: { type: 'string', enum: ['text', 'attribute', 'inner_html', 'outer_html', 'count', 'regex_group'] },
@@ -1528,8 +1528,10 @@ export function buildTools(): Tool[] {
         bridgeRequest<unknown>('POST', '/v1/action/graph-snapshot', args),
     },
 
-    // ---- Faz 0.5 Increment 4 — Integration fetch (GSC + GA4 only;
-    // PSI/AI/SEO have progress streaming and stay in the desktop UI). ----
+    // ---- Faz 0.5 Increment 4 — Integration fetch (GSC + GA4 — single
+    // blocking API calls, no progress streaming). PSI / AI / SEO have
+    // multi-URL batches with progress + cancellation and are exposed via
+    // the non-blocking start/poll/cancel tools in Increment 5 below. ----
 
     {
       requiresDb: false,
@@ -1631,5 +1633,164 @@ export function buildTools(): Tool[] {
           meta: unknown;
         }>('POST', '/v1/action/ga4-fetch', args),
     },
+
+    // ---- Faz 0.5 Increment 5 — slow batch-fetch run triggers
+    // (PageSpeed / AI / SEO). These are the multi-URL fetchers that
+    // stream progress + support cancellation. The MCP wire protocol
+    // can't stream, so — exactly like start_crawl — each *_run tool
+    // returns immediately after kicking off the batch; poll
+    // get_fetch_progress to watch it and cancel_fetch to stop. Target
+    // URLs come from an explicit `urls` list OR a `category` resolved
+    // server-side (preferred for large sets — the bridge caps request
+    // bodies at 64 KB, so a big `urls` array is rejected). ----
+
+    {
+      requiresDb: false,
+      name: 'pagespeed_run',
+      description:
+        'Kick off a PageSpeed Insights (Lighthouse) batch audit in the desktop app and return immediately — it does NOT block (audits take ~10–30 s each). Poll `get_fetch_progress` (kind "pagespeed") to watch it and `cancel_fetch` to stop. Pick targets with an explicit `urls` array OR a `category` resolved server-side (preferred for large sets). Keyless PageSpeed has a near-zero daily quota — add a free key under Settings → Integrations → PageSpeed Insights for real throughput. Results land where `query_pagespeed` / `get_url_analytics` read. Returns `{started, total, truncated, state}`; `started:false` + `reason` when a run is already active or nothing matched.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          urls: {
+            type: 'array',
+            items: { type: 'string' },
+            description: 'Explicit URLs to audit. Wins over `category`. Keep the list small — the bridge caps the request body at 64 KB.',
+          },
+          category: {
+            type: 'string',
+            enum: URL_CATEGORY_VALUES,
+            description: 'Resolve target URLs from the active project by category (e.g. "internal:html") when `urls` is omitted.',
+          },
+          search: {
+            type: 'string',
+            description: 'Optional substring filter applied when resolving via `category`.',
+          },
+          limit: {
+            type: 'integer',
+            minimum: 1,
+            maximum: 5000,
+            description: 'Max URLs to pull when resolving via `category`. Default 100.',
+          },
+          strategy: {
+            type: 'string',
+            enum: ['mobile', 'desktop', 'both'],
+            description: 'Form factor. `both` doubles the API calls. Default "mobile".',
+          },
+        },
+      },
+      handler: async (args) =>
+        bridgeRequest<unknown>('POST', '/v1/action/pagespeed-run', args),
+    },
+
+    {
+      requiresDb: false,
+      name: 'ai_run',
+      description:
+        'Kick off an AI batch over the matched URLs and return immediately (non-blocking). Each page\'s fields can be interpolated into `prompt` via {url}/{title}/{description}/{h1}/{body}. Poll `get_fetch_progress` (kind "ai") and `cancel_fetch` to stop. COSTS REAL MONEY for hosted providers (openai/anthropic) — one API call per matched URL — so scope tightly with `category` + `limit` first. The provider must be configured in Settings → AI. URLs not crawled in this project are skipped (no page context). Results land where `query_ai` reads. Returns `{started, total, truncated, state}`.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          provider: {
+            type: 'string',
+            enum: ['openai', 'anthropic', 'ollama'],
+          },
+          prompt: {
+            type: 'string',
+            description: 'Prompt template, substituted per URL. Supports {url}/{title}/{description}/{h1}/{body} placeholders.',
+          },
+          model: {
+            type: 'string',
+            description: 'Optional model override; blank uses the provider default.',
+          },
+          urls: {
+            type: 'array',
+            items: { type: 'string' },
+            description: 'Explicit URLs. Wins over `category`. Keep small (64 KB body cap).',
+          },
+          category: {
+            type: 'string',
+            enum: URL_CATEGORY_VALUES,
+            description: 'Resolve targets by category when `urls` is omitted.',
+          },
+          search: { type: 'string' },
+          limit: {
+            type: 'integer',
+            minimum: 1,
+            maximum: 5000,
+            description: 'Max URLs when resolving via `category`. Default 100.',
+          },
+        },
+        required: ['provider', 'prompt'],
+      },
+      handler: async (args) =>
+        bridgeRequest<unknown>('POST', '/v1/action/ai-run', args),
+    },
+
+    {
+      requiresDb: false,
+      name: 'seo_run',
+      description:
+        'Kick off an SEO-authority batch (domain authority / backlinks / referring domains) over the matched URLs and return immediately (non-blocking). Poll `get_fetch_progress` (kind "seo") and `cancel_fetch` to stop. The provider must be configured in Settings → SEO Authority. Results land where `query_seo` reads. Returns `{started, total, truncated, state}`.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          provider: {
+            type: 'string',
+            enum: ['ahrefs', 'majestic', 'moz', 'semrush'],
+          },
+          urls: {
+            type: 'array',
+            items: { type: 'string' },
+            description: 'Explicit URLs. Wins over `category`. Keep small (64 KB body cap).',
+          },
+          category: {
+            type: 'string',
+            enum: URL_CATEGORY_VALUES,
+            description: 'Resolve targets by category when `urls` is omitted.',
+          },
+          search: { type: 'string' },
+          limit: {
+            type: 'integer',
+            minimum: 1,
+            maximum: 5000,
+            description: 'Max URLs when resolving via `category`. Default 100.',
+          },
+        },
+        required: ['provider'],
+      },
+      handler: async (args) =>
+        bridgeRequest<unknown>('POST', '/v1/action/seo-run', args),
+    },
+
+    {
+      requiresDb: false,
+      name: 'get_fetch_progress',
+      description:
+        'Poll the live state of the PageSpeed / AI / SEO batch fetchers started by pagespeed_run / ai_run / seo_run (or by the desktop UI — the snapshot is shared). Returns per-kind `{running, done, total, completed, failed, currentUrl, cancelled, startedAt, finishedAt, provider, error}`. Omit `kind` for all three; pass one to narrow. Poll every 1–2 s; once `running` flips to false the batch is done. This is the fetch-side analogue of get_crawl_progress.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          kind: { type: 'string', enum: ['pagespeed', 'ai', 'seo'] },
+        },
+      },
+      handler: async (args) =>
+        bridgeRequest<unknown>('POST', '/v1/action/fetch-progress', args),
+    },
+
+    {
+      requiresDb: false,
+      name: 'cancel_fetch',
+      description:
+        'Request cooperative cancellation of a running batch fetcher. In-flight items finish and are saved; queued items are skipped. Pass `kind` ("pagespeed" / "ai" / "seo") to cancel one, or omit to cancel all three. Returns which kinds were running when the cancel landed. No-op for kinds that are idle.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          kind: { type: 'string', enum: ['pagespeed', 'ai', 'seo'] },
+        },
+      },
+      handler: async (args) =>
+        bridgeRequest<unknown>('POST', '/v1/action/fetch-cancel', args),
+    },
   ];
 }

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "freecrawl-seo",
-  "version": "0.6.8",
+  "version": "0.7.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "freecrawl-seo",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "hasInstallScript": true,
       "license": "MIT",
       "workspaces": [
@@ -26,7 +26,7 @@
     },
     "apps/cli": {
       "name": "@freecrawl/cli",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "dependencies": {
         "@freecrawl/core": "*",
         "@freecrawl/db": "*",
@@ -43,7 +43,7 @@
     },
     "apps/desktop": {
       "name": "@freecrawl/desktop",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "dependencies": {
         "@dnd-kit/core": "^6.3.1",
         "@dnd-kit/sortable": "^10.0.0",
@@ -62,6 +62,7 @@
         "react-dom": "^19.2.0",
         "react-i18next": "^17.0.8",
         "react-resizable-panels": "^2.1.9",
+        "undici": "^8.0.0",
         "zustand": "^5.0.2"
       },
       "devDependencies": {
@@ -90,7 +91,7 @@
     },
     "apps/mcp-server": {
       "name": "@freecrawl/mcp-server",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*"
@@ -5653,6 +5654,15 @@
       "dev": true,
       "license": "ISC"
     },
+    "node_modules/ip-address": {
+      "version": "10.2.0",
+      "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.2.0.tgz",
+      "integrity": "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 12"
+      }
+    },
     "node_modules/is-binary-path": {
       "version": "2.1.0",
       "resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz",
@@ -7558,14 +7568,26 @@
       "version": "4.2.0",
       "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz",
       "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==",
-      "dev": true,
       "license": "MIT",
-      "optional": true,
       "engines": {
         "node": ">= 6.0.0",
         "npm": ">= 3.0.0"
       }
     },
+    "node_modules/socks": {
+      "version": "2.8.9",
+      "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.9.tgz",
+      "integrity": "sha512-LJhUYUvItdQ0LkJTmPeaEObWXAqFyfmP85x0tch/ez9cahmhlBBLbIqDFnvBnUJGagb0JbIQrkBs1wJ+yRYpEw==",
+      "license": "MIT",
+      "dependencies": {
+        "ip-address": "^10.1.1",
+        "smart-buffer": "^4.2.0"
+      },
+      "engines": {
+        "node": ">= 10.0.0",
+        "npm": ">= 3.0.0"
+      }
+    },
     "node_modules/source-map": {
       "version": "0.6.1",
       "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",
@@ -8934,15 +8956,17 @@
     },
     "packages/core": {
       "name": "@freecrawl/core",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*",
         "cacheable-lookup": "^7.0.0",
         "cheerio": "^1.0.0",
+        "domhandler": "^5.0.3",
         "p-queue": "^8.0.1",
         "playwright": "^1.49.0",
         "robots-parser": "^3.0.1",
+        "socks": "^2.8.3",
         "undici": "^8.0.0"
       },
       "devDependencies": {
@@ -8961,7 +8985,7 @@
     },
     "packages/db": {
       "name": "@freecrawl/db",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "dependencies": {
         "@freecrawl/shared-types": "*"
       },
@@ -8973,7 +8997,7 @@
     },
     "packages/shared-types": {
       "name": "@freecrawl/shared-types",
-      "version": "0.6.8",
+      "version": "0.7.0",
       "devDependencies": {
         "rimraf": "^6.0.1",
         "typescript": "^5.7.0"

packages/core/package.json

@@ -21,9 +21,11 @@
     "@freecrawl/shared-types": "*",
     "cacheable-lookup": "^7.0.0",
     "cheerio": "^1.0.0",
+    "domhandler": "^5.0.3",
     "p-queue": "^8.0.1",
     "playwright": "^1.49.0",
     "robots-parser": "^3.0.1",
+    "socks": "^2.8.3",
     "undici": "^8.0.0"
   },
   "devDependencies": {

packages/core/src/crawler.ts

@@ -1,6 +1,7 @@
 import { EventEmitter } from 'node:events';
 import * as os from 'node:os';
 import { fetch as undiciFetch } from 'undici';
+import { load as cheerioLoad } from 'cheerio';
 import PQueue from 'p-queue';
 import type {
   ContentKind,
@@ -26,14 +27,18 @@ import { parseHtml, estimatePixelWidth } from './html-parser.js';
 import { analyseCookies, extractSetCookies } from './cookies.js';
 import { loadRobots, type RobotsChecker } from './robots.js';
 import {
+  buildDigestAuthHeader,
   collectNetworkDiagnostics,
   defaultRequestHeaders,
   detectHttpProtocol,
   formatFetchError,
   initHttpClient,
+  parseDigestChallenge,
 } from './http-client.js';
 import { setActiveDnsHook } from './dns-resolver.js';
 import { discoverSitemapUrls, fetchSitemaps } from './sitemap.js';
+import { runJsonExtractionRules } from './extraction.js';
+import { SessionCookieJar } from './cookie-jar.js';
 
 export interface CrawlerEvents {
   progress: (p: CrawlProgress) => void;
@@ -100,6 +105,9 @@ export class Crawler extends EventEmitter {
    * adds itself in `try` and removes itself in `finally`.
    */
   private inFlightFetchControllers = new Set<AbortController>();
+  /** Authenticated session cookies established by the pre-crawl form-login
+   *  sequence, replayed on every request. Null when form login is off. */
+  private sessionJar: SessionCookieJar | null = null;
   private robots: RobotsChecker | null = null;
   private progressTimer: NodeJS.Timeout | null = null;
   private memoryWatchdogTimer: NodeJS.Timeout | null = null;
@@ -518,6 +526,11 @@ export class Crawler extends EventEmitter {
     // disabled TLS validation, all of which affect what crawls can reach.
     this.emitEnvDiagnostics();
 
+    // V2 Faz 2 — establish an authenticated session before any fetch when
+    // form-based login is configured. Best-effort; the crawl proceeds
+    // either way. Runs for both spider and list mode.
+    await this.runFormLogin();
+
     if (this.config.mode === 'list') {
       await this.startListMode();
       return;
@@ -805,6 +818,9 @@ export class Crawler extends EventEmitter {
     this.setOp('post-crawl:social-image-probes');
     await this.runSocialImageProbes();
     await yieldToEventLoop();
+    this.setOp('post-crawl:pdf-probes');
+    await this.runPdfMetadataProbes();
+    await yieldToEventLoop();
     this.setOp('post-crawl:performance-budget');
     this.runBudgetPass();
     this.running = false;
@@ -1129,6 +1145,100 @@ export class Crawler extends EventEmitter {
     );
   }
 
+  /**
+   * V2 Faz 16 — extract document metadata (title / author / page count /
+   * creation date / producer) from internal PDFs. Mirrors the manifest /
+   * social-image probe shape, but PDFs are their own URL rows so each
+   * result is a plain per-row update (no fan-out). Small concurrency pool;
+   * each probe is ranged + best-effort so one bad PDF never aborts the
+   * pass. `pdf_probe_status` records 1 / 0 / negative so a probed PDF
+   * drops out of `unprobedPdfUrls` and isn't re-fetched on the next crawl.
+   */
+  private async runPdfMetadataProbes(): Promise<void> {
+    if (!this.config.probePdfMetadata) return;
+    if (this.stopped) return;
+    let urls: string[] = [];
+    try {
+      urls = this.db.unprobedPdfUrls(2_000);
+    } catch (err) {
+      this.emit(
+        'info',
+        `PDF metadata probe skipped (DB query failed): ${
+          err instanceof Error ? err.message : String(err)
+        }`,
+      );
+      return;
+    }
+    if (urls.length === 0) return;
+    this.emit('info', `Probing ${urls.length} PDF(s) for metadata…`);
+
+    const { probePdfMetadata } = await import('./pdf-probe.js');
+    const concurrency = Math.max(1, Math.min(this.config.maxConcurrency, 6));
+    let cursor = 0;
+    let withMeta = 0;
+    let failed = 0;
+
+    const worker = async (): Promise<void> => {
+      while (!this.stopped) {
+        const idx = cursor++;
+        if (idx >= urls.length) return;
+        const url = urls[idx];
+        if (!url) return;
+        try {
+          const result = await probePdfMetadata(url, {
+            userAgent: this.resolveUserAgent(url),
+            acceptLanguage: this.config.acceptLanguage,
+            customHeaders: this.config.customHeaders,
+            auth: this.config.auth,
+          });
+          const hasMeta =
+            result.title !== null ||
+            result.author !== null ||
+            result.pageCount !== null ||
+            result.creationDate !== null ||
+            result.producer !== null;
+          // 1 = metadata found, 0 = probed-but-empty, -1 = fetch/parse error.
+          const status = result.error ? -1 : hasMeta ? 1 : 0;
+          this.db.setPdfMetadata({
+            url,
+            title: result.title,
+            author: result.author,
+            pageCount: result.pageCount,
+            creationDate: result.creationDate,
+            producer: result.producer,
+            status,
+          });
+          if (result.error) failed++;
+          else if (hasMeta) withMeta++;
+        } catch {
+          // Couldn't even fetch/record — mark errored so we don't loop on it.
+          try {
+            this.db.setPdfMetadata({
+              url,
+              title: null,
+              author: null,
+              pageCount: null,
+              creationDate: null,
+              producer: null,
+              status: -1,
+            });
+          } catch {
+            /* best-effort */
+          }
+          failed++;
+        }
+      }
+    };
+
+    const workers: Promise<void>[] = [];
+    for (let i = 0; i < concurrency; i++) workers.push(worker());
+    await Promise.all(workers);
+    this.emit(
+      'info',
+      `PDF metadata probe complete: ${withMeta}/${urls.length} with metadata, ${failed} failed`,
+    );
+  }
+
   /**
    * V2 Faz 15 — performance budget pass. Stamps each internal 200 HTML
    * page with a bitmask of which configured ceilings it exceeded
@@ -1372,6 +1482,8 @@ export class Crawler extends EventEmitter {
     await yieldToEventLoop();
     await this.runSocialImageProbes();
     await yieldToEventLoop();
+    await this.runPdfMetadataProbes();
+    await yieldToEventLoop();
     this.runBudgetPass();
     this.running = false;
     this.stopMemoryMonitor();
@@ -2224,11 +2336,32 @@ export class Crawler extends EventEmitter {
       }
 
       if (kind !== 'html' || statusCode >= 400) {
+        // Non-HTML / error responses are normally body-less in our model.
+        // Exception: JSON responses (`application/json` APIs) are routed to
+        // the JSONPath custom-extraction path — the only place non-HTML
+        // bodies are parsed. We capture the body once so a configured
+        // jsonpath rule can read it; otherwise it's drained and discarded.
+        const needsJsonExtraction =
+          statusCode < 400 &&
+          (this.config.customExtractionRules ?? []).some((r) => r.type === 'jsonpath');
+        let nonHtmlBody = '';
         try {
-          await res.text();
+          nonHtmlBody = await res.text();
         } catch {
           /* ignore */
         }
+        let jsonExtraction: Record<string, unknown> | null = null;
+        if (needsJsonExtraction) {
+          const ctLower = (contentType ?? '').toLowerCase();
+          const looksJson = ctLower.includes('json') || /^\s*[[{]/.test(nonHtmlBody);
+          if (looksJson) {
+            try {
+              jsonExtraction = runJsonExtractionRules(nonHtmlBody, this.config.customExtractionRules);
+            } catch {
+              jsonExtraction = null;
+            }
+          }
+        }
         const indexability: Indexability =
           statusCode >= 500
             ? 'non-indexable:server-error'
@@ -2245,6 +2378,7 @@ export class Crawler extends EventEmitter {
             indexability,
             indexabilityReason: indexability === 'indexable' ? null : `HTTP ${statusCode}`,
             contentType,
+            extractionResults: jsonExtraction ? JSON.stringify(jsonExtraction) : null,
             contentLength: parseIntSafe(contentLengthHeader),
             xRobotsTag,
             responseTimeMs,
@@ -2774,6 +2908,163 @@ export class Crawler extends EventEmitter {
     return this.config.userAgent;
   }
 
+  /**
+   * Build request headers for a crawl fetch: defaults + auth + any
+   * established session cookies. A user-supplied `Cookie` header (via
+   * customHeaders) always wins over the session jar; `extra` (e.g. a
+   * computed Digest Authorization) is applied last.
+   */
+  private requestHeaders(url: string, extra?: Record<string, string>): Record<string, string> {
+    const headers = defaultRequestHeaders(
+      this.resolveUserAgent(url),
+      this.config.acceptLanguage,
+      this.config.customHeaders,
+      this.config.auth,
+    );
+    if (this.sessionJar) {
+      const hasCookie = Object.keys(headers).some((k) => k.toLowerCase() === 'cookie');
+      if (!hasCookie) {
+        const ck = this.sessionJar.cookieHeader(url);
+        if (ck) headers['cookie'] = ck;
+      }
+    }
+    if (extra) Object.assign(headers, extra);
+    return headers;
+  }
+
+  /** Capture Set-Cookie from a response into the session jar so a rotating
+   *  login session stays alive. No-op when form login is off. */
+  private refreshSessionJar(url: string, res: Response): void {
+    if (!this.sessionJar) return;
+    try {
+      const setCookies =
+        (res.headers as { getSetCookie?: () => string[] }).getSetCookie?.() ?? [];
+      if (setCookies.length) this.sessionJar.setFromResponse(url, setCookies);
+    } catch {
+      /* best-effort */
+    }
+  }
+
+  /**
+   * Execute the form-based login sequence before the crawl starts. Steps
+   * run in order over one shared cookie jar; each step can capture values
+   * (CSRF tokens, etc.) from its response via CSS selectors for `{{var}}`
+   * interpolation in later steps. On success the jar is installed so its
+   * cookies are replayed on every crawl request. Best-effort — a failed
+   * step is logged and the crawl proceeds unauthenticated.
+   */
+  private async runFormLogin(): Promise<void> {
+    const cfg = this.config.formLogin;
+    if (!cfg || !cfg.enabled || cfg.steps.length === 0) return;
+
+    const jar = new SessionCookieJar();
+    const vars: Record<string, string> = {};
+    const interp = (s: string): string =>
+      s.replace(/\{\{\s*([\w.-]+)\s*\}\}/g, (_m, name: string) => vars[name] ?? '');
+    const cookieHeaderFor = (target: string): Record<string, string> => {
+      const h = defaultRequestHeaders(
+        this.config.userAgent,
+        this.config.acceptLanguage,
+        this.config.customHeaders,
+        this.config.auth,
+      );
+      const ck = jar.cookieHeader(target);
+      if (ck && !Object.keys(h).some((k) => k.toLowerCase() === 'cookie')) h['cookie'] = ck;
+      return h;
+    };
+
+    this.setOp('crawl:login');
+    for (let i = 0; i < cfg.steps.length; i++) {
+      const step = cfg.steps[i]!;
+      const stepUrl = interp(step.url).trim();
+      if (!stepUrl) continue;
+      const controller = new AbortController();
+      const timer = setTimeout(
+        () => controller.abort(),
+        Math.max(5000, this.config.requestTimeoutMs),
+      );
+      try {
+        const headers = cookieHeaderFor(stepUrl);
+        let body: string | undefined;
+        if (step.method === 'POST') {
+          const form = new URLSearchParams();
+          for (const f of step.fields) {
+            if (f.name.trim()) form.append(f.name, interp(f.value));
+          }
+          body = form.toString();
+          headers['content-type'] = 'application/x-www-form-urlencoded';
+        }
+        let current = await undiciFetch(stepUrl, {
+          method: step.method,
+          headers,
+          body,
+          redirect: 'manual',
+          signal: controller.signal,
+        });
+        // Capture cookies, then follow up to 5 redirects manually so the
+        // Set-Cookie on a post-login 30x is captured too.
+        let hops = 0;
+        for (;;) {
+          const sc =
+            (current.headers as { getSetCookie?: () => string[] }).getSetCookie?.() ?? [];
+          if (sc.length) jar.setFromResponse(stepUrl, sc);
+          const loc = current.headers.get('location');
+          if (!loc || hops >= 5 || current.status < 300 || current.status >= 400) break;
+          hops++;
+          const next = new URL(loc, stepUrl).toString();
+          try {
+            await current.body?.cancel();
+          } catch {
+            /* ignore */
+          }
+          current = await undiciFetch(next, {
+            method: 'GET',
+            headers: cookieHeaderFor(next),
+            redirect: 'manual',
+            signal: controller.signal,
+          });
+        }
+        if (step.captures.length) {
+          const html = await current.text().catch(() => '');
+          if (html) {
+            const $ = cheerioLoad(html);
+            for (const cap of step.captures) {
+              if (!cap.name.trim() || !cap.selector.trim()) continue;
+              const el = $(cap.selector).first();
+              const attr = (cap.attribute ?? 'value').trim() || 'value';
+              const val = attr.toLowerCase() === 'text' ? el.text() : el.attr(attr);
+              if (val !== undefined && val !== null) vars[cap.name.trim()] = String(val).trim();
+            }
+          }
+        } else {
+          try {
+            await current.body?.cancel();
+          } catch {
+            /* ignore */
+          }
+        }
+        this.emit(
+          'info',
+          `Form login step ${i + 1}/${cfg.steps.length}: ${step.method} ${stepUrl} -> ${current.status}`,
+        );
+      } catch (err) {
+        this.emit(
+          'warn',
+          `Form login step ${i + 1} failed (${stepUrl}): ${formatFetchError(err)} — continuing unauthenticated`,
+        );
+      } finally {
+        clearTimeout(timer);
+      }
+    }
+
+    if (jar.size > 0) {
+      this.sessionJar = jar;
+      this.emit('info', `Form login established ${jar.size} session cookie(s).`);
+    } else {
+      this.emit('warn', 'Form login produced no cookies — the crawl will run unauthenticated.');
+    }
+  }
+
   private async fetchWithRetry(
     url: string,
     signal: AbortSignal,
@@ -2799,17 +3090,44 @@ export class Crawler extends EventEmitter {
           this.config.renderingMode === 'ajax'
             ? toEscapedFragmentUrl(url)
             : url;
-        const res = await undiciFetch(fetchUrl, {
+        let res = await undiciFetch(fetchUrl, {
           method: 'GET',
-          headers: defaultRequestHeaders(
-            this.resolveUserAgent(url),
-            this.config.acceptLanguage,
-            this.config.customHeaders,
-            this.config.auth,
-          ),
+          headers: this.requestHeaders(url),
           redirect: 'manual',
           signal,
         });
+        // HTTP Digest — the first 401 carries the `WWW-Authenticate: Digest`
+        // challenge; recompute the Authorization header from it and retry
+        // once. Only when digest auth is configured with a username.
+        if (
+          res.status === 401 &&
+          this.config.auth.type === 'digest' &&
+          (this.config.auth.username ?? '')
+        ) {
+          const challenge = parseDigestChallenge(res.headers.get('www-authenticate'));
+          if (challenge) {
+            try {
+              await res.body?.cancel();
+            } catch {
+              /* ignore */
+            }
+            const u = new URL(fetchUrl);
+            const authHeader = buildDigestAuthHeader(challenge, {
+              method: 'GET',
+              uri: u.pathname + u.search,
+              username: this.config.auth.username ?? '',
+              password: this.config.auth.password ?? '',
+            });
+            res = await undiciFetch(fetchUrl, {
+              method: 'GET',
+              headers: this.requestHeaders(url, { authorization: authHeader }),
+              redirect: 'manual',
+              signal,
+            });
+          }
+        }
+        // Keep the login session alive — capture any rotated session cookies.
+        this.refreshSessionJar(url, res);
         const ttfbMs = Date.now() - tStart;
         // Final attempt or non-retryable status — return as-is. We attach
         // ttfbMs as a non-enumerable property so the existing call sites

packages/core/src/extraction.ts

@@ -1,6 +1,8 @@
 import * as cheerio from 'cheerio';
 import type { CheerioAPI } from 'cheerio';
 import type { CustomExtractionRule } from '@freecrawl/shared-types';
+import { evaluateXPath } from './xpath-lite.js';
+import { evaluateJsonPath } from './jsonpath-lite.js';
 
 /**
  * Run all configured Custom Extraction rules against a single page.
@@ -33,12 +35,18 @@ export function runExtractionRules(
     const name = (rule.name ?? '').trim();
     if (!name) continue;
 
+    // `jsonpath` rules only apply to non-HTML JSON responses — they're run
+    // by runJsonExtractionRules from the crawler's non-HTML branch, not here.
+    if (rule.type === 'jsonpath') continue;
+
     let value: unknown = null;
     try {
       if (rule.type === 'css') {
         value = runCssRule($, rule);
       } else if (rule.type === 'regex') {
         value = runRegexRule(rawHtml, rule);
+      } else if (rule.type === 'xpath') {
+        value = runXpathRule($, rule);
       }
     } catch {
       // Per-rule failure isolation — bad selector shouldn't poison the
@@ -74,6 +82,23 @@ export function previewExtractionRules(
 ): Array<{ name: string; value: unknown; error?: string }> {
   if (rules.length === 0) return [];
   const $ = cheerio.load(rawHtml);
+  // JSON body parsed lazily and only once — preview works against both HTML
+  // pages (css/regex/xpath) and JSON endpoints (jsonpath).
+  let jsonParsed = false;
+  let jsonValue: unknown = undefined;
+  let jsonError: string | null = null;
+  const getJson = (): unknown => {
+    if (!jsonParsed) {
+      jsonParsed = true;
+      try {
+        jsonValue = JSON.parse(rawHtml);
+      } catch {
+        jsonError = 'response body is not valid JSON';
+      }
+    }
+    if (jsonError) throw new Error(jsonError);
+    return jsonValue;
+  };
   const out: Array<{ name: string; value: unknown; error?: string }> = [];
   for (const rule of rules) {
     const name = (rule.name ?? '').trim() || '(unnamed)';
@@ -83,6 +108,10 @@ export function previewExtractionRules(
         value = runCssRule($, rule);
       } else if (rule.type === 'regex') {
         value = runRegexRule(rawHtml, rule);
+      } else if (rule.type === 'xpath') {
+        value = runXpathRule($, rule);
+      } else if (rule.type === 'jsonpath') {
+        value = runJsonPathRule(getJson(), rule);
       }
       out.push({ name, value });
     } catch (e) {
@@ -93,6 +122,44 @@ export function previewExtractionRules(
   return out;
 }
 
+/**
+ * Crawler-side variant of {@link runExtractionRules} for **non-HTML JSON
+ * responses**. Only `jsonpath` rules apply; everything else is skipped.
+ * Returns the same `{ ruleName: value }` map shape (or `null` when the body
+ * isn't JSON or nothing matched), so it stores into the same
+ * `extraction_results` column.
+ */
+export function runJsonExtractionRules(
+  rawBody: string,
+  rules: ReadonlyArray<CustomExtractionRule>,
+): Record<string, unknown> | null {
+  const jsonRules = rules.filter((r) => r.type === 'jsonpath');
+  if (jsonRules.length === 0) return null;
+  let json: unknown;
+  try {
+    json = JSON.parse(rawBody);
+  } catch {
+    return null; // not JSON — nothing to extract
+  }
+  const out: Record<string, unknown> = {};
+  let anyMatch = false;
+  for (const rule of jsonRules) {
+    const name = (rule.name ?? '').trim();
+    if (!name) continue;
+    let value: unknown = null;
+    try {
+      value = runJsonPathRule(json, rule);
+    } catch {
+      value = null;
+    }
+    out[name] = value;
+    if (value !== null && value !== undefined && value !== 0 && value !== '') {
+      anyMatch = true;
+    }
+  }
+  return anyMatch ? out : null;
+}
+
 function runCssRule($: CheerioAPI, rule: CustomExtractionRule): unknown {
   const $els = $(rule.selector);
   const total = $els.length;
@@ -165,3 +232,64 @@ function runRegexRule(rawHtml: string, rule: CustomExtractionRule): unknown {
   if (rule.multi === 'concat') return all.join(' | ');
   return all[0] ?? null;
 }
+
+function runXpathRule($: CheerioAPI, rule: CustomExtractionRule): unknown {
+  const res = evaluateXPath($, rule.selector);
+  const total = res.kind === 'nodes' ? res.nodes.length : res.values.length;
+
+  if (rule.output === 'count' || rule.multi === 'count') return total;
+  if (total === 0) return null;
+
+  const readOne = (i: number): string | null => {
+    // `@attr` / `text()` terminals already produced string values; the
+    // output mode only applies when the XPath selected elements.
+    if (res.kind === 'values') return res.values[i] ?? null;
+    const $el = $(res.nodes[i]!);
+    switch (rule.output) {
+      case 'attribute': {
+        if (!rule.attribute) return null;
+        const v = $el.attr(rule.attribute);
+        return v === undefined ? null : v;
+      }
+      case 'inner_html':
+        return $el.html() ?? null;
+      case 'outer_html': {
+        const v = $.html($el);
+        return v ?? null;
+      }
+      case 'text':
+      default:
+        return $el.text().replace(/\s+/g, ' ').trim() || null;
+    }
+  };
+
+  if (rule.multi === 'first') return readOne(0);
+  if (rule.multi === 'last') return readOne(total - 1);
+
+  const all: string[] = [];
+  for (let i = 0; i < total; i++) {
+    const v = readOne(i);
+    if (v !== null) all.push(v);
+  }
+  if (rule.multi === 'all') return all;
+  if (rule.multi === 'concat') return all.join(' | ');
+  return all[0] ?? null;
+}
+
+function runJsonPathRule(json: unknown, rule: CustomExtractionRule): unknown {
+  const matches = evaluateJsonPath(json, rule.selector);
+  const total = matches.length;
+
+  if (rule.output === 'count' || rule.multi === 'count') return total;
+  if (total === 0) return null;
+
+  // JSON values are kept as-is (object/array/number/string/bool); the
+  // crawler JSON-stringifies the whole extraction map for storage.
+  if (rule.multi === 'first') return matches[0] ?? null;
+  if (rule.multi === 'last') return matches[total - 1] ?? null;
+  if (rule.multi === 'all') return matches;
+  if (rule.multi === 'concat') {
+    return matches.map((v) => (typeof v === 'string' ? v : JSON.stringify(v))).join(' | ');
+  }
+  return matches[0] ?? null;
+}

packages/core/src/html-report.ts

@@ -153,6 +153,7 @@ const ISSUES: IssueDef[] = [
   { key: 'queryStringTooLong', label: 'Query String >100 Chars', severity: 'info' },
   { key: 'folderDepthTooDeep', label: 'Folder Depth >4', severity: 'info' },
   { key: 'http2NotSupported', label: 'HTTP/2 Not Advertised', severity: 'info' },
+  { key: 'http3NotSupported', label: 'HTTP/3 Not Advertised', severity: 'info' },
   { key: 'renderBlocking', label: 'Render-Blocking Head Resources >5', severity: 'warn' },
   { key: 'keepaliveDisabled', label: 'Keep-Alive Disabled (Connection: close)', severity: 'warn' },
   { key: 'titlePlaceholder', label: 'Title Placeholder (Untitled / Default)', severity: 'error' },

packages/core/src/http-client.ts

@@ -1,8 +1,13 @@
 import dns from 'node:dns';
-import { Agent, ProxyAgent, setGlobalDispatcher } from 'undici';
+import crypto from 'node:crypto';
+import { Agent, ProxyAgent, buildConnector, setGlobalDispatcher } from 'undici';
+import { SocksClient, type SocksProxy } from 'socks';
 import { createResilientLookup, type DnsResolverHook } from './dns-resolver.js';
 
-let initialized = false;
+let dnsInitialized = false;
+/** Effective proxy applied to the global dispatcher on the last call, so we
+ *  only rebuild when it actually changes (a second crawl can switch proxy). */
+let lastProxyKey: string | null = null;
 
 /**
  * Configure the global undici dispatcher and Node DNS once per process.
@@ -19,21 +24,27 @@ let initialized = false;
  * - If HTTPS_PROXY / HTTP_PROXY env vars are set (corporate networks),
  *   route through ProxyAgent so packaged-app users don't get ECONNREFUSED
  *   against origins they can only reach via their company proxy.
+ * - SOCKS proxies (`socks5://`, `socks5h://`, `socks4://`, `socks4a://`)
+ *   are routed through a custom undici Agent whose `connect` dials the
+ *   destination over a SOCKS tunnel (TLS upgraded for https origins).
  * - The Agent is tuned for crawler-style workloads: many concurrent
  *   connections per origin, long keep-alive, tight headers timeout so a
  *   stuck origin can't freeze the pool.
+ *
+ * Idempotent per effective-proxy: DNS is configured once, and the global
+ * dispatcher is only rebuilt when the resolved proxy changes — so a second
+ * crawl can switch to a different proxy instead of the first crawl's
+ * dispatcher winning for the whole process lifetime.
  */
 export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsResolverHook } = {}): void {
-  if (initialized) return;
-  initialized = true;
-
-  dns.setDefaultResultOrder('ipv4first');
-
-  const lookup = createResilientLookup({ onEvent: opts.onDnsEvent });
+  if (!dnsInitialized) {
+    dnsInitialized = true;
+    dns.setDefaultResultOrder('ipv4first');
+  }
 
   // Corporate proxy detection — env vars are the universal contract,
   // matching curl / git / npm / pip behaviour. A non-empty config
-  // override (Settings → Auth → Proxy URL) takes precedence so the
+  // override (Settings → Network → Proxy URL) takes precedence so the
   // user can route a single project through a different proxy.
   // ECMAScript forbids mixing `||` with `??` in the same expression
   // without parentheses (SyntaxError on parse). The `??` chain must be
@@ -44,14 +55,23 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
     process.env['HTTP_PROXY'] ??
     process.env['http_proxy'] ??
     null;
-  const proxyUrl =
-    (opts.proxyOverride && opts.proxyOverride.trim()) || envProxy;
+  const proxyUrl = (opts.proxyOverride && opts.proxyOverride.trim()) || envProxy || '';
+
+  const key = proxyUrl || '<direct>';
+  if (key === lastProxyKey) return;
+  lastProxyKey = key;
 
   if (proxyUrl) {
+    const scheme = proxyUrl.slice(0, Math.max(0, proxyUrl.indexOf(':'))).toLowerCase();
+    if (scheme.startsWith('socks')) {
+      setGlobalDispatcher(buildSocksDispatcher(proxyUrl));
+      return;
+    }
     setGlobalDispatcher(new ProxyAgent({ uri: proxyUrl }));
     return;
   }
 
+  const lookup = createResilientLookup({ onEvent: opts.onDnsEvent });
   const agent = new Agent({
     connections: 128,
     pipelining: 1,
@@ -73,6 +93,65 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
   setGlobalDispatcher(agent);
 }
 
+/**
+ * Build an undici dispatcher that tunnels every connection through a SOCKS
+ * proxy. undici's own `ProxyAgent` only speaks HTTP CONNECT, so for SOCKS
+ * we supply a custom `connect`: the `socks` client opens the tunnel to the
+ * destination, then for `https:` origins we hand the raw socket to undici's
+ * TLS connector (`httpSocket`) so the certificate handshake still happens
+ * end-to-end. `socks5h` / `socks4a` resolve DNS at the proxy (the default
+ * here, since we pass the destination host name through untouched).
+ */
+function buildSocksDispatcher(proxyUrl: string): Agent {
+  const u = new URL(proxyUrl);
+  const scheme = u.protocol.replace(':', '').toLowerCase();
+  const type: 4 | 5 = scheme === 'socks4' || scheme === 'socks4a' ? 4 : 5;
+  const proxy: SocksProxy = {
+    host: u.hostname,
+    port: Number(u.port) || 1080,
+    type,
+    ...(u.username ? { userId: decodeURIComponent(u.username) } : {}),
+    ...(u.password ? { password: decodeURIComponent(u.password) } : {}),
+  };
+
+  const tlsConnector = buildConnector({});
+  const connect: typeof tlsConnector = (options, callback) => {
+    void (async () => {
+      const host = options.hostname;
+      const port = Number(options.port) || (options.protocol === 'https:' ? 443 : 80);
+      let socket;
+      try {
+        const result = await SocksClient.createConnection({
+          proxy,
+          command: 'connect',
+          destination: { host, port },
+          timeout: 10_000,
+        });
+        socket = result.socket;
+      } catch (err) {
+        callback(err as Error, null);
+        return;
+      }
+      if (options.protocol === 'https:') {
+        tlsConnector({ ...options, httpSocket: socket }, callback);
+        return;
+      }
+      socket.setNoDelay(true);
+      callback(null, socket);
+    })();
+  };
+
+  return new Agent({
+    connections: 128,
+    pipelining: 1,
+    keepAliveTimeout: 60_000,
+    keepAliveMaxTimeout: 120_000,
+    headersTimeout: 10_000,
+    bodyTimeout: 30_000,
+    connect,
+  });
+}
+
 /**
  * Best-effort HTTP protocol detector. We can't ask undici-fetch which
  * ALPN/protocol was actually negotiated for the connection that served
@@ -92,8 +171,13 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
 export function detectHttpProtocol(altSvcHeader: string | null): string | null {
   if (altSvcHeader === null) return 'http/1.1';
   const v = altSvcHeader.toLowerCase();
-  if (v.includes('h3=')) return 'h3';
-  if (v.includes('h2=')) return 'h2';
+  // Alt-Svc is a comma-separated list of advertised protocols, e.g.
+  // `h3=":443"; ma=2592000, h3-29=":443", h2=":443"`. Match the `h3`
+  // token at a boundary INCLUDING draft variants (`h3-29=`, `h3-Q050=`)
+  // — a plain `includes('h3=')` misses draft-only origins. `Alt-Svc:
+  // clear` (origin withdrawing advertisements) matches neither token.
+  if (/(?:^|[\s,])h3(?:-[a-z0-9]+)?=/.test(v)) return 'h3';
+  if (/(?:^|[\s,])h2=/.test(v)) return 'h2';
   return 'http/1.1';
 }
 
@@ -245,7 +329,12 @@ export function defaultRequestHeaders(
   userAgent: string,
   acceptLanguage: string,
   custom: Record<string, string> = {},
-  auth?: { type: 'none' | 'basic' | 'bearer'; username?: string; password?: string; token?: string },
+  auth?: {
+    type: 'none' | 'basic' | 'bearer' | 'digest';
+    username?: string;
+    password?: string;
+    token?: string;
+  },
 ): Record<string, string> {
   const headers: Record<string, string> = {
     'user-agent': userAgent,
@@ -276,3 +365,94 @@ export function defaultRequestHeaders(
   }
   return headers;
 }
+
+/* ------------------------------------------------------------------ */
+/* HTTP Digest authentication (RFC 2617 / RFC 7616)                    */
+/* ------------------------------------------------------------------ */
+
+export interface DigestChallenge {
+  realm: string;
+  nonce: string;
+  /** May be a comma-separated list, e.g. `auth,auth-int`. */
+  qop?: string;
+  opaque?: string;
+  /** `MD5` | `MD5-sess` | `SHA-256` | `SHA-256-sess` (default MD5). */
+  algorithm?: string;
+}
+
+/**
+ * Parse a `WWW-Authenticate` response header and extract the Digest
+ * challenge parameters. Returns `null` when the header isn't a Digest
+ * challenge (e.g. it advertises only Basic), so the caller can skip the
+ * digest round-trip. Tolerates a header that lists multiple schemes by
+ * scanning from the `Digest` token.
+ */
+export function parseDigestChallenge(headerValue: string | null): DigestChallenge | null {
+  if (!headerValue) return null;
+  const idx = headerValue.toLowerCase().indexOf('digest ');
+  if (idx === -1) return null;
+  const params = headerValue.slice(idx + 'digest '.length);
+  const map: Record<string, string> = {};
+  // key=value pairs, value either quoted ("...") or a bare token.
+  const re = /([A-Za-z0-9_-]+)\s*=\s*(?:"((?:[^"\\]|\\.)*)"|([^,\s]+))/g;
+  let m: RegExpExecArray | null;
+  while ((m = re.exec(params)) !== null) {
+    const key = m[1]!.toLowerCase();
+    map[key] = m[2] !== undefined ? m[2].replace(/\\(.)/g, '$1') : (m[3] ?? '');
+  }
+  if (!map['nonce'] || map['realm'] === undefined) return null;
+  return {
+    realm: map['realm'] ?? '',
+    nonce: map['nonce'] ?? '',
+    qop: map['qop'],
+    opaque: map['opaque'],
+    algorithm: map['algorithm'],
+  };
+}
+
+/**
+ * Compute the `Authorization: Digest …` header value for a request, given
+ * a parsed challenge and the request's method + request-target URI.
+ * Supports MD5 / SHA-256, the `-sess` variants, and `qop=auth`. `auth-int`
+ * (which needs a body hash) falls back to the legacy RFC 2069 form.
+ */
+export function buildDigestAuthHeader(
+  challenge: DigestChallenge,
+  req: { method: string; uri: string; username: string; password: string; nc?: string; cnonce?: string },
+): string {
+  const algo = challenge.algorithm ?? 'MD5';
+  const hashName = algo.toUpperCase().startsWith('SHA-256') ? 'sha256' : 'md5';
+  const sess = algo.toLowerCase().endsWith('-sess');
+  const H = (s: string): string => crypto.createHash(hashName).update(s).digest('hex');
+
+  const cnonce = req.cnonce ?? crypto.randomBytes(8).toString('hex');
+  const nc = req.nc ?? '00000001';
+
+  let ha1 = H(`${req.username}:${challenge.realm}:${req.password}`);
+  if (sess) ha1 = H(`${ha1}:${challenge.nonce}:${cnonce}`);
+  const ha2 = H(`${req.method}:${req.uri}`);
+
+  const qops = (challenge.qop ?? '')
+    .split(',')
+    .map((q) => q.trim().toLowerCase())
+    .filter(Boolean);
+  const useQop = qops.includes('auth') ? 'auth' : '';
+
+  const parts = [
+    `username="${req.username}"`,
+    `realm="${challenge.realm}"`,
+    `nonce="${challenge.nonce}"`,
+    `uri="${req.uri}"`,
+  ];
+  let response: string;
+  if (useQop) {
+    response = H(`${ha1}:${challenge.nonce}:${nc}:${cnonce}:${useQop}:${ha2}`);
+    parts.push(`qop=${useQop}`, `nc=${nc}`, `cnonce="${cnonce}"`);
+  } else {
+    response = H(`${ha1}:${challenge.nonce}:${ha2}`);
+  }
+  parts.push(`response="${response}"`);
+  if (challenge.algorithm) parts.push(`algorithm=${challenge.algorithm}`);
+  if (challenge.opaque) parts.push(`opaque="${challenge.opaque}"`);
+  return `Digest ${parts.join(', ')}`;
+}

packages/core/src/jsonpath-lite.ts

@@ -0,0 +1,605 @@
+/**
+ * Minimal, dependency-free JSONPath evaluator.
+ *
+ * Covers the practical subset SEO/scraping users reach for when pulling
+ * values out of JSON API responses — deliberately hand-rolled (no
+ * `jsonpath-plus` dependency) to match the project's zero-heavy-dep
+ * convention and keep the desktop bundle lean.
+ *
+ * Supported syntax
+ * ----------------
+ *   $                         root (optional leading token)
+ *   .name  /  ['name']        child by key
+ *   ..name                    recursive descent then child
+ *   *  /  [*]  /  ..*          wildcard (all array items / object values)
+ *   [0]  [-1]                 array index (negatives count from the end)
+ *   [0,2,4]  ['a','b']        union of indices / names
+ *   [start:end:step]          array slice (Python semantics, all optional)
+ *   [?(<expr>)]               filter — keeps array items / object values
+ *                             matching a boolean expression on `@`
+ *
+ * Filter expression grammar (`@` = the current item):
+ *   @            @.a.b        @['a']['b']         (path on the current item)
+ *   'str'  "str"  12  3.5  true  false  null      (literals)
+ *   == != < <= > >=          (comparison, RHS is a literal or @-path)
+ *   && || !  ( )             (boolean composition)
+ *   a bare @-path            (existence test — true when defined)
+ *
+ * Anything outside this grammar throws — the caller surfaces that as a
+ * per-rule error in the preview and a clean `null` during a crawl, so a
+ * mistyped path never produces silently-wrong data.
+ */
+
+export class JsonPathError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = 'JsonPathError';
+  }
+}
+
+type Selector =
+  | { kind: 'wildcard' }
+  | { kind: 'name'; name: string }
+  | { kind: 'names'; names: string[] }
+  | { kind: 'index'; index: number }
+  | { kind: 'indices'; indices: number[] }
+  | { kind: 'slice'; start: number | null; end: number | null; step: number | null }
+  | { kind: 'filter'; test: (item: unknown) => boolean };
+
+interface Step {
+  /** `..` recursive-descent prefix — apply the selector to this node and
+   *  every descendant rather than only this node. */
+  descendant: boolean;
+  selector: Selector;
+}
+
+/**
+ * Evaluate a JSONPath expression against an already-parsed JSON value.
+ * Returns every matching value in document order. An empty array means
+ * "no match" (not an error).
+ */
+export function evaluateJsonPath(root: unknown, path: string): unknown[] {
+  const steps = parseJsonPath(path);
+  let current: unknown[] = [root];
+  for (const step of steps) {
+    const next: unknown[] = [];
+    for (const node of current) applyStep(node, step, next);
+    current = next;
+  }
+  return current;
+}
+
+function applyStep(node: unknown, step: Step, out: unknown[]): void {
+  const candidates = step.descendant ? descendantsAndSelf(node) : [node];
+  for (const c of candidates) applySelector(c, step.selector, out);
+}
+
+function applySelector(node: unknown, sel: Selector, out: unknown[]): void {
+  switch (sel.kind) {
+    case 'wildcard':
+      if (Array.isArray(node)) out.push(...node);
+      else if (isObject(node)) out.push(...Object.values(node));
+      return;
+    case 'name':
+      if (isObject(node) && Object.prototype.hasOwnProperty.call(node, sel.name)) {
+        out.push((node as Record<string, unknown>)[sel.name]);
+      }
+      return;
+    case 'names':
+      if (isObject(node)) {
+        for (const n of sel.names) {
+          if (Object.prototype.hasOwnProperty.call(node, n)) {
+            out.push((node as Record<string, unknown>)[n]);
+          }
+        }
+      }
+      return;
+    case 'index':
+      if (Array.isArray(node)) {
+        const i = sel.index < 0 ? node.length + sel.index : sel.index;
+        if (i >= 0 && i < node.length) out.push(node[i]);
+      }
+      return;
+    case 'indices':
+      if (Array.isArray(node)) {
+        for (const raw of sel.indices) {
+          const i = raw < 0 ? node.length + raw : raw;
+          if (i >= 0 && i < node.length) out.push(node[i]);
+        }
+      }
+      return;
+    case 'slice':
+      if (Array.isArray(node)) pushSlice(node, sel, out);
+      return;
+    case 'filter':
+      if (Array.isArray(node)) {
+        for (const el of node) if (safeTest(sel.test, el)) out.push(el);
+      } else if (isObject(node)) {
+        for (const v of Object.values(node)) if (safeTest(sel.test, v)) out.push(v);
+      }
+      return;
+  }
+}
+
+function pushSlice(
+  arr: unknown[],
+  sel: { start: number | null; end: number | null; step: number | null },
+  out: unknown[],
+): void {
+  const len = arr.length;
+  const step = sel.step ?? 1;
+  if (step === 0) return;
+  const norm = (v: number | null, dflt: number): number => {
+    if (v === null) return dflt;
+    return v < 0 ? Math.max(len + v, step > 0 ? 0 : -1) : Math.min(v, step > 0 ? len : len - 1);
+  };
+  if (step > 0) {
+    const start = norm(sel.start, 0);
+    const end = norm(sel.end, len);
+    for (let i = start; i < end; i += step) out.push(arr[i]);
+  } else {
+    const start = norm(sel.start, len - 1);
+    const end = norm(sel.end, -1);
+    for (let i = start; i > end; i += step) out.push(arr[i]);
+  }
+}
+
+function safeTest(test: (item: unknown) => boolean, item: unknown): boolean {
+  try {
+    return test(item);
+  } catch {
+    return false;
+  }
+}
+
+function descendantsAndSelf(node: unknown): unknown[] {
+  const out: unknown[] = [];
+  const walk = (n: unknown): void => {
+    out.push(n);
+    if (Array.isArray(n)) {
+      for (const el of n) walk(el);
+    } else if (isObject(n)) {
+      for (const v of Object.values(n)) walk(v);
+    }
+  };
+  walk(node);
+  return out;
+}
+
+function isObject(v: unknown): v is Record<string, unknown> {
+  return typeof v === 'object' && v !== null && !Array.isArray(v);
+}
+
+/* ------------------------------------------------------------------ */
+/* Path parsing                                                        */
+/* ------------------------------------------------------------------ */
+
+function parseJsonPath(path: string): Step[] {
+  const s = path.trim();
+  if (!s) throw new JsonPathError('empty JSONPath expression');
+  let i = s[0] === '$' ? 1 : 0;
+  const steps: Step[] = [];
+  let descendant = false;
+
+  while (i < s.length) {
+    const ch = s[i];
+    if (ch === ' ') {
+      i++;
+      continue;
+    }
+    if (ch === '.') {
+      if (s[i + 1] === '.') {
+        descendant = true;
+        i += 2;
+        // `..` may be followed by `[` (handled below), `*`, or a name.
+        if (i < s.length && s[i] !== '[') {
+          if (s[i] === '*') {
+            steps.push({ descendant: true, selector: { kind: 'wildcard' } });
+            descendant = false;
+            i++;
+          } else {
+            const { name, next } = readName(s, i);
+            steps.push({ descendant: true, selector: { kind: 'name', name } });
+            descendant = false;
+            i = next;
+          }
+        }
+        continue;
+      }
+      i++; // single dot
+      if (s[i] === '*') {
+        steps.push({ descendant, selector: { kind: 'wildcard' } });
+        descendant = false;
+        i++;
+        continue;
+      }
+      const { name, next } = readName(s, i);
+      steps.push({ descendant, selector: { kind: 'name', name } });
+      descendant = false;
+      i = next;
+      continue;
+    }
+    if (ch === '[') {
+      const close = findMatchingBracket(s, i);
+      const inner = s.slice(i + 1, close).trim();
+      steps.push({ descendant, selector: parseBracket(inner) });
+      descendant = false;
+      i = close + 1;
+      continue;
+    }
+    if (ch === '*') {
+      steps.push({ descendant, selector: { kind: 'wildcard' } });
+      descendant = false;
+      i++;
+      continue;
+    }
+    // Bare leading name, e.g. `store.book` without the `$.` prefix.
+    const { name, next } = readName(s, i);
+    steps.push({ descendant, selector: { kind: 'name', name } });
+    descendant = false;
+    i = next;
+  }
+  return steps;
+}
+
+function readName(s: string, start: number): { name: string; next: number } {
+  let i = start;
+  while (i < s.length && s[i] !== '.' && s[i] !== '[' && s[i] !== ' ') i++;
+  const name = s.slice(start, i);
+  if (!name) throw new JsonPathError(`expected a property name at position ${start}`);
+  return { name, next: i };
+}
+
+function findMatchingBracket(s: string, open: number): number {
+  let depth = 0;
+  let quote = '';
+  for (let i = open; i < s.length; i++) {
+    const ch = s[i];
+    if (quote) {
+      if (ch === '\\') i++;
+      else if (ch === quote) quote = '';
+      continue;
+    }
+    if (ch === "'" || ch === '"') quote = ch;
+    else if (ch === '[') depth++;
+    else if (ch === ']') {
+      depth--;
+      if (depth === 0) return i;
+    }
+  }
+  throw new JsonPathError('unbalanced `[` in JSONPath expression');
+}
+
+function parseBracket(inner: string): Selector {
+  if (inner === '*') return { kind: 'wildcard' };
+  if (inner.startsWith('?')) {
+    // `?(<expr>)` filter.
+    const exprStart = inner.indexOf('(');
+    if (exprStart === -1 || !inner.endsWith(')')) {
+      throw new JsonPathError(`malformed filter: [${inner}]`);
+    }
+    const expr = inner.slice(exprStart + 1, -1);
+    return { kind: 'filter', test: buildFilter(expr) };
+  }
+  // Quoted name(s): 'a' or 'a','b'
+  if (inner.startsWith("'") || inner.startsWith('"')) {
+    const names = splitTopLevel(inner, ',').map((part) => unquote(part.trim()));
+    return names.length === 1 ? { kind: 'name', name: names[0]! } : { kind: 'names', names };
+  }
+  // Slice `start:end:step` (only when a top-level `:` is present and it's
+  // not a union of names).
+  if (inner.includes(':')) {
+    const parts = inner.split(':');
+    if (parts.length <= 3) {
+      const toNum = (p: string): number | null => {
+        const t = p.trim();
+        if (t === '') return null;
+        const n = Number(t);
+        if (!Number.isInteger(n)) throw new JsonPathError(`invalid slice bound: ${t}`);
+        return n;
+      };
+      return {
+        kind: 'slice',
+        start: toNum(parts[0] ?? ''),
+        end: toNum(parts[1] ?? ''),
+        step: parts.length === 3 ? toNum(parts[2] ?? '') : null,
+      };
+    }
+  }
+  // Index or union of indices.
+  const idxParts = splitTopLevel(inner, ',').map((p) => p.trim());
+  const indices = idxParts.map((p) => {
+    const n = Number(p);
+    if (!Number.isInteger(n)) throw new JsonPathError(`invalid array index: ${p}`);
+    return n;
+  });
+  return indices.length === 1 ? { kind: 'index', index: indices[0]! } : { kind: 'indices', indices };
+}
+
+function unquote(s: string): string {
+  if ((s.startsWith("'") && s.endsWith("'")) || (s.startsWith('"') && s.endsWith('"'))) {
+    return s.slice(1, -1).replace(/\\(['"\\])/g, '$1');
+  }
+  return s;
+}
+
+function splitTopLevel(s: string, sep: string): string[] {
+  const out: string[] = [];
+  let buf = '';
+  let quote = '';
+  for (let i = 0; i < s.length; i++) {
+    const ch = s[i]!;
+    if (quote) {
+      if (ch === '\\') {
+        buf += ch + (s[i + 1] ?? '');
+        i++;
+        continue;
+      }
+      if (ch === quote) quote = '';
+      buf += ch;
+      continue;
+    }
+    if (ch === "'" || ch === '"') {
+      quote = ch;
+      buf += ch;
+      continue;
+    }
+    if (ch === sep) {
+      out.push(buf);
+      buf = '';
+      continue;
+    }
+    buf += ch;
+  }
+  out.push(buf);
+  return out;
+}
+
+/* ------------------------------------------------------------------ */
+/* Filter expression parser  (`@.a > 3 && @.b == 'x'`)                 */
+/* ------------------------------------------------------------------ */
+
+type FilterToken =
+  | { t: 'path'; parts: string[] }
+  | { t: 'num'; v: number }
+  | { t: 'str'; v: string }
+  | { t: 'bool'; v: boolean }
+  | { t: 'null' }
+  | { t: 'op'; v: string }
+  | { t: 'lparen' }
+  | { t: 'rparen' };
+
+function buildFilter(expr: string): (item: unknown) => boolean {
+  const tokens = tokenizeFilter(expr);
+  const parser = new FilterParser(tokens);
+  const node = parser.parseOr();
+  parser.expectEnd();
+  return (item: unknown) => Boolean(node(item));
+}
+
+function tokenizeFilter(expr: string): FilterToken[] {
+  const tokens: FilterToken[] = [];
+  let i = 0;
+  const isNameChar = (c: string): boolean => /[A-Za-z0-9_$-]/.test(c);
+  while (i < expr.length) {
+    const ch = expr[i]!;
+    if (ch === ' ' || ch === '\t' || ch === '\n') {
+      i++;
+      continue;
+    }
+    if (ch === '@') {
+      i++;
+      const parts: string[] = [];
+      // `@.a.b`, `@['a']['b']`, or bare `@`.
+      while (i < expr.length) {
+        if (expr[i] === '.') {
+          i++;
+          let name = '';
+          while (i < expr.length && isNameChar(expr[i]!)) name += expr[i++];
+          if (name) parts.push(name);
+          else break;
+        } else if (expr[i] === '[') {
+          const close = expr.indexOf(']', i);
+          if (close === -1) throw new JsonPathError('unterminated `[` in filter');
+          parts.push(unquote(expr.slice(i + 1, close).trim()));
+          i = close + 1;
+        } else break;
+      }
+      tokens.push({ t: 'path', parts });
+      continue;
+    }
+    if (ch === "'" || ch === '"') {
+      let j = i + 1;
+      let str = '';
+      while (j < expr.length && expr[j] !== ch) {
+        if (expr[j] === '\\') {
+          str += expr[j + 1] ?? '';
+          j += 2;
+        } else str += expr[j++];
+      }
+      if (j >= expr.length) throw new JsonPathError('unterminated string literal in filter');
+      tokens.push({ t: 'str', v: str });
+      i = j + 1;
+      continue;
+    }
+    if (/[0-9]/.test(ch) || (ch === '-' && /[0-9.]/.test(expr[i + 1] ?? ''))) {
+      let num = ch;
+      i++;
+      while (i < expr.length && /[0-9.eE+-]/.test(expr[i]!)) num += expr[i++];
+      const v = Number(num);
+      if (Number.isNaN(v)) throw new JsonPathError(`invalid number in filter: ${num}`);
+      tokens.push({ t: 'num', v });
+      continue;
+    }
+    if (ch === '(') {
+      tokens.push({ t: 'lparen' });
+      i++;
+      continue;
+    }
+    if (ch === ')') {
+      tokens.push({ t: 'rparen' });
+      i++;
+      continue;
+    }
+    const two = expr.slice(i, i + 2);
+    if (two === '==' || two === '!=' || two === '<=' || two === '>=' || two === '&&' || two === '||') {
+      tokens.push({ t: 'op', v: two });
+      i += 2;
+      continue;
+    }
+    if (ch === '<' || ch === '>' || ch === '!') {
+      tokens.push({ t: 'op', v: ch });
+      i++;
+      continue;
+    }
+    // Bare identifier: true / false / null.
+    if (isNameChar(ch)) {
+      let word = '';
+      while (i < expr.length && isNameChar(expr[i]!)) word += expr[i++];
+      if (word === 'true') tokens.push({ t: 'bool', v: true });
+      else if (word === 'false') tokens.push({ t: 'bool', v: false });
+      else if (word === 'null') tokens.push({ t: 'null' });
+      else throw new JsonPathError(`unexpected token in filter: ${word}`);
+      continue;
+    }
+    throw new JsonPathError(`unexpected character in filter: ${ch}`);
+  }
+  return tokens;
+}
+
+type FilterFn = (item: unknown) => unknown;
+
+class FilterParser {
+  private pos = 0;
+  constructor(private readonly tokens: FilterToken[]) {}
+
+  private peek(): FilterToken | undefined {
+    return this.tokens[this.pos];
+  }
+  private next(): FilterToken | undefined {
+    return this.tokens[this.pos++];
+  }
+  expectEnd(): void {
+    if (this.pos !== this.tokens.length) {
+      throw new JsonPathError('trailing tokens in filter expression');
+    }
+  }
+
+  parseOr(): FilterFn {
+    let left = this.parseAnd();
+    while (this.isOp('||')) {
+      this.next();
+      const right = this.parseAnd();
+      const l = left;
+      left = (item) => Boolean(l(item)) || Boolean(right(item));
+    }
+    return left;
+  }
+
+  private parseAnd(): FilterFn {
+    let left = this.parseUnary();
+    while (this.isOp('&&')) {
+      this.next();
+      const right = this.parseUnary();
+      const l = left;
+      left = (item) => Boolean(l(item)) && Boolean(right(item));
+    }
+    return left;
+  }
+
+  private parseUnary(): FilterFn {
+    if (this.isOp('!')) {
+      this.next();
+      const operand = this.parseUnary();
+      return (item) => !operand(item);
+    }
+    return this.parseComparison();
+  }
+
+  private parseComparison(): FilterFn {
+    if (this.peek()?.t === 'lparen') {
+      this.next();
+      const inner = this.parseOr();
+      if (this.peek()?.t !== 'rparen') throw new JsonPathError('missing `)` in filter');
+      this.next();
+      return inner;
+    }
+    const left = this.parseValue();
+    const op = this.peek();
+    if (op && op.t === 'op' && ['==', '!=', '<', '<=', '>', '>='].includes(op.v)) {
+      this.next();
+      const right = this.parseValue();
+      return (item) => compare(left(item), op.v, right(item));
+    }
+    // No operator → existence/truthiness test.
+    return (item) => left(item) !== undefined;
+  }
+
+  private parseValue(): FilterFn {
+    const tok = this.next();
+    if (!tok) throw new JsonPathError('unexpected end of filter expression');
+    switch (tok.t) {
+      case 'path':
+        return (item) => resolvePath(item, tok.parts);
+      case 'num':
+        return () => tok.v;
+      case 'str':
+        return () => tok.v;
+      case 'bool':
+        return () => tok.v;
+      case 'null':
+        return () => null;
+      default:
+        throw new JsonPathError('expected a value in filter expression');
+    }
+  }
+
+  private isOp(v: string): boolean {
+    const tok = this.peek();
+    return !!tok && tok.t === 'op' && tok.v === v;
+  }
+}
+
+function resolvePath(item: unknown, parts: string[]): unknown {
+  let cur: unknown = item;
+  for (const p of parts) {
+    if (isObject(cur)) cur = cur[p];
+    else if (Array.isArray(cur)) {
+      const n = Number(p);
+      cur = Number.isInteger(n) ? cur[n < 0 ? cur.length + n : n] : undefined;
+    } else return undefined;
+  }
+  return cur;
+}
+
+function compare(a: unknown, op: string, b: unknown): boolean {
+  switch (op) {
+    case '==':
+      return looseEq(a, b);
+    case '!=':
+      return !looseEq(a, b);
+    case '<':
+    case '<=':
+    case '>':
+    case '>=': {
+      if (typeof a === 'number' && typeof b === 'number') {
+        return op === '<' ? a < b : op === '<=' ? a <= b : op === '>' ? a > b : a >= b;
+      }
+      if (typeof a === 'string' && typeof b === 'string') {
+        const c = a < b ? -1 : a > b ? 1 : 0;
+        return op === '<' ? c < 0 : op === '<=' ? c <= 0 : op === '>' ? c > 0 : c >= 0;
+      }
+      return false;
+    }
+    default:
+      return false;
+  }
+}
+
+function looseEq(a: unknown, b: unknown): boolean {
+  if (a === b) return true;
+  // Number/string cross-compare so `@.id == '5'` works against numeric JSON.
+  if (typeof a === 'number' && typeof b === 'string') return a === Number(b);
+  if (typeof a === 'string' && typeof b === 'number') return Number(a) === b;
+  return false;
+}

packages/core/src/pdf-probe.ts

@@ -0,0 +1,398 @@
+import { fetch as undiciFetch } from 'undici';
+import { defaultRequestHeaders } from './http-client.js';
+import type { CrawlConfig } from '@freecrawl/shared-types';
+
+/**
+ * V2 Faz 16 — lightweight PDF metadata extractor.
+ *
+ * Fetches a PDF (ranged GET, first few MB) and pulls document metadata
+ * without a full PDF-parsing library. Two sources, in priority order:
+ *
+ *   1. XMP packet — `<x:xmpmeta>…</x:xmpmeta>`. The XMP spec mandates the
+ *      packet be an *uncompressed* UTF-8 stream so non-PDF tools can read
+ *      it, which makes it reliably parseable with plain string scanning.
+ *      We read `dc:title`, `dc:creator`, `xmp:CreateDate`, `pdf:Producer`.
+ *   2. Info dictionary — `/Title (...)`, `/Author (...)`,
+ *      `/CreationDate (D:...)`, `/Producer (...)`. Works when the Info dict
+ *      is not inside a compressed object stream (common in simpler/older
+ *      PDFs). Decodes PDF literal strings (UTF-16BE BOM, PDFDocEncoding ≈
+ *      Latin-1, escape sequences) and hex `<...>` strings.
+ *
+ * Page count is best-effort: the `/Type /Pages … /Count N` root, else a
+ * count of `/Type /Page` leaf markers. Null when neither is visible.
+ *
+ * This is deliberately a "smoke" extractor — in the same spirit as
+ * FreeCrawl's hand-rolled AMP / structured-data validators — not a
+ * spec-complete PDF parser. Metadata that lives only inside a compressed
+ * stream of a very large PDF may be missed; that is the accepted coverage
+ * trade-off for adding zero new dependencies.
+ */
+
+/** Bytes read per PDF — bounds memory + time. Metadata past this slice in
+ *  a very large PDF is an accepted miss. */
+const MAX_BYTES = 4 * 1024 * 1024;
+const PROBE_TIMEOUT_MS = 12_000;
+
+export interface PdfProbeResult {
+  title: string | null;
+  author: string | null;
+  pageCount: number | null;
+  creationDate: string | null;
+  producer: string | null;
+  /** Free-form error string when the fetch / read fails. Null otherwise. */
+  error: string | null;
+}
+
+type PdfMetadata = Omit<PdfProbeResult, 'error'>;
+
+const EMPTY: PdfMetadata = {
+  title: null,
+  author: null,
+  pageCount: null,
+  creationDate: null,
+  producer: null,
+};
+
+export async function probePdfMetadata(
+  url: string,
+  config: Pick<CrawlConfig, 'userAgent' | 'acceptLanguage' | 'customHeaders' | 'auth'>,
+): Promise<PdfProbeResult> {
+  const controller = new AbortController();
+  const timer = setTimeout(() => controller.abort(), PROBE_TIMEOUT_MS);
+  try {
+    const headers = defaultRequestHeaders(
+      config.userAgent,
+      config.acceptLanguage,
+      config.customHeaders,
+      config.auth,
+    );
+    headers['range'] = `bytes=0-${MAX_BYTES - 1}`;
+    headers['accept-encoding'] = 'identity';
+    headers['accept'] = 'application/pdf,*/*;q=0.8';
+
+    const res = await undiciFetch(url, {
+      method: 'GET',
+      headers,
+      redirect: 'follow',
+      signal: controller.signal,
+    });
+    // 200 (Range ignored) and 206 (Partial Content) are both fine.
+    if (res.status < 200 || res.status >= 300 || !res.body) {
+      try {
+        await res.body?.cancel();
+      } catch {
+        /* ignore */
+      }
+      return { ...EMPTY, error: `HTTP ${res.status}` };
+    }
+
+    const reader = res.body.getReader();
+    const chunks: Uint8Array[] = [];
+    let total = 0;
+    try {
+      while (total < MAX_BYTES) {
+        const { done, value } = await reader.read();
+        if (done) break;
+        if (value && value.length > 0) {
+          chunks.push(value);
+          total += value.length;
+        }
+      }
+    } finally {
+      try {
+        await reader.cancel();
+      } catch {
+        /* ignore */
+      }
+    }
+    if (total === 0) return { ...EMPTY, error: 'empty body' };
+
+    const cap = Math.min(total, MAX_BYTES);
+    const buf = Buffer.concat(chunks).subarray(0, cap);
+    return { ...parsePdfMetadata(buf), error: null };
+  } catch (err) {
+    const name = (err as { name?: string } | null)?.name;
+    const message =
+      name === 'AbortError' || name === 'TimeoutError'
+        ? `timed out after ${PROBE_TIMEOUT_MS / 1000}s`
+        : err instanceof Error
+          ? err.message
+          : String(err);
+    return { ...EMPTY, error: message };
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+// ── Parsing ─────────────────────────────────────────────────────────────
+
+function parsePdfMetadata(buf: Buffer): PdfMetadata {
+  const out: PdfMetadata = { ...EMPTY };
+
+  // 1. XMP packet — the reliable, uncompressed source.
+  const xmp = extractXmp(buf);
+  if (xmp) {
+    out.title = xmpRdfLi(xmp, 'dc:title') ?? xmpSimple(xmp, 'dc:title');
+    out.author = xmpRdfLi(xmp, 'dc:creator') ?? xmpSimple(xmp, 'dc:creator');
+    out.producer = xmpSimple(xmp, 'pdf:Producer');
+    const created = xmpSimple(xmp, 'xmp:CreateDate');
+    if (created) out.creationDate = isoFromXmpDate(created);
+  }
+
+  // 2. Info-dictionary fallback for anything XMP didn't supply.
+  const latin = buf.toString('latin1');
+  if (!out.title) out.title = infoDictString(latin, 'Title');
+  if (!out.author) out.author = infoDictString(latin, 'Author');
+  if (!out.producer) out.producer = infoDictString(latin, 'Producer');
+  if (!out.creationDate) {
+    const cd = infoDictString(latin, 'CreationDate');
+    if (cd) out.creationDate = isoFromPdfDate(cd);
+  }
+
+  // 3. Page count — best-effort.
+  out.pageCount = extractPageCount(latin);
+
+  return out;
+}
+
+/** Carve out the `<x:xmpmeta>…</x:xmpmeta>` packet and re-decode it as UTF-8. */
+function extractXmp(buf: Buffer): string | null {
+  const hay = buf.toString('latin1');
+  const start = hay.indexOf('<x:xmpmeta');
+  if (start < 0) return null;
+  const endTag = '</x:xmpmeta>';
+  const end = hay.indexOf(endTag, start);
+  if (end < 0) return null;
+  return buf.toString('utf8', start, end + endTag.length);
+}
+
+/** First `<rdf:li>` value inside an `<rdf:Alt>` / `<rdf:Seq>` property
+ *  (dc:title, dc:creator). */
+function xmpRdfLi(xmp: string, tag: string): string | null {
+  const re = new RegExp(
+    `<${escapeRe(tag)}\\b[\\s\\S]*?<rdf:li[^>]*>([\\s\\S]*?)</rdf:li>`,
+    'i',
+  );
+  const m = re.exec(xmp);
+  return m && m[1] !== undefined ? cleanXmp(m[1]) : null;
+}
+
+/** Simple element `<q>VALUE</q>` or compact attribute `q="VALUE"` form. */
+function xmpSimple(xmp: string, qname: string): string | null {
+  const el = new RegExp(
+    `<${escapeRe(qname)}\\b[^>]*>([\\s\\S]*?)</${escapeRe(qname)}>`,
+    'i',
+  ).exec(xmp);
+  if (el && el[1] !== undefined && el[1].trim() && !el[1].includes('<')) {
+    return cleanXmp(el[1]);
+  }
+  const attr = new RegExp(`\\b${escapeRe(qname)}="([^"]*)"`, 'i').exec(xmp);
+  if (attr && attr[1] !== undefined && attr[1].trim()) return cleanXmp(attr[1]);
+  return null;
+}
+
+function cleanXmp(s: string): string | null {
+  return tidy(decodeXmlEntities(s));
+}
+
+function decodeXmlEntities(s: string): string {
+  return s
+    .replace(/&lt;/g, '<')
+    .replace(/&gt;/g, '>')
+    .replace(/&quot;/g, '"')
+    .replace(/&apos;/g, "'")
+    .replace(/&#x([0-9a-fA-F]+);/g, (_m, h: string) => fromCp(parseInt(h, 16)))
+    .replace(/&#(\d+);/g, (_m, d: string) => fromCp(parseInt(d, 10)))
+    .replace(/&amp;/g, '&');
+}
+
+/**
+ * Read a PDF string value for `/<key>` from the (latin1-decoded) bytes.
+ * Handles literal `(...)` strings (balanced parens + escapes) and hex
+ * `<...>` strings. Takes the LAST boundary-matched occurrence so an
+ * incrementally-updated PDF's newest Info dict wins.
+ */
+function infoDictString(latin: string, key: string): string | null {
+  const marker = `/${key}`;
+  let idx = -1;
+  let from = 0;
+  for (;;) {
+    const i = latin.indexOf(marker, from);
+    if (i < 0) break;
+    const after = latin[i + marker.length];
+    if (after === undefined || !/[A-Za-z0-9]/.test(after)) idx = i;
+    from = i + marker.length;
+  }
+  if (idx < 0) return null;
+
+  let p = idx + marker.length;
+  while (p < latin.length) {
+    const w = latin[p];
+    if (w === undefined || !/\s/.test(w)) break;
+    p++;
+  }
+  const ch = latin[p];
+  if (ch === '(') {
+    p++;
+    let depth = 1;
+    let raw = '';
+    while (p < latin.length && depth > 0) {
+      const c = latin[p]!;
+      if (c === '\\') {
+        raw += c + (latin[p + 1] ?? '');
+        p += 2;
+        continue;
+      }
+      if (c === '(') {
+        depth++;
+        raw += c;
+      } else if (c === ')') {
+        depth--;
+        if (depth === 0) {
+          p++;
+          break;
+        }
+        raw += c;
+      } else {
+        raw += c;
+      }
+      p++;
+    }
+    return decodePdfLiteral(raw);
+  }
+  if (ch === '<') {
+    p++;
+    let hex = '';
+    while (p < latin.length && latin[p] !== '>') {
+      hex += latin[p];
+      p++;
+    }
+    return decodePdfHex(hex);
+  }
+  return null;
+}
+
+/** Unescape a PDF literal-string body, then decode its text bytes. */
+function decodePdfLiteral(raw: string): string | null {
+  let bytes = '';
+  for (let i = 0; i < raw.length; i++) {
+    const c = raw[i]!;
+    if (c !== '\\') {
+      bytes += c;
+      continue;
+    }
+    const n = raw[i + 1];
+    if (n === undefined) break;
+    if (n === 'n') bytes += '\n';
+    else if (n === 'r') bytes += '\r';
+    else if (n === 't') bytes += '\t';
+    else if (n === 'b') bytes += '\b';
+    else if (n === 'f') bytes += '\f';
+    else if (n === '(' || n === ')' || n === '\\') bytes += n;
+    else if (n >= '0' && n <= '7') {
+      let oct = n;
+      i++;
+      for (let k = 0; k < 2; k++) {
+        const d = raw[i + 1];
+        if (d !== undefined && d >= '0' && d <= '7') {
+          oct += d;
+          i++;
+        } else break;
+      }
+      bytes += String.fromCharCode(parseInt(oct, 8) & 0xff);
+      continue;
+    } else if (n === '\n' || n === '\r') {
+      // line continuation — drop the backslash + newline
+    } else {
+      bytes += n;
+    }
+    i++;
+  }
+  return decodePdfTextBytes(bytes);
+}
+
+function decodePdfHex(hex: string): string | null {
+  const clean = hex.replace(/[^0-9A-Fa-f]/g, '');
+  const padded = clean.length % 2 ? `${clean}0` : clean;
+  let bytes = '';
+  for (let i = 0; i < padded.length; i += 2) {
+    bytes += String.fromCharCode(parseInt(padded.slice(i, i + 2), 16));
+  }
+  return decodePdfTextBytes(bytes);
+}
+
+/** PDF text strings are either UTF-16BE (FE FF BOM) or PDFDocEncoding (≈ Latin-1). */
+function decodePdfTextBytes(bytes: string): string | null {
+  if (bytes.length === 0) return null;
+  if (bytes.charCodeAt(0) === 0xfe && bytes.charCodeAt(1) === 0xff) {
+    let out = '';
+    for (let i = 2; i + 1 < bytes.length; i += 2) {
+      out += String.fromCharCode((bytes.charCodeAt(i) << 8) | bytes.charCodeAt(i + 1));
+    }
+    return tidy(out);
+  }
+  if (
+    bytes.charCodeAt(0) === 0xef &&
+    bytes.charCodeAt(1) === 0xbb &&
+    bytes.charCodeAt(2) === 0xbf
+  ) {
+    return tidy(Buffer.from(bytes.slice(3), 'latin1').toString('utf8'));
+  }
+  return tidy(bytes);
+}
+
+/** Page count from the page-tree root `/Count`, else a `/Type /Page` tally. */
+function extractPageCount(latin: string): number | null {
+  const reA = /\/Type\s*\/Pages\b[\s\S]{0,300}?\/Count\s+(\d+)/;
+  const reB = /\/Count\s+(\d+)[\s\S]{0,300}?\/Type\s*\/Pages\b/;
+  const m = reA.exec(latin) ?? reB.exec(latin);
+  if (m && m[1] !== undefined) {
+    const n = parseInt(m[1], 10);
+    if (n > 0 && n < 1_000_000) return n;
+  }
+  const pageRe = /\/Type\s*\/Page(?![sA-Za-z])/g;
+  let count = 0;
+  while (pageRe.exec(latin) !== null) {
+    count++;
+    if (count > 100_000) break;
+  }
+  return count > 0 ? count : null;
+}
+
+function isoFromXmpDate(s: string): string | null {
+  const t = s.trim();
+  return t.length > 0 ? t.slice(0, 40) : null;
+}
+
+function isoFromPdfDate(s: string): string | null {
+  const m = /(\d{4})(\d{2})(\d{2})(\d{2})?(\d{2})?(\d{2})?/.exec(s.replace(/^D:/, ''));
+  if (!m || m[1] === undefined || m[2] === undefined || m[3] === undefined) {
+    const t = s.trim();
+    return t.length > 0 ? t.slice(0, 40) : null;
+  }
+  return `${m[1]}-${m[2]}-${m[3]}T${m[4] ?? '00'}:${m[5] ?? '00'}:${m[6] ?? '00'}`;
+}
+
+/** Trim, collapse whitespace, strip control chars, cap length — without any
+ *  control-char regex literal (keeps the source bytes clean). */
+function tidy(s: string): string | null {
+  let cleaned = '';
+  for (let i = 0; i < s.length; i++) {
+    const code = s.charCodeAt(i);
+    cleaned += code < 0x20 || code === 0x7f ? ' ' : (s[i] ?? '');
+  }
+  cleaned = cleaned.replace(/\s+/g, ' ').trim();
+  return cleaned.length > 0 ? cleaned.slice(0, 500) : null;
+}
+
+function fromCp(cp: number): string {
+  try {
+    return cp >= 0 && cp <= 0x10ffff ? String.fromCodePoint(cp) : '';
+  } catch {
+    return '';
+  }
+}
+
+function escapeRe(s: string): string {
+  return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}

packages/core/src/xpath-lite.ts

@@ -0,0 +1,717 @@
+/**
+ * Minimal, dependency-free XPath 1.0 subset evaluated directly over the
+ * cheerio / domhandler tree that the crawler already builds.
+ *
+ * Hand-rolled rather than pulling in `xpath` + `@xmldom/xmldom` (strict
+ * XML, chokes on real-world HTML) or `fontoxpath` (heavy) — cheerio gives
+ * us lenient HTML parsing for free, so we only implement the location-path
+ * navigation and predicate semantics on top of it. Matches the project's
+ * "hand-rolled, no heavy dep" convention.
+ *
+ * Supported syntax
+ * ----------------
+ *   /step  //step                absolute child / descendant axis
+ *   //div   //*                  element name test / wildcard
+ *   a/b//c                       chained child + descendant steps
+ *   .../@attr   .../@*           attribute terminal (reads attribute values)
+ *   .../text()                   direct text-node terminal
+ *   [n]  [last()]  [last()-1]    positional predicates (per-parent, like XPath)
+ *   [@attr]                      attribute existence
+ *   [@attr='v']  [@attr!='v']    attribute comparison ( = != < <= > >= )
+ *   [contains(@class,'x')]       substring test
+ *   [starts-with(@href,'/a')]    prefix test
+ *   [text()='x']  [.='x']        text comparison
+ *   [normalize-space()='x']      whitespace-collapsed text comparison
+ *   [child]  [.//desc]  [a/@x]   relative-path existence / value
+ *   [not(...)]  [a and b]  [a or b]   boolean composition
+ *
+ * Anything outside this subset throws an {@link XPathError}; callers turn
+ * that into a clean per-rule error (preview) or `null` (crawl) so a
+ * mistyped expression never yields silently-wrong data.
+ */
+
+import { isTag, isText, type AnyNode, type Element } from 'domhandler';
+import type { CheerioAPI } from 'cheerio';
+
+export class XPathError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = 'XPathError';
+  }
+}
+
+export type XPathResult =
+  | { kind: 'nodes'; nodes: Element[] }
+  | { kind: 'values'; values: string[] };
+
+type NodeTest =
+  | { kind: 'name'; name: string }
+  | { kind: 'wildcard' }
+  | { kind: 'node' }
+  | { kind: 'text' }
+  | { kind: 'attr'; name: string };
+
+// A node-set is `Element[]`; an attribute / text() value-set is `string[]`.
+// They're discriminated at runtime by element typeof in the coercion helpers.
+type XVal = string | number | boolean | Element[] | string[] | undefined;
+
+interface PredEnv {
+  $: CheerioAPI;
+  node: Element;
+  position: number;
+  last: number;
+}
+
+type PredFn = (env: PredEnv) => XVal;
+
+interface PathStep {
+  axis: 'child' | 'descendant';
+  nodeTest: NodeTest;
+  predicates: PredFn[];
+}
+
+/**
+ * Evaluate an XPath location path against a loaded cheerio document.
+ */
+export function evaluateXPath($: CheerioAPI, expr: string): XPathResult {
+  const trimmed = expr.trim();
+  if (!trimmed) throw new XPathError('empty XPath expression');
+  const { steps } = parsePath(trimmed);
+  return runPath($, [rootNode($)], steps);
+}
+
+function runPath($: CheerioAPI, ctx: AnyNode[], steps: PathStep[]): XPathResult {
+  let current: AnyNode[] = ctx;
+  for (let s = 0; s < steps.length; s++) {
+    const step = steps[s]!;
+    if (step.nodeTest.kind === 'attr' || step.nodeTest.kind === 'text') {
+      if (s !== steps.length - 1) {
+        throw new XPathError('@attribute / text() node test must be the last step');
+      }
+      return { kind: 'values', values: extractTerminal(current, step.nodeTest) };
+    }
+    current = evalElementStep($, current, step);
+  }
+  return { kind: 'nodes', nodes: current.filter(isTag) };
+}
+
+function rootNode($: CheerioAPI): AnyNode {
+  const r = $.root()[0];
+  if (!r) throw new XPathError('document has no root node');
+  return r as AnyNode;
+}
+
+/* ------------------------------------------------------------------ */
+/* Step evaluation                                                     */
+/* ------------------------------------------------------------------ */
+
+function evalElementStep($: CheerioAPI, ctxNodes: AnyNode[], step: PathStep): Element[] {
+  // Gather candidates in document order: the descendant axis walks the
+  // whole subtree, the child axis just the immediate children. Predicates
+  // are applied afterwards with per-parent positional semantics.
+  const seen = new Set<AnyNode>();
+  const cand: Element[] = [];
+  for (const ctx of ctxNodes) {
+    const pool = step.axis === 'descendant' ? descendantElements(ctx) : elementChildren(ctx);
+    for (const el of pool) {
+      if (matchNodeTest(el, step.nodeTest) && !seen.has(el)) {
+        seen.add(el);
+        cand.push(el);
+      }
+    }
+  }
+  return applyPredicates($, cand, step.predicates);
+}
+
+function applyPredicates($: CheerioAPI, cand: Element[], predicates: PredFn[]): Element[] {
+  let list = cand;
+  for (const pred of predicates) {
+    // XPath position() is relative to the context node — i.e. per parent —
+    // and recomputed after each predicate. Group the current (document-
+    // ordered) list by parent, evaluate within each group, then filter the
+    // flat list so document order is preserved.
+    const groups = new Map<AnyNode | null, Element[]>();
+    for (const node of list) {
+      const parent = (node.parent as AnyNode | null) ?? null;
+      const g = groups.get(parent);
+      if (g) g.push(node);
+      else groups.set(parent, [node]);
+    }
+    const survivors = new Set<Element>();
+    for (const group of groups.values()) {
+      const last = group.length;
+      group.forEach((node, idx) => {
+        const v = pred({ $, node, position: idx + 1, last });
+        const keep = typeof v === 'number' ? idx + 1 === v : toBoolean(v);
+        if (keep) survivors.add(node);
+      });
+    }
+    list = list.filter((n) => survivors.has(n));
+  }
+  return list;
+}
+
+function matchNodeTest(node: Element, test: NodeTest): boolean {
+  switch (test.kind) {
+    case 'wildcard':
+    case 'node':
+      return true;
+    case 'name':
+      return (node.name ?? '').toLowerCase() === test.name.toLowerCase();
+    default:
+      return false;
+  }
+}
+
+function extractTerminal(ctxNodes: AnyNode[], test: NodeTest): string[] {
+  const vals: string[] = [];
+  for (const n of ctxNodes) {
+    if (!isTag(n)) continue;
+    if (test.kind === 'attr') {
+      if (test.name === '*') {
+        for (const v of Object.values(n.attribs ?? {})) vals.push(v);
+      } else {
+        const v = n.attribs?.[test.name];
+        if (v !== undefined) vals.push(v);
+      }
+    } else if (test.kind === 'text') {
+      const txt = directText(n);
+      if (txt) vals.push(txt);
+    }
+  }
+  return vals;
+}
+
+/* ------------------------------------------------------------------ */
+/* DOM helpers                                                         */
+/* ------------------------------------------------------------------ */
+
+function elementChildren(node: AnyNode): Element[] {
+  const kids = (node as { children?: AnyNode[] }).children ?? [];
+  return kids.filter(isTag);
+}
+
+/** All descendant elements of `node` in document order (pre-order DFS),
+ *  excluding `node` itself. */
+function descendantElements(node: AnyNode): Element[] {
+  const out: Element[] = [];
+  const visit = (n: AnyNode): void => {
+    for (const c of elementChildren(n)) {
+      out.push(c);
+      visit(c);
+    }
+  };
+  visit(node);
+  return out;
+}
+
+function directText(node: Element): string {
+  const kids = (node.children ?? []).filter(isText).map((t) => t.data);
+  return normalize(kids.join(''));
+}
+
+function normalize(s: string): string {
+  return s.replace(/\s+/g, ' ').trim();
+}
+
+/* ------------------------------------------------------------------ */
+/* Path parsing                                                        */
+/* ------------------------------------------------------------------ */
+
+function parsePath(expr: string): { steps: PathStep[]; fromRoot: boolean } {
+  // Split into raw step strings, tracking the axis ('/' child, '//'
+  // descendant) that precedes each step. A leading '//' or relative path
+  // starts on the descendant / child axis respectively.
+  let i = 0;
+  let fromRoot = false;
+  let nextAxis: 'child' | 'descendant';
+  if (expr.startsWith('//')) {
+    nextAxis = 'descendant';
+    fromRoot = true;
+    i = 2;
+  } else if (expr.startsWith('/')) {
+    nextAxis = 'child';
+    fromRoot = true;
+    i = 1;
+  } else {
+    nextAxis = 'child';
+  }
+
+  const steps: PathStep[] = [];
+  while (i < expr.length) {
+    const { raw, next } = readStepRaw(expr, i);
+    i = next;
+    if (raw) steps.push(parseStep(raw, nextAxis));
+    // Determine the axis for the following step from the separator.
+    if (i < expr.length) {
+      if (expr.startsWith('//', i)) {
+        nextAxis = 'descendant';
+        i += 2;
+      } else if (expr[i] === '/') {
+        nextAxis = 'child';
+        i += 1;
+      } else {
+        throw new XPathError(`unexpected character '${expr[i]}' in XPath at ${i}`);
+      }
+    }
+  }
+  if (steps.length === 0) throw new XPathError('XPath expression has no steps');
+  return { steps, fromRoot };
+}
+
+/** Read one step substring (node test + predicates), respecting brackets
+ *  and quotes so a '/' inside a predicate doesn't split the step. */
+function readStepRaw(expr: string, start: number): { raw: string; next: number } {
+  let depth = 0;
+  let quote = '';
+  let i = start;
+  for (; i < expr.length; i++) {
+    const ch = expr[i]!;
+    if (quote) {
+      if (ch === quote) quote = '';
+      continue;
+    }
+    if (ch === "'" || ch === '"') quote = ch;
+    else if (ch === '[') depth++;
+    else if (ch === ']') depth--;
+    else if (ch === '/' && depth === 0) break;
+  }
+  return { raw: expr.slice(start, i).trim(), next: i };
+}
+
+function parseStep(raw: string, axis: 'child' | 'descendant'): PathStep {
+  // Node test ends at the first top-level '['.
+  let testEnd = raw.length;
+  let depth = 0;
+  let quote = '';
+  for (let i = 0; i < raw.length; i++) {
+    const ch = raw[i]!;
+    if (quote) {
+      if (ch === quote) quote = '';
+      continue;
+    }
+    if (ch === "'" || ch === '"') quote = ch;
+    else if (ch === '[') {
+      if (depth === 0) {
+        testEnd = i;
+        break;
+      }
+      depth++;
+    }
+  }
+  const testStr = raw.slice(0, testEnd).trim();
+  const nodeTest = parseNodeTest(testStr);
+  const predicates: PredFn[] = [];
+  let i = testEnd;
+  while (i < raw.length) {
+    if (raw[i] === '[') {
+      const close = matchBracket(raw, i);
+      predicates.push(compilePredicate(raw.slice(i + 1, close).trim()));
+      i = close + 1;
+    } else if (raw[i] === ' ') {
+      i++;
+    } else {
+      throw new XPathError(`unexpected '${raw[i]}' after node test in '${raw}'`);
+    }
+  }
+  return { axis, nodeTest, predicates };
+}
+
+function parseNodeTest(s: string): NodeTest {
+  if (s === '*') return { kind: 'wildcard' };
+  if (s === 'node()') return { kind: 'node' };
+  if (s === 'text()') return { kind: 'text' };
+  if (s.startsWith('@')) return { kind: 'attr', name: s.slice(1) };
+  if (/^[A-Za-z_][\w\-:]*$/.test(s)) return { kind: 'name', name: s };
+  throw new XPathError(`unsupported node test: '${s}'`);
+}
+
+function matchBracket(s: string, open: number): number {
+  let depth = 0;
+  let quote = '';
+  for (let i = open; i < s.length; i++) {
+    const ch = s[i]!;
+    if (quote) {
+      if (ch === quote) quote = '';
+      continue;
+    }
+    if (ch === "'" || ch === '"') quote = ch;
+    else if (ch === '[') depth++;
+    else if (ch === ']') {
+      depth--;
+      if (depth === 0) return i;
+    }
+  }
+  throw new XPathError('unbalanced `[` in XPath predicate');
+}
+
+/* ------------------------------------------------------------------ */
+/* Predicate compilation                                               */
+/* ------------------------------------------------------------------ */
+
+function compilePredicate(src: string): PredFn {
+  const parser = new PredParser(src);
+  const fn = parser.parseOr();
+  parser.expectEnd();
+  return fn;
+}
+
+const PATH_CHARS = /[A-Za-z0-9_\-:@*./]/;
+
+class PredParser {
+  private p = 0;
+  constructor(private readonly s: string) {}
+
+  expectEnd(): void {
+    this.ws();
+    if (this.p !== this.s.length) {
+      throw new XPathError(`trailing tokens in predicate near '${this.s.slice(this.p)}'`);
+    }
+  }
+
+  parseOr(): PredFn {
+    let left = this.parseAnd();
+    while (this.matchWord('or')) {
+      const right = this.parseAnd();
+      const l = left;
+      left = (env) => toBoolean(l(env)) || toBoolean(right(env));
+    }
+    return left;
+  }
+
+  private parseAnd(): PredFn {
+    let left = this.parseComparison();
+    while (this.matchWord('and')) {
+      const right = this.parseComparison();
+      const l = left;
+      left = (env) => toBoolean(l(env)) && toBoolean(right(env));
+    }
+    return left;
+  }
+
+  private parseComparison(): PredFn {
+    const left = this.parseAdditive();
+    this.ws();
+    const op = this.readCompareOp();
+    if (!op) return left.fn;
+    const right = this.parseAdditive();
+    const l = left.fn;
+    const r = right.fn;
+    return (env) => compareVals(l(env), op, r(env));
+  }
+
+  private parseAdditive(): { fn: PredFn } {
+    let left = this.parseUnary();
+    for (;;) {
+      this.ws();
+      const ch = this.s[this.p];
+      if (ch === '+' || ch === '-') {
+        this.p++;
+        const right = this.parseUnary();
+        const l = left.fn;
+        const r = right.fn;
+        const op = ch;
+        left = {
+          fn: (env) => {
+            const a = toNumber(l(env));
+            const b = toNumber(r(env));
+            return op === '+' ? a + b : a - b;
+          },
+        };
+      } else break;
+    }
+    return left;
+  }
+
+  private parseUnary(): { fn: PredFn } {
+    this.ws();
+    if (this.s[this.p] === '-') {
+      this.p++;
+      const operand = this.parseUnary();
+      const f = operand.fn;
+      return { fn: (env) => -toNumber(f(env)) };
+    }
+    return this.parsePrimary();
+  }
+
+  private parsePrimary(): { fn: PredFn } {
+    this.ws();
+    const ch = this.s[this.p];
+    if (ch === undefined) throw new XPathError('unexpected end of predicate');
+    if (ch === '(') {
+      this.p++;
+      const inner = this.parseOr();
+      this.ws();
+      if (this.s[this.p] !== ')') throw new XPathError('missing `)` in predicate');
+      this.p++;
+      return { fn: inner };
+    }
+    if (ch === "'" || ch === '"') {
+      const str = this.readString();
+      return { fn: () => str };
+    }
+    if (/[0-9]/.test(ch)) {
+      const num = this.readNumber();
+      return { fn: () => num };
+    }
+    // Function call?  name '(' ...
+    const fnName = this.peekFunctionName();
+    if (fnName) return { fn: this.parseFunction(fnName) };
+    // Relative location path / attribute term.
+    if (PATH_CHARS.test(ch)) return this.parsePathTerm();
+    throw new XPathError(`unexpected character '${ch}' in predicate`);
+  }
+
+  /** A path term carries both a value (string-value of the node-set) and
+   *  an existence test, so `[@a]` means "has @a" while `[@a='x']` compares
+   *  its value. */
+  private parsePathTerm(): { fn: PredFn } {
+    const start = this.p;
+    while (this.p < this.s.length && PATH_CHARS.test(this.s[this.p]!)) this.p++;
+    const pathStr = this.s.slice(start, this.p);
+    if (this.s[this.p] === '[') {
+      throw new XPathError('nested predicates inside a predicate are not supported');
+    }
+    const fn: PredFn = (env) => {
+      const res = evalRelPath(env.$, env.node, pathStr);
+      // Bare path term → existence; used as a value only inside a
+      // comparison, where compareVals coerces via stringValue.
+      return res.kind === 'nodes' ? res.nodes : res.values;
+    };
+    return { fn };
+  }
+
+  private parseFunction(name: string): PredFn {
+    // Consume name + '('.
+    this.p += name.length;
+    this.ws();
+    this.p++; // '('
+    const args: PredFn[] = [];
+    this.ws();
+    if (this.s[this.p] !== ')') {
+      for (;;) {
+        args.push(this.parseOr());
+        this.ws();
+        if (this.s[this.p] === ',') {
+          this.p++;
+          continue;
+        }
+        break;
+      }
+    }
+    this.ws();
+    if (this.s[this.p] !== ')') throw new XPathError(`missing ) in ${name}()`);
+    this.p++;
+
+    switch (name) {
+      case 'text':
+        return (env) => directText(env.node);
+      case 'position':
+        return (env) => env.position;
+      case 'last':
+        return (env) => env.last;
+      case 'true':
+        return () => true;
+      case 'false':
+        return () => false;
+      case 'not':
+        requireArgs(name, args, 1);
+        return (env) => !toBoolean(args[0]!(env));
+      case 'contains':
+        requireArgs(name, args, 2);
+        return (env) => toStr(args[0]!(env)).includes(toStr(args[1]!(env)));
+      case 'starts-with':
+        requireArgs(name, args, 2);
+        return (env) => toStr(args[0]!(env)).startsWith(toStr(args[1]!(env)));
+      case 'ends-with':
+        requireArgs(name, args, 2);
+        return (env) => toStr(args[0]!(env)).endsWith(toStr(args[1]!(env)));
+      case 'normalize-space':
+        return (env) => normalize(args.length ? toStr(args[0]!(env)) : env.$(env.node).text());
+      case 'string':
+        return (env) => (args.length ? toStr(args[0]!(env)) : env.$(env.node).text());
+      case 'string-length':
+        return (env) => (args.length ? toStr(args[0]!(env)) : env.$(env.node).text()).length;
+      default:
+        throw new XPathError(`unsupported function: ${name}()`);
+    }
+  }
+
+  private peekFunctionName(): string | null {
+    const m = /^([A-Za-z][\w-]*)\s*\(/.exec(this.s.slice(this.p));
+    if (!m) return null;
+    // text() / node() are node tests, never reached here at primary
+    // position except as functions — both are handled in the switch.
+    return m[1]!;
+  }
+
+  private readString(): string {
+    const quote = this.s[this.p]!;
+    let i = this.p + 1;
+    let out = '';
+    while (i < this.s.length && this.s[i] !== quote) out += this.s[i++];
+    if (i >= this.s.length) throw new XPathError('unterminated string in predicate');
+    this.p = i + 1;
+    return out;
+  }
+
+  private readNumber(): number {
+    const start = this.p;
+    while (this.p < this.s.length && /[0-9.]/.test(this.s[this.p]!)) this.p++;
+    const n = Number(this.s.slice(start, this.p));
+    if (Number.isNaN(n)) throw new XPathError('invalid number in predicate');
+    return n;
+  }
+
+  private readCompareOp(): string | null {
+    const two = this.s.slice(this.p, this.p + 2);
+    if (two === '!=' || two === '<=' || two === '>=') {
+      this.p += 2;
+      return two;
+    }
+    const one = this.s[this.p];
+    if (one === '=' || one === '<' || one === '>') {
+      this.p += 1;
+      return one;
+    }
+    return null;
+  }
+
+  private matchWord(word: string): boolean {
+    this.ws();
+    const slice = this.s.slice(this.p, this.p + word.length);
+    if (slice !== word) return false;
+    const after = this.s[this.p + word.length];
+    if (after !== undefined && /[\w-]/.test(after)) return false; // not a word boundary
+    this.p += word.length;
+    return true;
+  }
+
+  private ws(): void {
+    while (this.p < this.s.length && /\s/.test(this.s[this.p]!)) this.p++;
+  }
+}
+
+function requireArgs(name: string, args: PredFn[], n: number): void {
+  if (args.length !== n) throw new XPathError(`${name}() expects ${n} argument(s)`);
+}
+
+/* ------------------------------------------------------------------ */
+/* Relative path evaluation (used by predicate path terms)             */
+/* ------------------------------------------------------------------ */
+
+function evalRelPath($: CheerioAPI, node: Element, pathStr: string): XPathResult {
+  // '.' = self; '@attr' = attribute of self; './/x' / '//x' = descendant
+  // of self; './x' / 'x' = child of self; '/x' = absolute from root.
+  let str = pathStr;
+  let ctx: AnyNode[] = [node];
+  let forceFirstDescendant = false;
+
+  if (str === '.') return { kind: 'nodes', nodes: [node] };
+  if (str.startsWith('@')) {
+    return { kind: 'values', values: extractTerminal([node], { kind: 'attr', name: str.slice(1) }) };
+  }
+  if (str.startsWith('.//')) {
+    str = str.slice(3);
+    forceFirstDescendant = true;
+  } else if (str.startsWith('./')) {
+    str = str.slice(2);
+  } else if (str.startsWith('//')) {
+    str = str.slice(2);
+    forceFirstDescendant = true;
+  } else if (str.startsWith('/')) {
+    str = str.slice(1);
+    ctx = [rootNode($)];
+  }
+
+  const { steps } = parsePath((forceFirstDescendant ? '//' : '') + str);
+  return runPath($, ctx, steps);
+}
+
+/* ------------------------------------------------------------------ */
+/* Value coercion                                                      */
+/* ------------------------------------------------------------------ */
+
+function toBoolean(v: XVal): boolean {
+  if (typeof v === 'boolean') return v;
+  if (typeof v === 'number') return v !== 0 && !Number.isNaN(v);
+  if (typeof v === 'string') return v.length > 0;
+  if (Array.isArray(v)) return v.length > 0;
+  return false;
+}
+
+function toNumber(v: XVal): number {
+  if (typeof v === 'number') return v;
+  if (typeof v === 'boolean') return v ? 1 : 0;
+  return Number(toStr(v));
+}
+
+function toStr(v: XVal): string {
+  if (typeof v === 'string') return v;
+  if (typeof v === 'number') return String(v);
+  if (typeof v === 'boolean') return v ? 'true' : 'false';
+  if (Array.isArray(v)) {
+    const first = v[0];
+    if (first === undefined) return '';
+    // Value-set (attribute / text() strings) → first string; node-set →
+    // string value (text content) of the first node.
+    return typeof first === 'string' ? first : normalize(textOf(first));
+  }
+  return '';
+}
+
+function textOf(node: Element): string {
+  // domhandler text aggregation without needing the cheerio instance.
+  let out = '';
+  const walk = (n: AnyNode): void => {
+    if (isText(n)) out += n.data;
+    else if (isTag(n)) for (const c of n.children) walk(c);
+  };
+  for (const c of node.children) walk(c);
+  return out;
+}
+
+function compareVals(a: XVal, op: string, b: XVal): boolean {
+  // String-set comparison: when a side is a node-set of strings (attribute
+  // values), XPath '=' is true if ANY member matches. Handles `[@a='x']`
+  // when multiple matches exist as well as the common single-value case.
+  if (op === '=' || op === '!=') {
+    const eq = looseEqual(a, b);
+    return op === '=' ? eq : !eq;
+  }
+  const na = toNumber(a);
+  const nb = toNumber(b);
+  if (Number.isFinite(na) && Number.isFinite(nb)) {
+    return op === '<' ? na < nb : op === '<=' ? na <= nb : op === '>' ? na > nb : na >= nb;
+  }
+  const sa = toStr(a);
+  const sb = toStr(b);
+  return op === '<' ? sa < sb : op === '<=' ? sa <= sb : op === '>' ? sa > sb : sa >= sb;
+}
+
+function looseEqual(a: XVal, b: XVal): boolean {
+  // If either side is a string node-set, compare each member.
+  const aMembers = members(a);
+  const bMembers = members(b);
+  for (const x of aMembers) {
+    for (const y of bMembers) {
+      if (memberEq(x, y)) return true;
+    }
+  }
+  return false;
+}
+
+function members(v: XVal): Array<string | number | boolean> {
+  if (Array.isArray(v)) {
+    // value-set → the strings as-is; node-set → each node's string value.
+    return v.map((n) => (typeof n === 'string' ? n : normalize(textOf(n))));
+  }
+  if (v === undefined) return [''];
+  return [v];
+}
+
+function memberEq(x: string | number | boolean, y: string | number | boolean): boolean {
+  if (x === y) return true;
+  if (typeof x === 'number' || typeof y === 'number') return Number(x) === Number(y);
+  return String(x) === String(y);
+}

packages/db/src/migrations.ts

@@ -1669,6 +1669,33 @@ const MIGRATIONS: Migration[] = [
       }
     },
   },
+  {
+    version: 71,
+    name: 'add_pdf_metadata',
+    // V2 Faz 16 — PDF document metadata, extracted by the post-crawl
+    // `runPdfMetadataProbes` pass (XMP packet preferred — uncompressed by
+    // the PDF spec, so parseable without a full PDF library — with an
+    // uncompressed Info-dictionary fallback). `pdf_probe_status`:
+    // NULL = unprobed, 1 = probed with metadata, 0 = probed but nothing
+    // found, <0 = fetch/parse error. The other columns are NULL until a
+    // successful probe fills them.
+    up: (db) => {
+      const cols = db
+        .prepare('PRAGMA table_info(urls)')
+        .all() as unknown as { name: string }[];
+      const add = (name: string, type: string): void => {
+        if (!cols.some((c) => c.name === name)) {
+          db.exec(`ALTER TABLE urls ADD COLUMN ${name} ${type}`);
+        }
+      };
+      add('pdf_title', 'TEXT');
+      add('pdf_author', 'TEXT');
+      add('pdf_page_count', 'INTEGER');
+      add('pdf_creation_date', 'TEXT');
+      add('pdf_producer', 'TEXT');
+      add('pdf_probe_status', 'INTEGER');
+    },
+  },
 ];
 
 export function runMigrations(db: DatabaseSync): void {

packages/db/src/project-db.ts

@@ -33,6 +33,7 @@ import type {
   UrlCategory,
   UrlDetail,
 } from '@freecrawl/shared-types';
+import { issueCategoriesBySeverity } from '@freecrawl/shared-types';
 import { runMigrations } from './migrations.js';
 
 interface UrlRowDb {
@@ -197,6 +198,12 @@ interface UrlRowDb {
   manifest_display: string | null;
   manifest_scope: string | null;
   manifest_icon_count: number;
+  pdf_title: string | null;
+  pdf_author: string | null;
+  pdf_page_count: number | null;
+  pdf_creation_date: string | null;
+  pdf_producer: string | null;
+  pdf_probe_status: number | null;
   landmark_main: number;
   skip_link_present: number;
   aria_invalid_roles: number;
@@ -4338,6 +4345,9 @@ export class ProjectDb {
       http2NotSupported: countWhere(
         `${html} AND http_protocol = 'http/1.1'`,
       ),
+      http3NotSupported: countWhere(
+        `${html} AND http_protocol IS NOT NULL AND http_protocol != 'h3'`,
+      ),
       renderBlocking: countWhere(`${html} AND render_blocking_count > 5`),
       keepaliveDisabled: countWhere('is_external = 0 AND keep_alive = 0'),
       titlePlaceholder: countWhere(
@@ -5715,6 +5725,63 @@ export class ProjectDb {
       });
   }
 
+  /**
+   * V2 Faz 16 — internal 2xx PDFs whose metadata hasn't been probed yet
+   * (`pdf_probe_status IS NULL`). A failed/empty probe sets the status
+   * (0 / negative) so the row drops out of this set and isn't re-fetched
+   * next crawl. PDFs are their own URL rows, so this is a plain per-row
+   * update (no fan-out, unlike manifests/images).
+   */
+  unprobedPdfUrls(limit = 1000): string[] {
+    const rows = this.db
+      .prepare(
+        `SELECT url FROM urls
+          WHERE content_kind = 'pdf'
+            AND is_external = 0
+            AND status_code >= 200 AND status_code < 300
+            AND pdf_probe_status IS NULL
+          LIMIT ?`,
+      )
+      .all(limit) as { url: string }[];
+    return rows.map((r) => r.url).filter((u) => typeof u === 'string' && u.length > 0);
+  }
+
+  /**
+   * Persist probed PDF metadata onto the PDF's own URL row. `status`:
+   * 1 = metadata found, 0 = probed but nothing parseable, negative =
+   * fetch/parse error. Metadata columns stay NULL unless a value was read.
+   */
+  setPdfMetadata(input: {
+    url: string;
+    title: string | null;
+    author: string | null;
+    pageCount: number | null;
+    creationDate: string | null;
+    producer: string | null;
+    status: number;
+  }): void {
+    this.db
+      .prepare(
+        `UPDATE urls
+            SET pdf_title = :title,
+                pdf_author = :author,
+                pdf_page_count = :page_count,
+                pdf_creation_date = :creation_date,
+                pdf_producer = :producer,
+                pdf_probe_status = :status
+          WHERE url = :url`,
+      )
+      .run({
+        url: input.url,
+        title: input.title,
+        author: input.author,
+        page_count: input.pageCount,
+        creation_date: input.creationDate,
+        producer: input.producer,
+        status: input.status,
+      });
+  }
+
   /**
    * V2 Faz 16 #1 — Distinct `og:image` / `twitter:image` URLs that
    * haven't had their pixel dimensions probed yet. A single image is
@@ -6495,6 +6562,11 @@ export class ProjectDb {
     manifestDisplay: r.manifest_display ?? null,
     manifestScope: r.manifest_scope ?? null,
     manifestIconCount: r.manifest_icon_count ?? 0,
+    pdfTitle: r.pdf_title ?? null,
+    pdfAuthor: r.pdf_author ?? null,
+    pdfPageCount: r.pdf_page_count ?? null,
+    pdfCreationDate: r.pdf_creation_date ?? null,
+    pdfProducer: r.pdf_producer ?? null,
     landmarkMain: r.landmark_main ?? 0,
     skipLinkPresent: r.skip_link_present ?? 0,
     ariaInvalidRoles: r.aria_invalid_roles ?? 0,
@@ -7343,6 +7415,32 @@ function categoryWhereClause(cat: UrlCategory): string | null {
       // a free CWV win on multi-resource pages.
       return `is_external = 0 AND content_kind = 'html'
               AND http_protocol = 'http/1.1'`;
+    case 'issues:http3-not-supported':
+      // Origin does not advertise HTTP/3 via Alt-Svc (http_protocol is
+      // 'h2' or 'http/1.1', not 'h3'). Informational — HTTP/3 (QUIC) cuts
+      // connection-setup latency, but adoption is still partial.
+      return `is_external = 0 AND content_kind = 'html'
+              AND http_protocol IS NOT NULL AND http_protocol != 'h3'`;
+    case 'issues:severity-critical':
+    case 'issues:severity-warning':
+    case 'issues:severity-info': {
+      // V2 Faz 15 — virtual severity rollup: OR-compose the WHERE clauses
+      // of every issue category mapped to this tier in ISSUE_SEVERITY, so
+      // a single category yields the deduped set of URLs flagged by any
+      // issue of that severity. Members whose clause is null (none) are
+      // skipped; an empty tier matches nothing rather than everything.
+      const tier =
+        cat === 'issues:severity-critical'
+          ? 'critical'
+          : cat === 'issues:severity-warning'
+            ? 'warning'
+            : 'info';
+      const clauses = issueCategoriesBySeverity(tier)
+        .map((c) => categoryWhereClause(c))
+        .filter((c): c is string => c !== null)
+        .map((c) => `(${c})`);
+      return clauses.length > 0 ? clauses.join(' OR ') : '1 = 0';
+    }
     case 'issues:render-blocking':
       // 5+ render-blocking head resources is the SF threshold — every one
       // delays first-paint until fetched + parsed. Lighthouse flags this
@@ -8112,8 +8210,9 @@ function categoryWhereClause(cat: UrlCategory): string | null {
       // V1 Faz 3 — Custom Extraction tab. Pages where at least one
       // custom-extraction rule produced a non-null result. The Detail
       // panel's "Extracted Data" sub-tab shows the full JSON map per
-      // page.
-      return `is_external = 0 AND content_kind = 'html'
+      // page. Content kind is not constrained: JSONPath rules (V2 Faz 6)
+      // produce results on non-HTML JSON responses too.
+      return `is_external = 0
               AND extraction_results IS NOT NULL
               AND extraction_results != ''
               AND extraction_results != '{}'`;

packages/shared-types/src/crawl.ts

@@ -217,6 +217,14 @@ export type UrlCategory =
   | 'issues:hreflang-inconsistent-lang'
   | 'issues:page-many-requests'
   | 'issues:over-budget'
+  | 'issues:http3-not-supported'
+  // V2 Faz 15 — virtual severity-rollup categories: the union of every
+  // issue category mapped to that tier in ISSUE_SEVERITY (issue-severity.ts).
+  // Backs the "split bulk export by severity" feature and is queryable
+  // anywhere a UrlCategory is accepted (query_urls, sidebar, MCP).
+  | 'issues:severity-critical'
+  | 'issues:severity-warning'
+  | 'issues:severity-info'
 
 export type Indexability =
   | 'indexable'
@@ -308,6 +316,16 @@ export interface CrawlUrlRow {
   manifestScope: string | null;
   /** Number of icons in the parsed manifest's `icons` array. */
   manifestIconCount: number;
+  /** V2 Faz 16 — PDF document title (XMP `dc:title` or Info `/Title`). Null when not a PDF or no metadata found. */
+  pdfTitle: string | null;
+  /** PDF author (XMP `dc:creator` or Info `/Author`). */
+  pdfAuthor: string | null;
+  /** PDF page count (best-effort; null when undeterminable from the fetched bytes). */
+  pdfPageCount: number | null;
+  /** PDF creation date as ISO-8601 (parsed from XMP `xmp:CreateDate` or Info `/CreationDate D:...`). */
+  pdfCreationDate: string | null;
+  /** PDF producer / generator string (XMP `pdf:Producer` or Info `/Producer`). */
+  pdfProducer: string | null;
   /** 1 when the page has a `<main>` element OR `role="main"`. */
   landmarkMain: number;
   /** 1 when the first focusable `<a href="#…">` matches a skip-link convention. */
@@ -763,13 +781,22 @@ export interface CrawlConfig {
   /**
    * HTTP authentication applied on every fetch. `none` is the default.
    * `basic` sends `Authorization: Basic <base64(user:pass)>`; `bearer`
-   * sends `Authorization: Bearer <token>`. Digest auth is not supported
-   * yet (challenge/response state-machine).
+   * sends `Authorization: Bearer <token>`; `digest` performs the RFC 2617
+   * challenge-response handshake on the first 401.
    */
   auth: HttpAuth;
+  /**
+   * Form-based (cookie session) login run once before the crawl. Lets
+   * FreeCrawl crawl content behind a multi-step HTML login form by
+   * establishing the session cookies first and injecting them into every
+   * subsequent request.
+   */
+  formLogin: FormLoginConfig;
   /**
    * Proxy URL — overrides `HTTPS_PROXY` / `HTTP_PROXY` env vars when
-   * non-empty. Same syntax: `http://user:pass@host:port`.
+   * non-empty. HTTP/HTTPS proxies use `http://user:pass@host:port`;
+   * SOCKS proxies use `socks5://`, `socks5h://`, `socks4://` or
+   * `socks4a://` (the `h`/`4a` variants resolve DNS at the proxy).
    */
   proxyUrl: string;
   /**
@@ -854,6 +881,15 @@ export interface CrawlConfig {
    * each image is read (enough for the header), so cost is minimal.
    */
   probeSocialImages: boolean;
+  /**
+   * V2 Faz 16 — after the crawl, fetch each internal PDF (ranged GET,
+   * first few MB) and extract document metadata: title, author, page
+   * count, creation date, producer. Prefers the XMP packet (uncompressed
+   * by the PDF spec, so reliably parseable without a full PDF library)
+   * and falls back to an uncompressed Info-dictionary scan. Default
+   * `true`. No new dependency — a lightweight hand-rolled parser.
+   */
+  probePdfMetadata: boolean;
   /**
    * When the post-norm "Duplicate URL" issue filter runs, normalise
    * URLs (lowercase, strip query, trim trailing slash) before
@@ -1082,12 +1118,55 @@ export interface JsRenderConfig {
 }
 
 export interface HttpAuth {
-  type: 'none' | 'basic' | 'bearer';
+  /**
+   *  - `none`   — no Authorization header.
+   *  - `basic`  — `Authorization: Basic <base64(user:pass)>`.
+   *  - `bearer` — `Authorization: Bearer <token>`.
+   *  - `digest` — RFC 2617/7616 challenge-response. The first request that
+   *               returns `401 WWW-Authenticate: Digest …` is retried with a
+   *               computed `Authorization: Digest …` header (MD5 / SHA-256,
+   *               `qop=auth`, `MD5-sess` supported).
+   */
+  type: 'none' | 'basic' | 'bearer' | 'digest';
   username?: string;
   password?: string;
   token?: string;
 }
 
+/**
+ * A single step in a form-based login sequence. Steps run in order before
+ * the crawl starts, sharing one session cookie jar so the authenticated
+ * session cookies established here are injected into every crawl request.
+ *
+ * Field values and the step URL may reference variables captured by an
+ * earlier step via `{{name}}` interpolation — the canonical multi-step
+ * pattern is: GET the login page → capture the CSRF token → POST the
+ * credentials together with the token.
+ */
+export interface FormLoginStep {
+  /** Absolute URL to request. Supports `{{var}}` interpolation. */
+  url: string;
+  /** `GET` (e.g. to fetch a login page for CSRF capture) or `POST`
+   *  (submit credentials as `application/x-www-form-urlencoded`). */
+  method: 'GET' | 'POST';
+  /** Form fields sent as the POST body. Values support `{{var}}`
+   *  interpolation. Ignored for GET (kept for editing convenience). */
+  fields: { name: string; value: string }[];
+  /** After the response, capture values into named variables usable by
+   *  later steps. `selector` is a CSS selector against the response HTML;
+   *  `attribute` (default `value`) is the attribute to read — typically
+   *  `input[name="_csrf"]` / `value` for a hidden CSRF token. */
+  captures: { name: string; selector: string; attribute?: string }[];
+}
+
+export interface FormLoginConfig {
+  /** Master switch — when off the login sequence never runs and no
+   *  session cookies are injected. */
+  enabled: boolean;
+  /** Ordered login steps. Empty = nothing to do even when enabled. */
+  steps: FormLoginStep[];
+}
+
 /**
  * One row of the Custom Extraction table. Either CSS-selector or regex
  * driven; output shape and multi-match handling are independently
@@ -1097,20 +1176,37 @@ export interface HttpAuth {
 export interface CustomExtractionRule {
   /** User-visible name. Stored verbatim — also used as the JSON key. */
   name: string;
-  /** Extraction strategy. `css` uses cheerio; `regex` runs against raw HTML. */
-  type: 'css' | 'regex';
-  /** CSS selector when `type = 'css'`; regex pattern (no flags) when `type = 'regex'`. */
+  /**
+   * Extraction strategy:
+   *  - `css`      — cheerio CSS selector against the parsed DOM.
+   *  - `regex`    — JavaScript RegExp against the raw HTML string.
+   *  - `xpath`    — XPath 1.0 subset evaluated over the parsed DOM.
+   *  - `jsonpath` — JSONPath against a non-HTML JSON response body. Only
+   *                 runs on responses whose body parses as JSON (e.g.
+   *                 `application/json` APIs); ignored on HTML pages.
+   */
+  type: 'css' | 'regex' | 'xpath' | 'jsonpath';
+  /**
+   * `css`  → CSS selector.
+   * `regex`→ regex pattern (no flags — /g is implicit).
+   * `xpath`→ XPath location path, e.g. `//meta[@property='og:title']/@content`.
+   * `jsonpath` → JSONPath expression, e.g. `$.products[*].price`.
+   */
   selector: string;
-  /** Attribute to read when `output = 'attribute'`. Ignored otherwise. */
+  /** Attribute to read when `output = 'attribute'`. Ignored otherwise.
+   *  For `xpath`, an `.../@attr` terminal step reads the attribute directly
+   *  and overrides this field. */
   attribute?: string;
   /**
    * What to read off each match.
-   *  - `text`        — visible text content (CSS only).
-   *  - `attribute`   — value of `attribute` (CSS only).
-   *  - `inner_html`  — innerHTML (CSS only).
-   *  - `outer_html`  — outerHTML (CSS only).
+   *  - `text`        — visible text content (CSS / XPath element matches).
+   *  - `attribute`   — value of `attribute` (CSS / XPath element matches).
+   *  - `inner_html`  — innerHTML (CSS / XPath).
+   *  - `outer_html`  — outerHTML (CSS / XPath).
    *  - `count`       — match count, ignores `multi`.
    *  - `regex_group` — regex capture group 1 (regex only).
+   * For `jsonpath` and XPath `@attr` / `text()` terminals the matched value
+   * is used directly (this field is then ignored).
    */
   output: 'text' | 'attribute' | 'inner_html' | 'outer_html' | 'count' | 'regex_group';
   /**
@@ -1322,6 +1418,8 @@ export interface OverviewCounts {
     queryStringTooLong: number;
     folderDepthTooDeep: number;
     http2NotSupported: number;
+    /** Indexable HTML pages whose origin does not advertise HTTP/3 via Alt-Svc. */
+    http3NotSupported: number;
     renderBlocking: number;
     keepaliveDisabled: number;
     titlePlaceholder: number;
@@ -1788,6 +1886,7 @@ export const DEFAULT_CRAWL_CONFIG: CrawlConfig = {
   webhookUrl: '',
   customExtractionRules: [],
   auth: { type: 'none' },
+  formLogin: { enabled: false, steps: [] },
   proxyUrl: '',
   excludeExtensions: [],
   maxRedirects: 10,
@@ -1801,6 +1900,7 @@ export const DEFAULT_CRAWL_CONFIG: CrawlConfig = {
   probeTlsCerts: true,
   probeManifestJson: true,
   probeSocialImages: true,
+  probePdfMetadata: true,
   dedupePreNormalize: true,
   cdnHosts: [],
   maxLinksPerPage: 100,

packages/shared-types/src/index.ts

@@ -1,4 +1,5 @@
 export * from './crawl.js';
+export * from './issue-severity.js';
 export * from './ipc.js';
 export * from './project.js';
 export * from './integrations.js';

packages/shared-types/src/issue-severity.ts

@@ -0,0 +1,217 @@
+/**
+ * V2 Faz 15 — issue → severity classification.
+ *
+ * A single source of truth mapping every `issues:*` UrlCategory to one of
+ * three severity tiers (Critical / Warning / Info). Powers the "split
+ * bulk export by severity" feature (three rollup files/sheets) and the
+ * virtual `issues:severity-*` categories, and is reusable anywhere a
+ * machine-readable issue severity is useful (sidebar, MCP, reports).
+ *
+ * Tiering rationale:
+ *   - `critical` — actively breaks indexing/crawling, security holes, or
+ *     broken resources (missing title, broken internal links, redirect
+ *     loops, canonical→non-200/noindex, expired TLS, mixed/insecure
+ *     content, empty page, exact duplicate content).
+ *   - `warning`  — suboptimal and worth fixing but not breaking (length
+ *     limits, duplicate meta/H1, missing alt/security headers, slow TTFB,
+ *     schema gaps, social-image problems, accessibility defects).
+ *   - `info`     — informational / stylistic / nice-to-have (URL cosmetics,
+ *     self-referencing canonical, missing favicon/manifest, protocol
+ *     advertisements, readability scores).
+ *
+ * Severity is inherently a judgement call; this is a sensible default an
+ * auditor can reason about, not an authoritative standard.
+ */
+import type { UrlCategory } from './crawl.js';
+
+export type IssueSeverity = 'critical' | 'warning' | 'info';
+
+/** Issue categories that break indexing/crawling, security, or resources. */
+const CRITICAL: UrlCategory[] = [
+  'issues:title-missing',
+  'issues:multiple-canonicals',
+  'issues:canonical-mismatch',
+  'issues:canonical-to-non-200',
+  'issues:canonical-to-noindex',
+  'issues:response-very-slow',
+  'issues:mixed-content',
+  'issues:redirect-loop',
+  'issues:redirect-self',
+  'issues:broken-links-all',
+  'issues:broken-links-internal',
+  'issues:duplicate-content-exact',
+  'issues:insecure-form-action',
+  'issues:title-placeholder',
+  'issues:ssl-cert-expired',
+  'issues:page-too-large-critical',
+  'issues:page-empty',
+  'issues:cors-wildcard-with-credentials',
+  'issues:http-not-https',
+  'issues:mixed-content-active',
+];
+
+/** Suboptimal but non-breaking issues worth fixing. */
+const WARNING: UrlCategory[] = [
+  'issues:title-too-long',
+  'issues:title-too-short',
+  'issues:title-duplicate',
+  'issues:meta-missing',
+  'issues:meta-too-long',
+  'issues:meta-too-short',
+  'issues:meta-duplicate',
+  'issues:h1-missing',
+  'issues:h1-duplicate',
+  'issues:h1-multiple',
+  'issues:canonical-missing',
+  'issues:canonical-to-redirect',
+  'issues:content-thin',
+  'issues:response-slow',
+  'issues:page-large',
+  'issues:viewport-missing',
+  'issues:hsts-missing',
+  'issues:x-frame-options-missing',
+  'issues:x-content-type-options-missing',
+  'issues:structured-data-invalid',
+  'issues:pagination-broken',
+  'issues:hreflang-x-default-missing',
+  'issues:redirect-chain-long',
+  'issues:compression-missing',
+  'issues:non-indexable-in-sitemap',
+  'issues:non-200-in-sitemap',
+  'issues:image-missing-alt',
+  'issues:meta-refresh-used',
+  'issues:charset-missing',
+  'issues:amp-validation-errors',
+  'issues:canonical-conflict-near-duplicate',
+  'issues:broken-links-external',
+  'issues:near-duplicate',
+  'issues:hreflang-invalid-code',
+  'issues:hreflang-self-ref-missing',
+  'issues:hreflang-reciprocity-missing',
+  'issues:hreflang-target-issues',
+  'issues:redirect-in-sitemap',
+  'issues:h1-empty',
+  'issues:title-multiple',
+  'issues:url-spaces',
+  'issues:url-malformed',
+  'issues:title-pixel-width-too-long',
+  'issues:meta-pixel-width-too-long',
+  'issues:ttfb-slow',
+  'issues:ttfb-very-slow',
+  'issues:cookie-no-secure',
+  'issues:cookie-no-httponly',
+  'issues:render-blocking',
+  'issues:analytics-multiple-ga4',
+  'issues:image-too-large',
+  'issues:ssl-cert-expiring-soon',
+  'issues:ssl-protocol-old',
+  'issues:ssl-signature-weak',
+  'issues:form-input-unlabeled',
+  'issues:image-broken-src',
+  'issues:aria-invalid-role',
+  'issues:pagination-canonical-conflict',
+  'issues:schema-duplicate-id',
+  'issues:schema-unknown-type',
+  'issues:schema-missing-required',
+  'issues:og-image-not-absolute',
+  'issues:twitter-image-not-absolute',
+  'issues:canonical-not-absolute',
+  'issues:title-single-word',
+  'issues:internal-link-to-redirect',
+  'issues:dead-external-domain',
+  'issues:canonical-chain-multi-hop',
+  'issues:image-slow-loading',
+  'issues:js-only-navigation',
+  'issues:mixed-content-passive',
+  'issues:render-blocking-critical',
+  'issues:og-image-too-large',
+  'issues:twitter-image-too-large',
+  'issues:og-image-wrong-aspect',
+  'issues:twitter-image-wrong-aspect',
+  'issues:low-contrast-text',
+  'issues:focus-outline-suppressed',
+  'issues:hreflang-inconsistent-lang',
+  'issues:over-budget',
+];
+
+/** Informational / stylistic / nice-to-have signals. */
+const INFO: UrlCategory[] = [
+  'issues:heading-skipped-level',
+  'issues:canonical-self-referencing',
+  'issues:canonical-non-self',
+  'issues:url-too-long',
+  'issues:url-uppercase',
+  'issues:url-underscore',
+  'issues:url-multiple-slashes',
+  'issues:url-non-ascii',
+  'issues:lang-missing',
+  'issues:og-missing',
+  'issues:twitter-missing',
+  'issues:csp-missing',
+  'issues:structured-data-missing',
+  'issues:favicon-missing',
+  'issues:url-many-params',
+  'issues:image-empty-alt',
+  'issues:high-boilerplate',
+  'issues:crawled-not-in-sitemap',
+  'issues:h1-too-long',
+  'issues:url-fragment',
+  'issues:link-empty-anchor',
+  'issues:apple-touch-icon-missing',
+  'issues:manifest-missing',
+  'issues:feed-missing',
+  'issues:missing-sri',
+  'issues:cookie-no-samesite',
+  'issues:query-string-too-long',
+  'issues:folder-depth-too-deep',
+  'issues:http2-not-supported',
+  'issues:http3-not-supported',
+  'issues:keepalive-disabled',
+  'issues:analytics-missing',
+  'issues:analytics-ua-legacy',
+  'issues:analytics-pixel-without-policy',
+  'issues:hsts-no-preload',
+  'issues:hsts-max-age-short',
+  'issues:hsts-no-includesubdomains',
+  'issues:anchor-text-too-long',
+  'issues:anchor-text-generic',
+  'issues:images-no-lazy-loading',
+  'issues:images-no-responsive',
+  'issues:landmark-main-missing',
+  'issues:skip-link-missing',
+  'issues:schema-missing-recommended',
+  'issues:target-blank-no-noopener',
+  'issues:description-equals-title',
+  'issues:external-links-too-many',
+  'issues:outlinks-zero',
+  'issues:h1-equals-title',
+  'issues:duplicate-url-post-norm',
+  'issues:description-equals-h1',
+  'issues:text-code-ratio-low',
+  'issues:flesch-very-difficult',
+  'issues:gunning-fog-very-high',
+  'issues:cors-wildcard-origin',
+  'issues:pagination-sequence-break',
+  'issues:links-per-page-too-many',
+  'issues:page-many-requests',
+];
+
+/** Lookup: `issues:*` category → severity tier. Non-issue categories absent. */
+export const ISSUE_SEVERITY: Partial<Record<UrlCategory, IssueSeverity>> = {};
+for (const cat of CRITICAL) ISSUE_SEVERITY[cat] = 'critical';
+for (const cat of WARNING) ISSUE_SEVERITY[cat] = 'warning';
+for (const cat of INFO) ISSUE_SEVERITY[cat] = 'info';
+
+/** The `issues:*` categories belonging to one severity tier. */
+export function issueCategoriesBySeverity(tier: IssueSeverity): UrlCategory[] {
+  if (tier === 'critical') return CRITICAL;
+  if (tier === 'warning') return WARNING;
+  return INFO;
+}
+
+/** Maps a severity tier to its virtual rollup UrlCategory. */
+export const SEVERITY_CATEGORY: Record<IssueSeverity, UrlCategory> = {
+  critical: 'issues:severity-critical',
+  warning: 'issues:severity-warning',
+  info: 'issues:severity-info',
+};