feat: cluster view + column pin/reorder + AMP smoke + MCP read parity

README.md

@@ -44,16 +44,17 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 <td width="50%" valign="top">
 
 ### 🔍 Analysis
-- **150+ SEO issue checks** across 25 top-level tabs
-- Near-duplicate clustering (SimHash + LSH) + exact-duplicate content hash
+- **150+ SEO issue checks** across 30 top-level tabs
+- Near-duplicate clustering (SimHash + LSH) + exact-duplicate content hash + **Cluster view** that groups every near-duplicate cluster on the Duplicates tab
 - Full hreflang validation (reciprocity, self-ref, inconsistent lang)
 - OpenGraph / Twitter Card / JSON-LD / AMP / **Web App Manifest** parsing
+- **AMP smoke validation** — pages declaring themselves AMP get flagged for missing boilerplate, runtime, canonical, or forbidden script/style/legacy tags
 - **Structured-data validation** — duplicate `@id`, malformed `@type`, missing required props
 - **Accessibility** — `<main>` landmark, skip-link, ARIA roles, heading order
 - SSL/TLS cert chain audit, security headers, **CORS audit**
 - **Active vs passive mixed content** split
 - Readability (Flesch, Flesch–Kincaid, Gunning Fog)
-- Custom CSS + Regex extraction (10 rules) + `/regex/` literal Custom Search
+- Custom CSS + Regex extraction (10 rules) — with **live Preview dialog** + **JSON import / export** for sharing rule sets — plus `/regex/` literal Custom Search
 - **PageSpeed Insights** — on-demand Lighthouse audits (mobile + desktop) per URL
 - **Google Search Console + GA4** — clicks / impressions / CTR / position / sessions / users joined onto crawled pages
 - **URL Inspection** — coverage verdict, last-crawl time, canonical decisions in batches
@@ -72,8 +73,9 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 - Live-streaming rows during crawl
 - Advanced AND/OR search (24 fields × 12 ops)
 - **Per-tab quick-filter dropdown** — instant filtering by Overview sub-categories without leaving the tab
-- **List ↔ Tree view toggle** — switch between flat virtual table and URL-path hierarchy with collapsible folders
-- Detail panel with 15 sub-tabs — adds **View Rendered** (post-JS DOM) + **Screenshot** (full/fold/mobile preview) when JS rendering is on
+- **List ↔ Tree ↔ Cluster view toggle** — flat virtual table, URL-path hierarchy with collapsible folders, or near-duplicate cluster groups (Duplicates tab)
+- **Column pin (sticky-left) + drag-to-reorder** — pin a column so it stays visible during horizontal scroll, drag any header's grip handle to rearrange
+- Detail panel with 16 sub-tabs — adds **View Rendered** (post-JS DOM) + **Screenshot** (full/fold/mobile preview) when JS rendering is on
 - **Visualization in its own native window** — open from menu, park on a second monitor while data tables stay free
 - Per-URL **Duplicates** view (cluster siblings)
 - **Live memory monitor** in status bar (RSS / Sys Free / per-URL cost / capacity projection)
@@ -97,7 +99,7 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
 - Project-vs-project compare diff
 - **Google Sheets** + **BigQuery** direct export (OAuth / service-account)
 - **Encrypted project snapshots** — password-protected `.seoproject.enc` (AES-256-GCM + PBKDF2)
-- **MCP server** for AI agents — **drive crawls live from Claude Code**, poll progress, plus full read-only DB access
+- **MCP server** for AI agents — **drive crawls live from Claude Code** with 47 tools covering every tab, detail sub-tab, integration row, and report
 - Webhook on completion + OS notifications
 
 </td>
@@ -268,15 +270,19 @@ FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the acti
 1. **Read-only data access** to the SQLite project — runs alongside the desktop app without contention (WAL allows concurrent readers).
 2. **Live crawl control** — when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to `<userData>/mcp-bridge.json` on app launch).
 
-**14 tools**:
+**47 tools**:
 
 | Group | Tools |
 | :--- | :--- |
-| 📊 **Data queries** (always available) | `get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail` |
+| 📊 **Top-level data** | `get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail` |
+| 🔬 **Per-URL detail sub-tabs** | `get_url_source` (raw + rendered + screenshot paths), `get_url_inlinks`, `get_url_outlinks`, `get_url_images`, `get_url_headers`, `get_url_duplicates`, `get_url_analytics`, `get_url_cert` |
+| 🧩 **Integration rows** | `query_gsc`, `query_ga4`, `query_pagespeed`, `query_ai`, `query_seo` |
+| 🔎 **Specialised queries** | `query_images`, `query_broken_links`, `list_duplicate_clusters` |
+| 📈 **Reports** | `report_status_code_histogram`, `report_indexability_distribution`, `report_content_kind_distribution`, `report_depth_histogram`, `report_response_time_histogram`, `report_inlinks_histogram`, `report_word_count_histogram`, `report_url_length_histogram`, `report_top_urls_by`, `report_top_anchor_texts`, `report_external_domain_health`, `report_image_weight_per_page`, `report_link_position_breakdown`, `report_pages_per_directory`, `report_word_count_per_directory`, `report_sitemap_orphans`, `report_analytics_coverage`, `report_server_header_breakdown` |
 | 📁 **Project management** | `list_projects`, `set_project`, `current_project` |
-| 🕷 **Crawl control** (desktop must be open) | `start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `get_crawl_progress`, `get_desktop_project` |
+| 🕷 **Crawl control** (desktop must be open) | `start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `clear_crawl`, `get_crawl_progress`, `get_desktop_project` |
 
-`start_crawl` accepts a `startUrl` plus optional whitelisted overrides (scope, maxDepth, maxUrls, maxConcurrency, maxRps, crawlDelayMs, requestTimeoutMs, respectRobotsTxt, followRedirects, crawlExternal, userAgent, include/excludePatterns) — anything you don't override keeps the desktop user's saved value. Crawls launched via MCP go through the **same code path** as the UI's Start button, so progress shows up in the desktop app live as the agent drives it.
+`start_crawl` accepts a `startUrl` plus optional whitelisted overrides (scope, maxDepth, maxUrls, maxConcurrency, maxRps, crawlDelayMs, requestTimeoutMs, respectRobotsTxt, followRedirects, crawlExternal, userAgent, include/excludePatterns) — anything you don't override keeps the desktop user's saved value. Crawls launched via MCP go through the **same code path** as the UI's Start button, so progress shows up in the desktop app live as the agent drives it. Call `clear_crawl` first when you want a fresh BFS after a completed crawl with the same seed URL (otherwise the crawler treats the re-start as a resume and exits because every URL is already in the DB).
 
 **1. Build it once:**
 

apps/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/cli",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "type": "module",
   "bin": {

apps/desktop/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/desktop",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "description": "FreeCrawl SEO — desktop application",
   "homepage": "https://github.com/kemalai/FreeCrawl-SEO-Tool",
@@ -26,6 +26,9 @@
     "typecheck": "npm run typecheck:node && npm run typecheck:web"
   },
   "dependencies": {
+    "@dnd-kit/core": "^6.3.1",
+    "@dnd-kit/sortable": "^10.0.0",
+    "@dnd-kit/utilities": "^3.2.2",
     "@freecrawl/core": "*",
     "@freecrawl/db": "*",
     "@freecrawl/shared-types": "*",

apps/desktop/src/main/index.ts

@@ -53,6 +53,11 @@ import {
   type SitemapValidateResult,
   type UrlRewritePreviewInput,
   type UrlRewritePreviewResult,
+  type CustomExtractionRule,
+  type ExtractionPreviewInput,
+  type ExtractionPreviewResult,
+  type ExtractionRulesExportResult,
+  type ExtractionRulesImportResult,
   type TopUrlsInput,
   type TopUrlsRow,
   type ExternalDomainHealthRow,
@@ -150,7 +155,9 @@ import {
   validateSitemap,
   normalizeUrl,
   compileUrlRegexRewrites,
+  previewExtractionRules,
 } from '@freecrawl/core';
+import { fetch as undiciFetch } from 'undici';
 import { ProjectDb } from '@freecrawl/db';
 import { buildAppMenu } from './menu.js';
 import { isMenuLang, type MenuLang } from './menu-i18n.js';
@@ -233,6 +240,11 @@ let crawlController: {
   stopCrawl: () => void;
   pauseCrawl: () => void;
   resumeCrawl: () => void;
+  /** Wipe URL/links/images/headers/queue tables. Same primitive the
+   *  desktop "Clear" button calls — wired here so the MCP bridge can
+   *  invoke it without duplicating the writer-pool serialisation +
+   *  recovery-checkpoint reset logic. */
+  clearCrawl: () => Promise<void>;
 } | null = null;
 /** In-memory snapshot of the previous session's checkpoint, captured
  * before `db.reset()` wipes it. Cleared once the user accepts or
@@ -1754,6 +1766,233 @@ function registerIpc(): void {
     },
   );
 
+  ipcMain.handle(
+    IPC.extractionPreview,
+    async (_e, input: ExtractionPreviewInput): Promise<ExtractionPreviewResult> => {
+      // Live preview of Custom Extraction rules against a single URL.
+      // Mirrors the production crawler's parse pipeline (cheerio + the
+      // shared `previewExtractionRules` helper) so what the user sees
+      // here matches what would be stored per-URL after a full crawl.
+      // The fetch is one-shot — no retry, no robots.txt, no rate
+      // limiting; the user explicitly asked for this URL so we honour
+      // it directly.
+      const url = (input.url ?? '').trim();
+      if (!url) {
+        return { ok: false, error: 'URL is required' };
+      }
+      let parsed: URL;
+      try {
+        parsed = new URL(url);
+      } catch {
+        return { ok: false, error: 'Invalid URL' };
+      }
+      if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
+        return { ok: false, error: 'Only http:// or https:// URLs are supported' };
+      }
+
+      const userAgent =
+        (input.userAgent ?? '').trim() ||
+        'FreeCrawl SEO Tool (extraction preview)';
+      const acceptLanguage = (input.acceptLanguage ?? '').trim() || 'en';
+
+      const controller = new AbortController();
+      const timeoutMs = 15_000;
+      const timer = setTimeout(() => controller.abort(), timeoutMs);
+      const startedAt = Date.now();
+
+      let response: Response | null = null;
+      try {
+        response = (await undiciFetch(parsed.href, {
+          method: 'GET',
+          redirect: 'follow',
+          headers: {
+            'User-Agent': userAgent,
+            Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+            'Accept-Language': acceptLanguage,
+          },
+          signal: controller.signal,
+        })) as unknown as Response;
+      } catch (err) {
+        clearTimeout(timer);
+        const msg = err instanceof Error ? err.message : String(err);
+        return {
+          ok: false,
+          error: controller.signal.aborted
+            ? `Fetch timed out after ${timeoutMs / 1000}s`
+            : `Fetch failed: ${msg}`,
+        };
+      }
+      clearTimeout(timer);
+
+      const fetchMs = Date.now() - startedAt;
+      const contentType = response.headers.get('content-type') ?? '';
+      const finalUrl = response.url || parsed.href;
+
+      let bodyText = '';
+      try {
+        bodyText = await response.text();
+      } catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        return {
+          ok: false,
+          statusCode: response.status,
+          finalUrl,
+          contentType,
+          fetchMs,
+          error: `Body read failed: ${msg}`,
+        };
+      }
+      const byteSize = Buffer.byteLength(bodyText, 'utf8');
+
+      const results = previewExtractionRules(bodyText, input.rules ?? []).map(
+        (r) => ({
+          name: r.name,
+          value: r.value,
+          ...(r.error !== undefined && { error: r.error }),
+        }),
+      );
+
+      return {
+        ok: true,
+        statusCode: response.status,
+        finalUrl,
+        contentType,
+        byteSize,
+        fetchMs,
+        results,
+      };
+    },
+  );
+
+  ipcMain.handle(
+    IPC.extractionRulesExport,
+    async (
+      _e,
+      rules: CustomExtractionRule[],
+    ): Promise<ExtractionRulesExportResult> => {
+      if (!mainWindow) return { filePath: '', bytesWritten: 0 };
+      const res = await dialog.showSaveDialog(mainWindow, {
+        title: 'Export Extraction Rules',
+        defaultPath: 'freecrawl-extraction-rules.json',
+        filters: [{ name: 'JSON', extensions: ['json'] }],
+      });
+      if (res.canceled || !res.filePath) {
+        return { filePath: '', bytesWritten: 0 };
+      }
+      // Versioned envelope — same pattern as prefsExportSettings so
+      // future imports can detect schema drift without breaking on the
+      // raw rule blob.
+      const payload = {
+        format: 'freecrawl/extraction-rules',
+        version: 1,
+        exportedAt: new Date().toISOString(),
+        rules,
+      };
+      const json = JSON.stringify(payload, null, 2);
+      writeFileSync(res.filePath, json, 'utf8');
+      return {
+        filePath: res.filePath,
+        bytesWritten: Buffer.byteLength(json, 'utf8'),
+      };
+    },
+  );
+
+  ipcMain.handle(
+    IPC.extractionRulesImport,
+    async (): Promise<ExtractionRulesImportResult> => {
+      if (!mainWindow) {
+        return { filePath: '', rules: [], skippedCount: 0 };
+      }
+      const res = await dialog.showOpenDialog(mainWindow, {
+        title: 'Import Extraction Rules',
+        properties: ['openFile'],
+        filters: [{ name: 'JSON', extensions: ['json'] }],
+      });
+      if (res.canceled || res.filePaths.length === 0) {
+        return { filePath: '', rules: [], skippedCount: 0 };
+      }
+      const filePath = res.filePaths[0]!;
+      let raw: unknown;
+      try {
+        const text = readFileSync(filePath, 'utf8');
+        raw = JSON.parse(text);
+      } catch (err) {
+        return {
+          filePath: '',
+          rules: [],
+          skippedCount: 0,
+          error: `Cannot parse JSON: ${(err as Error).message}`,
+        };
+      }
+      // Accept either the wrapped envelope { rules: [...] } or a bare
+      // array — bare arrays let users share rule snippets without the
+      // boilerplate.
+      const candidates: unknown[] = Array.isArray(raw)
+        ? raw
+        : raw &&
+            typeof raw === 'object' &&
+            Array.isArray((raw as { rules?: unknown }).rules)
+          ? ((raw as { rules: unknown[] }).rules)
+          : [];
+      if (candidates.length === 0) {
+        return {
+          filePath: '',
+          rules: [],
+          skippedCount: 0,
+          error: 'File does not contain a rules array.',
+        };
+      }
+      const validOutputs = new Set([
+        'text',
+        'attribute',
+        'inner_html',
+        'outer_html',
+        'count',
+        'regex_group',
+      ]);
+      const validMulti = new Set(['first', 'last', 'all', 'concat', 'count']);
+      const accepted: CustomExtractionRule[] = [];
+      let skippedCount = 0;
+      for (const item of candidates) {
+        if (!item || typeof item !== 'object') {
+          skippedCount++;
+          continue;
+        }
+        const o = item as Record<string, unknown>;
+        if (
+          typeof o['name'] !== 'string' ||
+          typeof o['selector'] !== 'string' ||
+          (o['type'] !== 'css' && o['type'] !== 'regex') ||
+          typeof o['output'] !== 'string' ||
+          !validOutputs.has(o['output'] as string) ||
+          typeof o['multi'] !== 'string' ||
+          !validMulti.has(o['multi'] as string)
+        ) {
+          skippedCount++;
+          continue;
+        }
+        // Hard cap at 10 to match the in-app limit; everything past the
+        // cap is reported as skipped so the user knows truncation happened.
+        if (accepted.length >= 10) {
+          skippedCount++;
+          continue;
+        }
+        const rule: CustomExtractionRule = {
+          name: o['name'] as string,
+          type: o['type'] as 'css' | 'regex',
+          selector: o['selector'] as string,
+          output: o['output'] as CustomExtractionRule['output'],
+          multi: o['multi'] as CustomExtractionRule['multi'],
+        };
+        if (typeof o['attribute'] === 'string') {
+          rule.attribute = o['attribute'];
+        }
+        accepted.push(rule);
+      }
+      return { filePath, rules: accepted, skippedCount };
+    },
+  );
+
   ipcMain.handle(
     IPC.sitemapValidate,
     async (_e, input: SitemapValidateInput): Promise<SitemapValidateResult> => {
@@ -3359,9 +3598,17 @@ function registerIpc(): void {
     stopCrawl: () => activeCrawler?.stop(),
     pauseCrawl: () => activeCrawler?.pause(),
     resumeCrawl: () => activeCrawler?.resume(),
+    clearCrawl: () => clearCrawlDb(),
   };
 
-  ipcMain.handle(IPC.crawlClear, async () => {
+  // Shared between the renderer's IPC `crawl:clear` and the MCP bridge's
+  // `POST /v1/crawl/clear` — keeps the two surfaces from drifting (e.g.
+  // remembering to drop `pendingRecoveryCheckpoint` and fire
+  // `dataChanged` in both places). Without this helper an MCP-driven
+  // re-start after a completed crawl is a no-op because the crawler's
+  // resume-vs-fresh decision keeps existing rows when the start URL
+  // matches the previous one.
+  async function clearCrawlDb(): Promise<void> {
     const t0 = Date.now();
     activeCrawler?.stop();
     activeCrawler = null;
@@ -3408,7 +3655,9 @@ function registerIpc(): void {
     // stale rows until the next refresh tick.
     fireDataChanged();
     logger.log('info', 'main', `Clear completed in ${Date.now() - t0} ms`);
-  });
+  }
+
+  ipcMain.handle(IPC.crawlClear, () => clearCrawlDb());
 
   ipcMain.handle(IPC.crawlAddUrl, (_e, url: string): { accepted: boolean } => {
     if (!activeCrawler) return { accepted: false };
@@ -3933,6 +4182,16 @@ function registerIpc(): void {
     getDb().urlClusterMembers(urlId),
   );
 
+  ipcMain.handle(
+    IPC.duplicateClustersList,
+    (_e, input: { offset: number; limit: number }) =>
+      getDb().listDuplicateClusters(input.offset, input.limit),
+  );
+
+  ipcMain.handle(IPC.duplicateClustersCount, () =>
+    getDb().countDuplicateClusterMembers(),
+  );
+
   ipcMain.handle(
     IPC.urlCertInfo,
     (_e, input: UrlCertInfoInput): UrlCertInfoResult => {
@@ -4866,6 +5125,9 @@ void app.whenReady().then(async () => {
       stopCrawl: () => crawlController?.stopCrawl(),
       pauseCrawl: () => crawlController?.pauseCrawl(),
       resumeCrawl: () => crawlController?.resumeCrawl(),
+      clearCrawl: async () => {
+        if (crawlController) await crawlController.clearCrawl();
+      },
       getProgress: () => latestProgress,
       getProjectPath: () => currentProjectPath || null,
       getUrlCount: () => {

apps/desktop/src/main/mcp-bridge.ts

@@ -32,6 +32,7 @@
  *   POST /v1/crawl/stop
  *   POST /v1/crawl/pause
  *   POST /v1/crawl/resume
+ *   POST /v1/crawl/clear
  *   GET  /v1/crawl/progress
  *   GET  /v1/project        — { projectPath, urlsCrawled }
  *
@@ -62,6 +63,15 @@ export interface McpBridgeDeps {
   stopCrawl: () => void;
   pauseCrawl: () => void;
   resumeCrawl: () => void;
+  /**
+   * Wipe the URLs table (and all dependent rows — links / images /
+   * headers / cookies / sitemap_urls / crawl_queue). Same primitive
+   * the desktop's "Clear" button calls. Required because the crawler's
+   * resume-vs-fresh decision is "same start URL → keep existing rows"
+   * — without an explicit clear, an MCP-driven re-start after a
+   * completed crawl finds zero pending work and exits immediately.
+   */
+  clearCrawl: () => Promise<void>;
   /**
    * Snapshot of the latest progress event the Crawler emitted. Null
    * when no crawl has ever run in this session. The bridge passes it
@@ -362,6 +372,17 @@ async function handleRequest(
     send(res, 200, { ok: true });
     return;
   }
+  if (method === 'POST' && path === '/v1/crawl/clear') {
+    try {
+      await deps.clearCrawl();
+      send(res, 200, { ok: true });
+    } catch (err) {
+      send(res, 500, {
+        error: err instanceof Error ? err.message : 'clear-failed',
+      });
+    }
+    return;
+  }
 
   send(res, 404, { error: 'not-found' });
 }

apps/desktop/src/preload/index.ts

@@ -42,7 +42,14 @@ import {
   type SitemapValidateResult,
   type UrlRewritePreviewInput,
   type UrlRewritePreviewResult,
+  type CustomExtractionRule,
+  type ExtractionPreviewInput,
+  type ExtractionPreviewResult,
+  type ExtractionRulesExportResult,
+  type ExtractionRulesImportResult,
   type UrlClusterMember,
+  type DuplicateClusterRow,
+  type DuplicateClustersListInput,
   type PagesPerDirectoryInput,
   type PagesPerDirectoryRow,
   type StatusCodeHistogramRow,
@@ -243,8 +250,24 @@ const api: FreeCrawlApi = {
   urlRewritePreview: (
     input: UrlRewritePreviewInput,
   ): Promise<UrlRewritePreviewResult> => ipcRenderer.invoke(IPC.urlRewritePreview, input),
+  extractionPreview: (
+    input: ExtractionPreviewInput,
+  ): Promise<ExtractionPreviewResult> =>
+    ipcRenderer.invoke(IPC.extractionPreview, input),
+  extractionRulesExport: (
+    rules: CustomExtractionRule[],
+  ): Promise<ExtractionRulesExportResult> =>
+    ipcRenderer.invoke(IPC.extractionRulesExport, rules),
+  extractionRulesImport: (): Promise<ExtractionRulesImportResult> =>
+    ipcRenderer.invoke(IPC.extractionRulesImport),
   urlClusterMembers: (urlId: number): Promise<UrlClusterMember[]> =>
     ipcRenderer.invoke(IPC.urlClusterMembers, urlId),
+  duplicateClustersList: (
+    input: DuplicateClustersListInput,
+  ): Promise<DuplicateClusterRow[]> =>
+    ipcRenderer.invoke(IPC.duplicateClustersList, input),
+  duplicateClustersCount: (): Promise<number> =>
+    ipcRenderer.invoke(IPC.duplicateClustersCount),
   reportsPagesPerDirectory: (
     input: PagesPerDirectoryInput,
   ): Promise<PagesPerDirectoryRow[]> => ipcRenderer.invoke(IPC.reportsPagesPerDirectory, input),

apps/desktop/src/renderer/src/components/BottomDetailPanel.tsx

@@ -1593,6 +1593,26 @@ function NameValueView({ row }: { row: CrawlUrlRow }) {
     ['Hreflang Count', row.hreflangCount > 0 ? row.hreflangCount : null],
     ['Hreflangs', summarizeHreflangs(row.hreflangs)],
     ['AMP HTML', row.amphtml],
+    ['AMP Page', row.ampPage ? 'Yes' : null],
+    [
+      'AMP Validation Errors',
+      // The DB stores a JSON array; show "OK" when the column is empty
+      // and the page IS AMP (clean validation), otherwise pretty-print
+      // the comma-joined error codes so users see exactly which checks
+      // failed without scrolling through a JSON literal.
+      row.ampPage
+        ? (() => {
+            if (!row.ampValidationErrors) return 'OK';
+            try {
+              const parsed = JSON.parse(row.ampValidationErrors) as string[];
+              if (!Array.isArray(parsed) || parsed.length === 0) return 'OK';
+              return parsed.join(', ');
+            } catch {
+              return row.ampValidationErrors;
+            }
+          })()
+        : null,
+    ],
     ['Favicon', row.favicon],
     ['Apple Touch Icon', row.appleTouchIcon],
     ['Android / PWA Icon', row.androidIcon],

apps/desktop/src/renderer/src/components/DuplicatesGroupedView.tsx

@@ -0,0 +1,382 @@
+import { useEffect, useMemo, useRef, useState } from 'react';
+import { useVirtualizer } from '@tanstack/react-virtual';
+import { ChevronDown, ChevronRight, Layers, Loader2 } from 'lucide-react';
+import clsx from 'clsx';
+import { useTranslation } from 'react-i18next';
+import type { DuplicateClusterRow } from '@freecrawl/shared-types';
+import { useAppStore } from '../store.js';
+
+const HEADER_ROW_HEIGHT = 32;
+const MEMBER_ROW_HEIGHT = 24;
+const PAGE_SIZE = 5000;
+/** Soft upper-bound on rows fetched in a single load. A cluster member
+ *  row is < 200 B in renderer-side memory, so 50k keeps the worst-case
+ *  payload around ~10 MB and the scroll grid responsive. Crawls that
+ *  exceed this cap will lose the long tail of the smallest clusters —
+ *  the largest, most actionable ones (sorted DESC by cluster size at
+ *  the DB layer) are always retained. */
+const MAX_ROWS = 50_000;
+
+/** One row in the virtualized list. Cluster headers sit above their
+ *  members; collapsing a header hides its members but keeps the header
+ *  visible so the user can see how many clusters exist at a glance. */
+type Item =
+  | {
+      kind: 'header';
+      clusterId: number;
+      clusterSize: number;
+      memberCount: number;
+      repUrl: string;
+      maxHamming: number;
+    }
+  | { kind: 'member'; clusterId: number; row: DuplicateClusterRow };
+
+export function DuplicatesGroupedView() {
+  const { t } = useTranslation();
+  const selectedUrlId = useAppStore((s) => s.selectedUrlId);
+  const setSelectedUrlId = useAppStore((s) => s.setSelectedUrlId);
+  const setSelectedUrlIds = useAppStore((s) => s.setSelectedUrlIds);
+  const dataVersion = useAppStore((s) => s.dataVersion);
+
+  const [rows, setRows] = useState<DuplicateClusterRow[]>([]);
+  const [total, setTotal] = useState(0);
+  const [loading, setLoading] = useState(true);
+  const [collapsed, setCollapsed] = useState<Set<number>>(new Set());
+  const scrollRef = useRef<HTMLDivElement>(null);
+
+  useEffect(() => {
+    let cancelled = false;
+    setLoading(true);
+
+    void (async () => {
+      try {
+        const count = await window.freecrawl.duplicateClustersCount();
+        if (cancelled) return;
+        setTotal(count);
+
+        const cap = Math.min(count, MAX_ROWS);
+        const all: DuplicateClusterRow[] = [];
+        for (let off = 0; off < cap; off += PAGE_SIZE) {
+          const page = await window.freecrawl.duplicateClustersList({
+            offset: off,
+            limit: Math.min(PAGE_SIZE, cap - off),
+          });
+          if (cancelled) return;
+          all.push(...page);
+          if (page.length < PAGE_SIZE) break;
+        }
+        if (!cancelled) {
+          setRows(all);
+          setLoading(false);
+        }
+      } catch {
+        if (!cancelled) {
+          setRows([]);
+          setLoading(false);
+        }
+      }
+    })();
+
+    return () => {
+      cancelled = true;
+    };
+  }, [dataVersion]);
+
+  /** Flatten paginated cluster rows into a [header, ...members]+ stream.
+   *  Rows are pre-sorted by the DB query (cluster_size DESC, cluster_id
+   *  ASC, url ASC), so a single pass detects cluster boundaries.
+   *  Collapsed clusters keep their header and drop their members.
+   *  Header carries derived stats — representative URL (alphabetically
+   *  first) and the max Hamming distance present in the cluster, which
+   *  is a quick "how tight is this cluster" signal. */
+  const items = useMemo<Item[]>(() => {
+    const list: Item[] = [];
+    let i = 0;
+    while (i < rows.length) {
+      const head = rows[i]!;
+      const cid = head.clusterId;
+      let j = i;
+      let repUrl = head.url;
+      let maxHamming = 0;
+      while (j < rows.length && rows[j]!.clusterId === cid) {
+        const m = rows[j]!;
+        if (m.url < repUrl) repUrl = m.url;
+        if (m.hammingFromRep > maxHamming) maxHamming = m.hammingFromRep;
+        j++;
+      }
+      list.push({
+        kind: 'header',
+        clusterId: cid,
+        clusterSize: head.clusterSize,
+        memberCount: j - i,
+        repUrl,
+        maxHamming,
+      });
+      if (!collapsed.has(cid)) {
+        for (let k = i; k < j; k++) {
+          list.push({ kind: 'member', clusterId: cid, row: rows[k]! });
+        }
+      }
+      i = j;
+    }
+    return list;
+  }, [rows, collapsed]);
+
+  /** Header rows are taller than members. Estimate accordingly so
+   *  virtual scrolling reserves space correctly without measuring. */
+  const estimateSize = (idx: number): number => {
+    const it = items[idx];
+    if (!it) return MEMBER_ROW_HEIGHT;
+    return it.kind === 'header' ? HEADER_ROW_HEIGHT : MEMBER_ROW_HEIGHT;
+  };
+
+  const virtualizer = useVirtualizer({
+    count: items.length,
+    getScrollElement: () => scrollRef.current,
+    estimateSize,
+    overscan: 25,
+    getItemKey: (idx) => {
+      const it = items[idx];
+      if (!it) return `idx-${idx}`;
+      return it.kind === 'header'
+        ? `h-${it.clusterId}`
+        : `m-${it.row.urlId}`;
+    },
+  });
+
+  const clusterCount = items.reduce((n, it) => (it.kind === 'header' ? n + 1 : n), 0);
+
+  const toggleCluster = (cid: number) => {
+    setCollapsed((prev) => {
+      const next = new Set(prev);
+      if (next.has(cid)) next.delete(cid);
+      else next.add(cid);
+      return next;
+    });
+  };
+
+  const expandAll = () => setCollapsed(new Set());
+  const collapseAll = () => {
+    const next = new Set<number>();
+    for (const it of items) if (it.kind === 'header') next.add(it.clusterId);
+    setCollapsed(next);
+  };
+
+  const handleMemberClick = (row: DuplicateClusterRow) => {
+    setSelectedUrlId(row.urlId);
+    setSelectedUrlIds([row.urlId]);
+  };
+
+  if (loading && rows.length === 0) {
+    return (
+      <div className="flex h-full flex-1 items-center justify-center text-xs text-surface-500">
+        <Loader2 className="mr-2 h-4 w-4 animate-spin" />
+        {t('duplicates.loadingClusters', { defaultValue: 'Loading near-duplicate clusters…' })}
+      </div>
+    );
+  }
+
+  if (!loading && rows.length === 0) {
+    return (
+      <div className="flex h-full flex-1 flex-col items-center justify-center gap-2 text-center text-xs text-surface-500">
+        <Layers className="h-6 w-6 text-surface-700" />
+        <div className="max-w-md">
+          <div className="mb-1 text-sm font-semibold text-surface-300">
+            {t('duplicates.noClustersTitle', { defaultValue: 'No near-duplicate clusters' })}
+          </div>
+          <div>
+            {t('duplicates.noClustersHint', {
+              defaultValue:
+                'Run a crawl and let post-crawl analysis finish — clusters appear once two or more pages share enough content to fall under the SimHash Hamming threshold (Settings → Duplicates).',
+            })}
+          </div>
+        </div>
+      </div>
+    );
+  }
+
+  const virtualItems = virtualizer.getVirtualItems();
+
+  return (
+    <div className="flex h-full flex-1 flex-col">
+      <div className="flex shrink-0 items-center gap-3 border-b border-surface-800 bg-surface-900/40 px-3 py-1.5 text-[11px] text-surface-400">
+        <span className="inline-flex items-center gap-1.5 text-surface-300">
+          <Layers className="h-3.5 w-3.5" />
+          {t('duplicates.statusClusters', {
+            defaultValue: '{{clusters}} cluster · {{members}} pages',
+            clusters: clusterCount.toLocaleString(),
+            members: rows.length.toLocaleString(),
+          })}
+        </span>
+        {total > rows.length && (
+          <span className="text-amber-400">
+            {t('duplicates.truncated', {
+              defaultValue: 'truncated to first {{cap}} rows',
+              cap: MAX_ROWS.toLocaleString(),
+            })}
+          </span>
+        )}
+        <div className="ml-auto flex items-center gap-1">
+          <button
+            type="button"
+            onClick={expandAll}
+            className="rounded border border-surface-700 px-1.5 py-0.5 text-[10px] text-surface-300 hover:bg-surface-800"
+          >
+            {t('urlsTab.expandAll', { defaultValue: 'Expand all' })}
+          </button>
+          <button
+            type="button"
+            onClick={collapseAll}
+            className="rounded border border-surface-700 px-1.5 py-0.5 text-[10px] text-surface-300 hover:bg-surface-800"
+          >
+            {t('urlsTab.collapseAll', { defaultValue: 'Collapse all' })}
+          </button>
+        </div>
+      </div>
+      <div ref={scrollRef} className="relative flex-1 select-none overflow-auto">
+        <div
+          className="relative"
+          style={{ height: virtualizer.getTotalSize(), minWidth: '100%' }}
+        >
+          {virtualItems.map((vi) => {
+            const it = items[vi.index];
+            if (!it) return null;
+            if (it.kind === 'header') {
+              const isCollapsed = collapsed.has(it.clusterId);
+              return (
+                <div
+                  key={vi.key}
+                  data-index={vi.index}
+                  className={clsx(
+                    'absolute left-0 top-0 flex w-full cursor-pointer items-center gap-2 border-b border-surface-800 bg-surface-900/80 px-3 text-[11px] font-semibold text-surface-100 hover:bg-surface-800/80',
+                  )}
+                  style={{
+                    transform: `translateY(${vi.start}px)`,
+                    height: HEADER_ROW_HEIGHT,
+                  }}
+                  onClick={() => toggleCluster(it.clusterId)}
+                  title={t('duplicates.toggleClusterTitle', {
+                    defaultValue: 'Click to expand/collapse this cluster',
+                  })}
+                >
+                  {isCollapsed ? (
+                    <ChevronRight className="h-3.5 w-3.5 shrink-0 text-surface-400" />
+                  ) : (
+                    <ChevronDown className="h-3.5 w-3.5 shrink-0 text-surface-400" />
+                  )}
+                  <span className="font-mono tabular-nums text-accent-300">
+                    #{it.clusterId}
+                  </span>
+                  <span className="text-surface-400">·</span>
+                  <span className="text-surface-200">
+                    {t('duplicates.headerSize', {
+                      defaultValue: '{{n}} pages',
+                      n: it.memberCount,
+                    })}
+                  </span>
+                  <span className="text-surface-400">·</span>
+                  <span
+                    className="truncate text-surface-300"
+                    title={it.repUrl}
+                  >
+                    {it.repUrl}
+                  </span>
+                  <span className="ml-auto shrink-0 rounded bg-surface-800 px-1.5 py-0.5 font-mono text-[10px] text-surface-300">
+                    {t('duplicates.maxHamming', {
+                      defaultValue: 'max Δ {{n}} bits',
+                      n: it.maxHamming,
+                    })}
+                  </span>
+                </div>
+              );
+            }
+            const row = it.row;
+            const isSelected = selectedUrlId === row.urlId;
+            return (
+              <div
+                key={vi.key}
+                data-index={vi.index}
+                className={clsx(
+                  'absolute left-0 top-0 flex w-full cursor-pointer items-center gap-2 border-b border-surface-900 pl-8 pr-3 text-[11px]',
+                  isSelected
+                    ? 'bg-accent-500/20 text-surface-50'
+                    : 'text-surface-300 hover:bg-surface-900/40',
+                )}
+                style={{
+                  transform: `translateY(${vi.start}px)`,
+                  height: MEMBER_ROW_HEIGHT,
+                }}
+                onClick={() => handleMemberClick(row)}
+                onContextMenu={(e) => {
+                  e.preventDefault();
+                  void window.freecrawl.urlContextMenu({
+                    url: row.url,
+                    urlId: row.urlId,
+                  });
+                }}
+                title={row.url}
+              >
+                <span
+                  className={clsx(
+                    'inline-block w-12 shrink-0 rounded px-1 text-center font-mono text-[10px]',
+                    statusClasses(row.statusCode),
+                  )}
+                >
+                  {row.statusCode ?? '—'}
+                </span>
+                <span
+                  className={clsx(
+                    'inline-block w-20 shrink-0 truncate',
+                    row.indexability === 'indexable'
+                      ? 'text-emerald-400'
+                      : 'text-amber-400',
+                  )}
+                  title={row.indexability}
+                >
+                  {row.indexability === 'indexable' ? 'Indexable' : 'Non-Indexable'}
+                </span>
+                <span className="flex-1 truncate font-mono text-surface-200">
+                  {row.url}
+                </span>
+                <span className="w-44 shrink-0 truncate text-surface-400" title={row.title ?? ''}>
+                  {row.title ?? <span className="text-surface-700">—</span>}
+                </span>
+                <span className="w-14 shrink-0 text-right font-mono tabular-nums text-surface-300">
+                  {row.wordCount === null ? '—' : row.wordCount.toLocaleString()}
+                </span>
+                <span className="w-14 shrink-0 text-right font-mono tabular-nums text-surface-300">
+                  {row.inlinks.toLocaleString()}
+                </span>
+                <span
+                  className={clsx(
+                    'w-14 shrink-0 text-right font-mono tabular-nums',
+                    row.hammingFromRep === 0
+                      ? 'text-emerald-400'
+                      : row.hammingFromRep <= 3
+                        ? 'text-amber-300'
+                        : 'text-orange-300',
+                  )}
+                  title={t('duplicates.hammingFromRepTitle', {
+                    defaultValue:
+                      'SimHash Hamming distance from the cluster representative — 0 means the rep itself, higher = further from the rep',
+                  })}
+                >
+                  Δ{row.hammingFromRep}
+                </span>
+              </div>
+            );
+          })}
+        </div>
+      </div>
+    </div>
+  );
+}
+
+function statusClasses(code: number | null): string {
+  if (code === null) return 'bg-surface-800 text-surface-400';
+  if (code >= 200 && code < 300) return 'bg-emerald-900/60 text-emerald-300';
+  if (code >= 300 && code < 400) return 'bg-amber-900/60 text-amber-300';
+  if (code >= 400 && code < 500) return 'bg-orange-900/60 text-orange-300';
+  if (code >= 500) return 'bg-red-900/60 text-red-300';
+  return 'bg-surface-800 text-surface-400';
+}

apps/desktop/src/renderer/src/components/ExtractionPreviewDialog.tsx

@@ -0,0 +1,331 @@
+import { useEffect, useState } from 'react';
+import { useTranslation } from 'react-i18next';
+import clsx from 'clsx';
+import { AlertCircle, Loader2, Play, X } from 'lucide-react';
+import type {
+  CustomExtractionRule,
+  ExtractionPreviewResult,
+} from '@freecrawl/shared-types';
+
+interface Props {
+  open: boolean;
+  onClose: () => void;
+  /** Current rules from the Custom Extraction panel — preview runs
+   *  them as-is, even if the user hasn't hit Save yet. That's the
+   *  whole point: validate before saving. */
+  rules: CustomExtractionRule[];
+  /** Optional default URL — typically the start URL from the in-form
+   *  crawl config so the user doesn't have to type it. */
+  defaultUrl?: string;
+  userAgent?: string;
+  acceptLanguage?: string;
+}
+
+export function ExtractionPreviewDialog({
+  open,
+  onClose,
+  rules,
+  defaultUrl,
+  userAgent,
+  acceptLanguage,
+}: Props) {
+  const { t } = useTranslation();
+  const [url, setUrl] = useState(defaultUrl ?? '');
+  const [running, setRunning] = useState(false);
+  const [result, setResult] = useState<ExtractionPreviewResult | null>(null);
+
+  useEffect(() => {
+    if (open) {
+      setUrl(defaultUrl ?? '');
+      setResult(null);
+    }
+  }, [open, defaultUrl]);
+
+  useEffect(() => {
+    if (!open) return;
+    const onKey = (e: KeyboardEvent) => {
+      if (e.key === 'Escape') onClose();
+    };
+    window.addEventListener('keydown', onKey);
+    return () => window.removeEventListener('keydown', onKey);
+  }, [open, onClose]);
+
+  if (!open) return null;
+
+  const runPreview = async () => {
+    if (running) return;
+    if (!url.trim()) return;
+    setRunning(true);
+    setResult(null);
+    try {
+      const r = await window.freecrawl.extractionPreview({
+        url: url.trim(),
+        rules,
+        userAgent,
+        acceptLanguage,
+      });
+      setResult(r);
+    } catch (e) {
+      setResult({
+        ok: false,
+        error: e instanceof Error ? e.message : String(e),
+      });
+    } finally {
+      setRunning(false);
+    }
+  };
+
+  const ruleCount = rules.filter((r) => r.name.trim() && r.selector.trim())
+    .length;
+
+  return (
+    <div
+      className="fixed inset-0 z-[120] flex items-center justify-center bg-black/60 p-4"
+      onClick={(e) => {
+        if (e.target === e.currentTarget) onClose();
+      }}
+    >
+      <div className="flex max-h-[90vh] w-full max-w-4xl flex-col overflow-hidden rounded-lg border border-surface-700 bg-surface-900 shadow-2xl">
+        {/* Header */}
+        <div className="flex shrink-0 items-center justify-between border-b border-surface-800 px-4 py-3">
+          <div>
+            <div className="text-sm font-semibold text-surface-100">
+              {t('extractionPreview.title', {
+                defaultValue: 'Custom Extraction Preview',
+              })}
+            </div>
+            <div className="text-[11px] text-surface-500">
+              {t('extractionPreview.subtitle', {
+                defaultValue:
+                  'Run all configured rules against a single URL to validate selectors / regex before crawling.',
+              })}
+            </div>
+          </div>
+          <button
+            type="button"
+            onClick={onClose}
+            className="rounded p-1 text-surface-400 hover:bg-surface-800 hover:text-surface-200"
+            title={t('common.close', { defaultValue: 'Close' })}
+          >
+            <X className="h-4 w-4" />
+          </button>
+        </div>
+
+        {/* URL input + run */}
+        <div className="flex shrink-0 items-center gap-2 border-b border-surface-800 bg-surface-900/40 px-4 py-3">
+          <input
+            type="text"
+            className="flex-1 rounded border border-surface-700 bg-surface-950 px-2 py-1.5 font-mono text-[12px] text-surface-100 focus:border-blue-500 focus:outline-none"
+            placeholder="https://example.com/page"
+            value={url}
+            onChange={(e) => setUrl(e.target.value)}
+            onKeyDown={(e) => {
+              if (e.key === 'Enter') void runPreview();
+            }}
+            spellCheck={false}
+            autoFocus
+          />
+          <button
+            type="button"
+            onClick={runPreview}
+            disabled={running || !url.trim() || ruleCount === 0}
+            className={clsx(
+              'inline-flex items-center gap-1.5 rounded border px-3 py-1.5 text-[12px] transition',
+              running || !url.trim() || ruleCount === 0
+                ? 'cursor-not-allowed border-surface-800 text-surface-600'
+                : 'border-accent-500 bg-accent-500/15 text-accent-300 hover:bg-accent-500/25',
+            )}
+            title={
+              ruleCount === 0
+                ? t('extractionPreview.noRules', {
+                    defaultValue:
+                      'Add at least one rule (with a name and selector) before previewing.',
+                  })
+                : t('extractionPreview.runTitle', {
+                    defaultValue: 'Fetch the URL and run every rule against the response.',
+                  })
+            }
+          >
+            {running ? (
+              <Loader2 className="h-3.5 w-3.5 animate-spin" />
+            ) : (
+              <Play className="h-3.5 w-3.5" />
+            )}
+            <span>
+              {running
+                ? t('extractionPreview.running', { defaultValue: 'Running…' })
+                : t('extractionPreview.run', {
+                    defaultValue: 'Run preview ({{n}} rule(s))',
+                    n: ruleCount,
+                  })}
+            </span>
+          </button>
+        </div>
+
+        {/* Results */}
+        <div className="min-h-0 flex-1 overflow-y-auto">
+          {!result && !running && (
+            <div className="flex h-48 items-center justify-center text-center text-[12px] text-surface-500">
+              {t('extractionPreview.idle', {
+                defaultValue:
+                  'Enter a URL above and click Run preview to fetch and apply every rule.',
+              })}
+            </div>
+          )}
+          {running && (
+            <div className="flex h-48 flex-col items-center justify-center gap-2 text-[12px] text-surface-400">
+              <Loader2 className="h-5 w-5 animate-spin text-accent-300" />
+              <span>
+                {t('extractionPreview.fetching', {
+                  defaultValue: 'Fetching URL and parsing response…',
+                })}
+              </span>
+            </div>
+          )}
+          {result && !result.ok && (
+            <div className="flex items-start gap-2 px-4 py-3 text-[12px] text-red-300">
+              <AlertCircle className="mt-0.5 h-4 w-4 shrink-0" />
+              <div>
+                <div className="font-semibold">
+                  {t('extractionPreview.failed', { defaultValue: 'Preview failed' })}
+                </div>
+                <div className="mt-1 text-surface-300">{result.error}</div>
+                {result.statusCode !== undefined && (
+                  <div className="mt-1 text-[11px] text-surface-500">
+                    HTTP {result.statusCode} · {result.contentType || '(no content-type)'} · {result.fetchMs} ms
+                  </div>
+                )}
+              </div>
+            </div>
+          )}
+          {result && result.ok && (
+            <>
+              {/* Status strip */}
+              <div className="flex flex-wrap items-center gap-3 border-b border-surface-800 bg-surface-950/40 px-4 py-2 text-[11px] text-surface-400">
+                <span
+                  className={clsx(
+                    'inline-block rounded px-1.5 font-mono text-[10px]',
+                    statusClasses(result.statusCode),
+                  )}
+                >
+                  {result.statusCode ?? '—'}
+                </span>
+                {result.contentType && (
+                  <span className="font-mono">{result.contentType}</span>
+                )}
+                {result.byteSize !== undefined && (
+                  <span>
+                    {(result.byteSize / 1024).toFixed(1)} KB
+                  </span>
+                )}
+                {result.fetchMs !== undefined && (
+                  <span>{result.fetchMs} ms</span>
+                )}
+                {result.finalUrl && result.finalUrl !== url && (
+                  <span className="truncate font-mono text-surface-500" title={result.finalUrl}>
+                    → {result.finalUrl}
+                  </span>
+                )}
+              </div>
+
+              {/* Per-rule results */}
+              {result.results && result.results.length > 0 ? (
+                <table className="w-full table-fixed text-[12px]">
+                  <thead>
+                    <tr className="border-b border-surface-800 bg-surface-900 text-left text-[10px] uppercase tracking-wider text-surface-500">
+                      <th className="w-1/4 px-4 py-2 font-medium">
+                        {t('extractionPreview.colName', { defaultValue: 'Rule' })}
+                      </th>
+                      <th className="px-4 py-2 font-medium">
+                        {t('extractionPreview.colValue', { defaultValue: 'Value' })}
+                      </th>
+                    </tr>
+                  </thead>
+                  <tbody>
+                    {result.results.map((r, i) => (
+                      <tr
+                        key={`${r.name}-${i}`}
+                        className="border-b border-surface-900 align-top hover:bg-surface-900/40"
+                      >
+                        <td className="break-all px-4 py-2 font-mono text-surface-200">
+                          {r.name}
+                        </td>
+                        <td className="px-4 py-2">
+                          <RuleValueCell value={r.value} error={r.error} />
+                        </td>
+                      </tr>
+                    ))}
+                  </tbody>
+                </table>
+              ) : (
+                <div className="flex h-32 items-center justify-center text-[12px] text-surface-500">
+                  {t('extractionPreview.noResults', {
+                    defaultValue: 'No results — add rules in the Custom Extraction panel.',
+                  })}
+                </div>
+              )}
+            </>
+          )}
+        </div>
+      </div>
+    </div>
+  );
+}
+
+function RuleValueCell({ value, error }: { value: unknown; error?: string }) {
+  const { t } = useTranslation();
+  if (error) {
+    return (
+      <div className="flex items-start gap-1.5 text-red-300">
+        <AlertCircle className="mt-0.5 h-3.5 w-3.5 shrink-0" />
+        <span className="font-mono text-[11px]">{error}</span>
+      </div>
+    );
+  }
+  if (value === null || value === undefined) {
+    return (
+      <span className="italic text-surface-600">
+        {t('extractionPreview.noMatch', { defaultValue: 'no match' })}
+      </span>
+    );
+  }
+  if (Array.isArray(value)) {
+    if (value.length === 0) {
+      return (
+        <span className="italic text-surface-600">
+          {t('extractionPreview.noMatch', { defaultValue: 'no match' })}
+        </span>
+      );
+    }
+    return (
+      <ol className="list-decimal space-y-0.5 pl-5 font-mono text-[11px] text-surface-100">
+        {value.map((v, i) => (
+          <li key={i} className="break-all">
+            {String(v)}
+          </li>
+        ))}
+      </ol>
+    );
+  }
+  if (typeof value === 'number') {
+    return (
+      <span className="font-mono tabular-nums text-accent-300">
+        {value.toLocaleString()}
+      </span>
+    );
+  }
+  return (
+    <span className="block break-all font-mono text-surface-100">
+      {String(value)}
+    </span>
+  );
+}
+
+function statusClasses(code: number | undefined): string {
+  if (code === undefined) return 'bg-surface-800 text-surface-400';
+  if (code >= 200 && code < 300) return 'bg-emerald-900/60 text-emerald-300';
+  if (code >= 300 && code < 400) return 'bg-amber-900/60 text-amber-300';
+  if (code >= 400 && code < 500) return 'bg-orange-900/60 text-orange-300';
+  if (code >= 500) return 'bg-red-900/60 text-red-300';
+  return 'bg-surface-800 text-surface-400';
+}

apps/desktop/src/renderer/src/components/OverviewSidebar.tsx

@@ -743,6 +743,12 @@ function buildTree(o: OverviewCounts | null): Node[] {
               count: o.issues.charsetMissing,
               category: 'issues:charset-missing',
             },
+            {
+              key: 'issues-amp-validation',
+              label: 'AMP Validation Errors',
+              count: o.issues.ampValidationErrors,
+              category: 'issues:amp-validation-errors',
+            },
           ],
         },
         {

apps/desktop/src/renderer/src/components/SettingsDialog.tsx

@@ -19,8 +19,11 @@ import {
   Code2,
   Cookie,
   Webhook,
+  Download,
+  Play,
   Plus,
   Trash2,
+  Upload,
   Shield,
   Network,
   Sparkles,
@@ -48,6 +51,7 @@ import type {
 import { DEFAULT_CRAWL_CONFIG, INTEGRATIONS } from '@freecrawl/shared-types';
 import { useAppStore } from '../store.js';
 import { InfoTip, type FieldInfo } from './InfoTip.js';
+import { ExtractionPreviewDialog } from './ExtractionPreviewDialog.js';
 
 interface Props {
   open: boolean;
@@ -1718,18 +1722,138 @@ const DEFAULT_RULE: CustomExtractionRule = {
 function CustomExtractionPanel({ form, update }: PanelProps) {
   const { t } = useTranslation();
   const rules = form.customExtractionRules;
+  // Default URL for the preview dialog — startUrl isn't part of
+  // FormState (TopBar owns it) so we pull from the store directly.
+  // Falls back to empty so the user types something themselves when no
+  // crawl has been kicked off yet.
+  const currentStartUrl = useAppStore((s) => s.config.startUrl);
   const setRules = (next: CustomExtractionRule[]) => update('customExtractionRules', next);
   const updateRule = (i: number, patch: Partial<CustomExtractionRule>) => {
     const next = rules.slice();
     next[i] = { ...next[i]!, ...patch };
     setRules(next);
   };
+  const [previewOpen, setPreviewOpen] = useState(false);
+  const previewableRules = rules.filter(
+    (r) => r.name.trim() && r.selector.trim(),
+  ).length;
   return (
     <>
-      <p className="mb-3 text-[11px] text-surface-400">
-        {t('settingsPanels.customExtraction.intro1', { defaultValue: 'Up to 10 custom extraction rules. Each runs against every crawled HTML page; results are stored on the URL row and visible in the URL Details panel under' })}{' '}
-        <strong>{t('settingsPanels.customExtraction.extractionLabel', { defaultValue: 'Extraction' })}</strong>.
-      </p>
+      <div className="mb-3 flex items-start justify-between gap-3">
+        <p className="text-[11px] text-surface-400">
+          {t('settingsPanels.customExtraction.intro1', { defaultValue: 'Up to 10 custom extraction rules. Each runs against every crawled HTML page; results are stored on the URL row and visible in the URL Details panel under' })}{' '}
+          <strong>{t('settingsPanels.customExtraction.extractionLabel', { defaultValue: 'Extraction' })}</strong>.
+        </p>
+        <div className="flex shrink-0 items-center gap-1">
+          <button
+            type="button"
+            onClick={async () => {
+              const result = await window.freecrawl.extractionRulesImport();
+              if (!result.filePath) {
+                if (result.error) {
+                  // Surface main-process parse errors inline rather than
+                  // silently dropping — invalid JSON is the most common
+                  // import failure mode and the user needs to know.
+                  // eslint-disable-next-line no-alert
+                  alert(`Import failed: ${result.error}`);
+                }
+                return;
+              }
+              if (result.rules.length === 0) {
+                // eslint-disable-next-line no-alert
+                alert(
+                  result.error ??
+                    'No valid rules found in the file.',
+                );
+                return;
+              }
+              if (rules.length > 0) {
+                // eslint-disable-next-line no-alert, no-restricted-globals
+                const ok = confirm(
+                  t('settingsPanels.customExtraction.importConfirm', {
+                    defaultValue:
+                      'Replace {{existing}} existing rule(s) with {{incoming}} imported rule(s)?',
+                    existing: rules.length,
+                    incoming: result.rules.length,
+                  }) as string,
+                );
+                if (!ok) return;
+              }
+              setRules(result.rules);
+              if (result.skippedCount > 0) {
+                // eslint-disable-next-line no-alert
+                alert(
+                  t('settingsPanels.customExtraction.importSkipped', {
+                    defaultValue:
+                      'Imported {{ok}} rule(s). {{skipped}} entries were skipped (invalid shape or over the 10-rule cap).',
+                    ok: result.rules.length,
+                    skipped: result.skippedCount,
+                  }) as string,
+                );
+              }
+            }}
+            className="inline-flex items-center gap-1.5 rounded border border-surface-700 px-2 py-1 text-[11px] text-surface-300 hover:bg-surface-800"
+            title={t('settingsPanels.customExtraction.importTitle', {
+              defaultValue:
+                'Replace current rules with a JSON file. Accepts {rules: [...]} envelope or a bare array.',
+            })}
+          >
+            <Upload className="h-3 w-3" />
+            {t('settingsPanels.customExtraction.import', { defaultValue: 'Import' })}
+          </button>
+          <button
+            type="button"
+            onClick={() => {
+              void window.freecrawl.extractionRulesExport(rules);
+            }}
+            disabled={rules.length === 0}
+            className={clsx(
+              'inline-flex items-center gap-1.5 rounded border px-2 py-1 text-[11px] transition',
+              rules.length === 0
+                ? 'cursor-not-allowed border-surface-800 text-surface-600'
+                : 'border-surface-700 text-surface-300 hover:bg-surface-800',
+            )}
+            title={
+              rules.length === 0
+                ? t('settingsPanels.customExtraction.exportDisabled', {
+                    defaultValue: 'Add at least one rule before exporting.',
+                  })
+                : t('settingsPanels.customExtraction.exportTitle', {
+                    defaultValue:
+                      'Save the current rule set as a JSON file. Versioned envelope, safe to commit to source control.',
+                  })
+            }
+          >
+            <Download className="h-3 w-3" />
+            {t('settingsPanels.customExtraction.export', { defaultValue: 'Export' })}
+          </button>
+          <button
+            type="button"
+            onClick={() => setPreviewOpen(true)}
+            disabled={previewableRules === 0}
+            className={clsx(
+              'inline-flex items-center gap-1.5 rounded border px-2 py-1 text-[11px] transition',
+              previewableRules === 0
+                ? 'cursor-not-allowed border-surface-800 text-surface-600'
+                : 'border-accent-500/60 text-accent-300 hover:bg-accent-500/15',
+            )}
+            title={
+              previewableRules === 0
+                ? t('settingsPanels.customExtraction.previewDisabled', {
+                    defaultValue:
+                      'Add at least one rule with a name and selector to enable preview.',
+                  })
+                : t('settingsPanels.customExtraction.previewTitle', {
+                    defaultValue:
+                      'Test current rules against a URL before saving — no full crawl needed.',
+                  })
+            }
+          >
+            <Play className="h-3 w-3" />
+            {t('settingsPanels.customExtraction.preview', { defaultValue: 'Preview' })}
+          </button>
+        </div>
+      </div>
 
       {rules.length === 0 && (
         <p className="mb-3 text-[11px] italic text-surface-500">{t('settingsPanels.customExtraction.empty', { defaultValue: 'No rules — click "Add Rule" to start.' })}</p>
@@ -1897,6 +2021,15 @@ function CustomExtractionPanel({ form, update }: PanelProps) {
       {rules.length >= 10 && (
         <p className="text-[10px] text-surface-500">{t('settingsPanels.customExtraction.limitReached', { defaultValue: 'Limit reached (10 rules).' })}</p>
       )}
+
+      <ExtractionPreviewDialog
+        open={previewOpen}
+        onClose={() => setPreviewOpen(false)}
+        rules={rules}
+        defaultUrl={currentStartUrl}
+        userAgent={form.userAgent}
+        acceptLanguage={form.acceptLanguage}
+      />
     </>
   );
 }

apps/desktop/src/renderer/src/i18n/labels.ts

@@ -140,6 +140,7 @@ const TR_LABELS: Record<string, string> = {
   'Gunning Fog >17': 'Gunning Fog >17',
   'Lang Attribute Missing': 'Lang Niteliği Eksik',
   'Charset Missing': 'Charset Eksik',
+  'AMP Validation Errors': 'AMP Doğrulama Hataları',
 
   // Response performance
   'Slow (>1s)': 'Yavaş (>1sn)',

apps/desktop/src/renderer/src/i18n/locales/en.json

@@ -429,7 +429,17 @@
       "multiConcat": "Concat (` | `)",
       "multiCount": "Count",
       "addRule": "Add Rule",
-      "limitReached": "Limit reached (10 rules)."
+      "limitReached": "Limit reached (10 rules).",
+      "preview": "Preview",
+      "previewTitle": "Test current rules against a URL before saving — no full crawl needed.",
+      "previewDisabled": "Add at least one rule with a name and selector to enable preview.",
+      "import": "Import",
+      "export": "Export",
+      "importTitle": "Replace current rules with a JSON file. Accepts {rules: [...]} envelope or a bare array.",
+      "exportTitle": "Save the current rule set as a JSON file. Versioned envelope, safe to commit to source control.",
+      "exportDisabled": "Add at least one rule before exporting.",
+      "importConfirm": "Replace {{existing}} existing rule(s) with {{incoming}} imported rule(s)?",
+      "importSkipped": "Imported {{ok}} rule(s). {{skipped}} entries were skipped (invalid shape or over the 10-rule cap)."
     },
     "urlRewriting": {
       "intro": "Normalisation applied before URLs are deduplicated and queued.",
@@ -665,7 +675,16 @@
     "noOtherMembers": "No other members in this cluster. (Cluster size {{n}}.)",
     "cluster": "Cluster",
     "membersTotal": "{{n}} member(s) total",
-    "hammingHint": "(Hamming distance from this URL — lower means more similar)"
+    "hammingHint": "(Hamming distance from this URL — lower means more similar)",
+    "loadingClusters": "Loading near-duplicate clusters…",
+    "noClustersTitle": "No near-duplicate clusters",
+    "noClustersHint": "Run a crawl and let post-crawl analysis finish — clusters appear once two or more pages share enough content to fall under the SimHash Hamming threshold (Settings → Duplicates).",
+    "statusClusters": "{{clusters}} cluster · {{members}} pages",
+    "truncated": "truncated to first {{cap}} rows",
+    "toggleClusterTitle": "Click to expand/collapse this cluster",
+    "headerSize": "{{n}} pages",
+    "maxHamming": "max Δ {{n}} bits",
+    "hammingFromRepTitle": "SimHash Hamming distance from the cluster representative — 0 means the rep itself, higher = further from the rep"
   },
   "resources": {
     "loadingSource": "Loading source…",
@@ -685,6 +704,21 @@
     "customExtraction": "Custom Extraction",
     "customSearchHits": "Custom Search hits"
   },
+  "extractionPreview": {
+    "title": "Custom Extraction Preview",
+    "subtitle": "Run all configured rules against a single URL to validate selectors / regex before crawling.",
+    "noRules": "Add at least one rule (with a name and selector) before previewing.",
+    "runTitle": "Fetch the URL and run every rule against the response.",
+    "run": "Run preview ({{n}} rule(s))",
+    "running": "Running…",
+    "idle": "Enter a URL above and click Run preview to fetch and apply every rule.",
+    "fetching": "Fetching URL and parsing response…",
+    "failed": "Preview failed",
+    "noMatch": "no match",
+    "noResults": "No results — add rules in the Custom Extraction panel.",
+    "colName": "Rule",
+    "colValue": "Value"
+  },
   "structured": {
     "blocks": "block(s)",
     "invalid": "{{n}} invalid",
@@ -1044,10 +1078,14 @@
     "hideAllExceptRow": "Hide every column except Row #",
     "showAll": "Show all",
     "hideAll": "Hide all",
+    "pinColumn": "Pin this column to the left edge",
+    "unpinColumn": "Unpin this column",
+    "reorderColumn": "Drag to reorder column",
     "columnHeaderHint": "(click to select · drag to multi-select · right-click to hide)",
     "quickFilterTitle": "Quick filter for this tab",
     "listView": "List view",
     "treeView": "Tree view (URL hierarchy)",
+    "clusterView": "Cluster view (grouped by near-duplicate clusters)",
     "treeStatus": "Tree view · {{rows}} URL · {{folders}} folder",
     "treeTruncated": "truncated to first {{cap}}",
     "treeLoading": "Loading URL tree…",

apps/desktop/src/renderer/src/i18n/locales/tr.json

@@ -428,7 +428,17 @@
       "multiConcat": "Birleştir (` | `)",
       "multiCount": "Sayı",
       "addRule": "Kural Ekle",
-      "limitReached": "Limit aşıldı (10 kural)."
+      "limitReached": "Limit aşıldı (10 kural).",
+      "preview": "Önizle",
+      "previewTitle": "Kaydetmeden önce mevcut kuralları bir URL üzerinde test et — tam crawl gerekmez.",
+      "previewDisabled": "Önizlemeyi etkinleştirmek için en az bir kural (ad + selector) ekle.",
+      "import": "İçe aktar",
+      "export": "Dışa aktar",
+      "importTitle": "Mevcut kuralları JSON dosyasıyla değiştir. {rules: [...]} sarmalı veya çıplak dizi kabul edilir.",
+      "exportTitle": "Mevcut kural setini JSON dosyasına kaydet. Versiyonlu sarmal, kaynak kontrolüne commit'lenebilir.",
+      "exportDisabled": "Dışa aktarmadan önce en az bir kural ekle.",
+      "importConfirm": "Mevcut {{existing}} kural, içe aktarılan {{incoming}} kuralla değiştirilsin mi?",
+      "importSkipped": "{{ok}} kural içe aktarıldı. {{skipped}} kayıt atlandı (geçersiz şema veya 10 kuralın üzerinde)."
     },
     "urlRewriting": {
       "intro": "URL'ler yinelemeden ve kuyruğa alınmadan önce uygulanan normalleştirme.",
@@ -664,7 +674,16 @@
     "noOtherMembers": "Bu kümede başka üye yok. (Küme boyutu {{n}}.)",
     "cluster": "Küme",
     "membersTotal": "Toplam {{n}} üye",
-    "hammingHint": "(Bu URL'den Hamming mesafesi — düşük = daha benzer)"
+    "hammingHint": "(Bu URL'den Hamming mesafesi — düşük = daha benzer)",
+    "loadingClusters": "Yakın-yinelenen kümeleri yükleniyor…",
+    "noClustersTitle": "Yakın-yinelenen küme yok",
+    "noClustersHint": "Bir crawl çalıştır ve post-crawl analizi tamamlansın — iki ya da daha fazla sayfa SimHash Hamming eşiğinin altında yeterli içerik paylaştığında kümeler görünür (Settings → Duplicates).",
+    "statusClusters": "{{clusters}} küme · {{members}} sayfa",
+    "truncated": "ilk {{cap}} satırla sınırlı",
+    "toggleClusterTitle": "Bu kümeyi açmak/kapatmak için tıkla",
+    "headerSize": "{{n}} sayfa",
+    "maxHamming": "maks. Δ {{n}} bit",
+    "hammingFromRepTitle": "Küme temsilcisinden SimHash Hamming mesafesi — 0 temsilcinin kendisi, yüksek = temsilciden daha uzak"
   },
   "resources": {
     "loadingSource": "Kaynak yükleniyor…",
@@ -684,6 +703,21 @@
     "customExtraction": "Özel Çıkarım",
     "customSearchHits": "Özel Arama Sonuçları"
   },
+  "extractionPreview": {
+    "title": "Özel Çıkarım Önizlemesi",
+    "subtitle": "Crawl başlatmadan önce tüm kuralları tek bir URL üzerinde çalıştırarak selector / regex'leri doğrula.",
+    "noRules": "Önizleme yapmadan önce en az bir kural (ad + selector ile) ekle.",
+    "runTitle": "URL'i çek ve yanıta her kuralı uygula.",
+    "run": "Önizlemeyi çalıştır ({{n}} kural)",
+    "running": "Çalışıyor…",
+    "idle": "Yukarıya bir URL gir ve her kuralı uygulamak için Önizlemeyi çalıştır'a bas.",
+    "fetching": "URL çekiliyor ve yanıt parse ediliyor…",
+    "failed": "Önizleme başarısız",
+    "noMatch": "eşleşme yok",
+    "noResults": "Sonuç yok — Custom Extraction panelinde kural ekleyin.",
+    "colName": "Kural",
+    "colValue": "Değer"
+  },
   "structured": {
     "blocks": "blok",
     "invalid": "{{n}} geçersiz",
@@ -1043,10 +1077,14 @@
     "hideAllExceptRow": "Satır # dışındaki her kolonu gizle",
     "showAll": "Tümünü göster",
     "hideAll": "Tümünü gizle",
+    "pinColumn": "Bu kolonu sol kenara sabitle",
+    "unpinColumn": "Bu kolonun sabitlemesini kaldır",
+    "reorderColumn": "Kolonu sürükleyerek yeniden sırala",
     "columnHeaderHint": "(seçmek için tıkla · çoklu seçim için sürükle · gizlemek için sağ-tıkla)",
     "quickFilterTitle": "Bu sekme için hızlı filtre",
     "listView": "Liste görünümü",
     "treeView": "Ağaç görünümü (URL hiyerarşisi)",
+    "clusterView": "Küme görünümü (yakın-tekrar kümelerine göre gruplu)",
     "treeStatus": "Ağaç görünümü · {{rows}} URL · {{folders}} klasör",
     "treeTruncated": "ilk {{cap}} satırla sınırlı",
     "treeLoading": "URL ağacı yükleniyor…",

apps/desktop/src/renderer/src/store.ts

@@ -515,6 +515,7 @@ function tabForCategory(cat: UrlCategory): TabKey | null {
     if (cat === 'issues:meta-refresh-used') return 'meta-refresh';
     return 'meta-description';
   }
+  if (cat === 'issues:amp-validation-errors') return 'amp';
   if (cat.startsWith('issues:h1-')) return 'h1';
   if (cat === 'issues:heading-skipped-level') return 'h2';
   if (cat.startsWith('issues:canonical-')) return 'canonicals';

apps/desktop/src/renderer/src/tabs/UrlsTab.tsx

@@ -1,12 +1,30 @@
-import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
+import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react';
+import { createPortal } from 'react-dom';
 import { useVirtualizer } from '@tanstack/react-virtual';
+import {
+  DndContext,
+  PointerSensor,
+  useSensor,
+  useSensors,
+  closestCenter,
+  type DragEndEvent,
+} from '@dnd-kit/core';
+import {
+  SortableContext,
+  horizontalListSortingStrategy,
+  useSortable,
+} from '@dnd-kit/sortable';
+import { CSS } from '@dnd-kit/utilities';
 import {
   ChevronsUpDown,
   Columns3,
   Download,
   Filter,
+  GripVertical,
+  Layers,
   List,
   Network,
+  Pin,
   X,
 } from 'lucide-react';
 import clsx from 'clsx';
@@ -20,6 +38,7 @@ import { useAppStore, TAB_QUICK_FILTERS, type TabKey } from '../store.js';
 import { COLUMN_SPECS, columnId, type ColumnSpec } from './columns.js';
 import { useLazyUrlRows } from '../hooks/useLazyUrlRows.js';
 import { AdvancedFilterDialog } from '../components/AdvancedFilterDialog.js';
+import { DuplicatesGroupedView } from '../components/DuplicatesGroupedView.js';
 import { UrlTreeView } from '../components/UrlTreeView.js';
 import { ErrorBoundary } from '../components/ErrorBoundary.js';
 import { ExportDialog } from '../components/ExportDialog.js';
@@ -34,6 +53,8 @@ const ROW_NUM_KEY = '__row_num__';
 const STATUS_BAR_HEIGHT = 22;
 const PREFS_PREFIX = 'col-widths:';
 const PREFS_HIDDEN_PREFIX = 'col-hidden:';
+const PREFS_PINNED_LEFT_PREFIX = 'col-pinned-left:';
+const PREFS_ORDER_PREFIX = 'col-order:';
 
 function loadStoredWidths(tab: TabKey): Record<string, number> {
   const value = window.freecrawl.prefsGet(PREFS_PREFIX + tab);
@@ -59,6 +80,47 @@ function saveHiddenColumns(tab: TabKey, hidden: Set<string>): void {
   }
 }
 
+/** Per-tab persistence for the pinned-left column set. Order is preserved
+ *  so users can pin columns in the order they want them stacked at the
+ *  left edge — first-pinned ends up nearest the row-number column. */
+function loadPinnedLeft(tab: TabKey): string[] {
+  const value = window.freecrawl.prefsGet(PREFS_PINNED_LEFT_PREFIX + tab);
+  if (Array.isArray(value)) {
+    return value.filter((v): v is string => typeof v === 'string');
+  }
+  return [];
+}
+
+function savePinnedLeft(tab: TabKey, pinned: string[]): void {
+  try {
+    window.freecrawl.prefsSet(PREFS_PINNED_LEFT_PREFIX + tab, pinned);
+  } catch {
+    /* best-effort */
+  }
+}
+
+/** Per-tab user-defined column ordering, applied AFTER hide-filter +
+ *  pin-prepend. Stored as an array of column ids in left-to-right order
+ *  for the *unpinned* region. Columns that don't appear in the stored
+ *  array fall through to their `COLUMN_SPECS` default position so a
+ *  partial preference (the user dragged only one column) doesn't reset
+ *  everything else. */
+function loadColumnOrder(tab: TabKey): string[] {
+  const value = window.freecrawl.prefsGet(PREFS_ORDER_PREFIX + tab);
+  if (Array.isArray(value)) {
+    return value.filter((v): v is string => typeof v === 'string');
+  }
+  return [];
+}
+
+function saveColumnOrder(tab: TabKey, order: string[]): void {
+  try {
+    window.freecrawl.prefsSet(PREFS_ORDER_PREFIX + tab, order);
+  } catch {
+    /* best-effort */
+  }
+}
+
 export function UrlsTab() {
   const { t, i18n } = useTranslation();
   const lang = i18n.language;
@@ -74,11 +136,14 @@ export function UrlsTab() {
   const [filter, setFilter] = useState<AdvancedFilter | null>(null);
   const [filterDialogOpen, setFilterDialogOpen] = useState(false);
   const [exportDialogOpen, setExportDialogOpen] = useState(false);
-  /** View mode toggle — list (default flat virtual table) or tree
-   *  (URL-path hierarchy). Per-tab + persisted across mounts via prefs. */
-  const [viewMode, setViewMode] = useState<'list' | 'tree'>(() => {
+  /** View mode toggle — list (flat virtual table), tree (URL-path
+   *  hierarchy), or cluster (Duplicates-tab-only near-duplicate cluster
+   *  grouping). Per-tab + persisted across mounts via prefs. */
+  const [viewMode, setViewMode] = useState<'list' | 'tree' | 'cluster'>(() => {
     const stored = window.freecrawl.prefsGet(`view-mode:${activeTab}`);
-    return stored === 'tree' ? 'tree' : 'list';
+    if (stored === 'tree') return 'tree';
+    if (stored === 'cluster' && activeTab === 'duplicates') return 'cluster';
+    return 'list';
   });
   const quickFilters = TAB_QUICK_FILTERS[activeTab];
   const activeQuickFilter = useMemo(() => {
@@ -133,18 +198,74 @@ export function UrlsTab() {
   const [hiddenColumns, setHiddenColumnsState] = useState<Set<string>>(() =>
     loadHiddenColumns(activeTab),
   );
+  /** Ordered list of column ids pinned to the left edge. Order = stacking
+   *  order from inner (next to row-num) outward. Stored as a list (not a
+   *  Set) so the stacking order is deterministic across reloads. */
+  const [pinnedLeftOrder, setPinnedLeftOrderState] = useState<string[]>(() =>
+    loadPinnedLeft(activeTab),
+  );
+  /** User-defined order of UNPINNED columns. Pinned columns are
+   *  positioned via {@link pinnedLeftOrder}; this list governs only the
+   *  scrollable region. Drag-drop on a header writes here. */
+  const [columnOrder, setColumnOrderState] = useState<string[]>(() =>
+    loadColumnOrder(activeTab),
+  );
   const [columnPickerOpen, setColumnPickerOpen] = useState(false);
   const scrollRef = useRef<HTMLDivElement>(null);
+  /** Ref on the "Columns" toolbar button so the popover (rendered via
+   *  portal at document.body) can compute its position from the
+   *  anchor's bounding rect — without the portal it would be clipped
+   *  by the resizable bottom detail panel which lives in a separate
+   *  stacking context. */
+  const columnsAnchorRef = useRef<HTMLButtonElement>(null);
 
   const allColumns = COLUMN_SPECS[activeTab];
   // Visible columns drive the rendered header + body. Hidden ids that no
   // longer exist on the active tab (e.g. user switched between tabs with
   // different specs) are silently filtered out — they remain in the saved
   // set so toggling back to the originating tab restores them.
-  const columns = useMemo(
-    () => allColumns.filter((c) => !hiddenColumns.has(columnId(c))),
-    [allColumns, hiddenColumns],
-  );
+  //
+  // Pinned columns are floated to the head of the visible list (in the
+  // order persisted to prefs) so the row render loop sees pinned cells
+  // first — that order, combined with the cumulative-offset table built
+  // below, drives the CSS `position: sticky; left: …` layout. Visual
+  // outcome: the row-number cell, then the pinned strip in user-chosen
+  // order, then the scrollable region.
+  const columns = useMemo(() => {
+    const visible = allColumns.filter((c) => !hiddenColumns.has(columnId(c)));
+    const byId = new Map(visible.map((c) => [columnId(c), c] as const));
+
+    // (1) Pinned region in pin-stack order.
+    const pinned: ColumnSpec[] = [];
+    for (const id of pinnedLeftOrder) {
+      const c = byId.get(id);
+      if (c) {
+        pinned.push(c);
+        byId.delete(id);
+      }
+    }
+
+    // (2) Unpinned region — explicit user order (drag-drop) first, then
+    // anything left over in `COLUMN_SPECS` default order so partial
+    // preferences don't reset newly-added columns.
+    const unpinned: ColumnSpec[] = [];
+    for (const id of columnOrder) {
+      const c = byId.get(id);
+      if (c) {
+        unpinned.push(c);
+        byId.delete(id);
+      }
+    }
+    for (const c of allColumns) {
+      const id = columnId(c);
+      if (byId.has(id)) {
+        unpinned.push(c);
+        byId.delete(id);
+      }
+    }
+
+    return [...pinned, ...unpinned];
+  }, [allColumns, hiddenColumns, pinnedLeftOrder, columnOrder]);
 
   const setHiddenColumns = useCallback(
     (next: Set<string>) => {
@@ -154,13 +275,79 @@ export function UrlsTab() {
     [activeTab],
   );
 
+  const setPinnedLeftOrder = useCallback(
+    (next: string[]) => {
+      setPinnedLeftOrderState(next);
+      savePinnedLeft(activeTab, next);
+    },
+    [activeTab],
+  );
+
+  const setColumnOrder = useCallback(
+    (next: string[]) => {
+      setColumnOrderState(next);
+      saveColumnOrder(activeTab, next);
+    },
+    [activeTab],
+  );
+
+  /** Set view for callers that only need membership lookups. */
+  const pinnedLeftSet = useMemo(
+    () => new Set(pinnedLeftOrder),
+    [pinnedLeftOrder],
+  );
+
+  /** Toggle a column id between pinned and unpinned. When pinning, the
+   *  new id is appended to the right end of the pinned strip so the
+   *  newest pin sits next to the unpinned region — keeps existing pins
+   *  visually stable. */
+  const togglePinned = useCallback(
+    (id: string) => {
+      const idx = pinnedLeftOrder.indexOf(id);
+      if (idx >= 0) {
+        setPinnedLeftOrder(pinnedLeftOrder.filter((x) => x !== id));
+      } else {
+        setPinnedLeftOrder([...pinnedLeftOrder, id]);
+      }
+    },
+    [pinnedLeftOrder, setPinnedLeftOrder],
+  );
+
+  /** dnd-kit pointer sensor with a 5 px activation constraint so a quick
+   *  click on the grip handle doesn't start a drag (the user still has
+   *  to deliberately drag). Without this, clicks would race the cell's
+   *  onMouseDown handler for column-selection drag. */
+  const dndSensors = useSensors(
+    useSensor(PointerSensor, { activationConstraint: { distance: 5 } }),
+  );
+
   useEffect(() => {
     setColumnWidths(loadStoredWidths(activeTab));
     setHiddenColumnsState(loadHiddenColumns(activeTab));
+    setPinnedLeftOrderState(loadPinnedLeft(activeTab));
+    setColumnOrderState(loadColumnOrder(activeTab));
     setColumnPickerOpen(false);
-    // Pick up the persisted view-mode for the new tab.
+    // Scope ephemeral view state (search box / sort column / advanced
+    // filter) to the active tab — these are intentionally NOT persisted
+    // because each tab has a different column / category vocabulary
+    // (a sort by "depth" on Internal makes no sense on Response Codes,
+    // a filter clause on `images_missing_alt` is meaningless outside
+    // Images, etc). Without these resets the filter chips of one tab
+    // would silently apply to every other tab the user visits — which
+    // is exactly the bug surfaced in 2026-06-01.
+    setSearch('');
+    setSortBy(undefined);
+    setSortDir('asc');
+    setFilter(null);
+    setFilterDialogOpen(false);
+    // Pick up the persisted view-mode for the new tab. Cluster mode is
+    // duplicates-tab-only — switching to any other tab while it was the
+    // active mode falls back to list so the toolbar toggle doesn't get
+    // stuck in an unreachable state.
     const stored = window.freecrawl.prefsGet(`view-mode:${activeTab}`);
-    setViewMode(stored === 'tree' ? 'tree' : 'list');
+    if (stored === 'tree') setViewMode('tree');
+    else if (stored === 'cluster' && activeTab === 'duplicates') setViewMode('cluster');
+    else setViewMode('list');
   }, [activeTab]);
 
   // Persist viewMode whenever the user toggles it.
@@ -222,6 +409,29 @@ export function UrlsTab() {
     [columns, getWidth, rowNumWidth],
   );
 
+  /** Cumulative left-offset for each pinned column. Maps column id →
+   *  pixel offset from the table's left edge so the cell's
+   *  `position: sticky; left: <offset>` lands flush against the previous
+   *  pinned column. Pinned columns are always rendered first in
+   *  `columns`, so iterating from the front stops as soon as we hit a
+   *  non-pinned column. */
+  const pinnedLeftOffsets = useMemo(() => {
+    const map = new Map<string, number>();
+    let acc = rowNumWidth;
+    for (const c of columns) {
+      const id = columnId(c);
+      if (!pinnedLeftSet.has(id)) break;
+      map.set(id, acc);
+      acc += getWidth(c);
+    }
+    return map;
+  }, [columns, pinnedLeftSet, getWidth, rowNumWidth]);
+
+  /** Number of pinned columns currently rendered as sticky — used by
+   *  the resize handler to recompute downstream offsets implicitly via
+   *  state change. */
+  const pinnedCount = pinnedLeftOffsets.size;
+
   // Only dataVersion (context-menu mutations like Remove / Re-Spider) forces
   // a snapshot rebuild. Crawler progress ticks intentionally do NOT rebuild
   // — the user sees a "N new rows · Refresh" pill instead so the sorted
@@ -263,6 +473,52 @@ export function UrlsTab() {
     lazy.ensureRange(first, last);
   }, [virtualRows, lazy]);
 
+  /** Apply a drag-drop column reorder. Pin state is preserved: drags
+   *  inside the pinned strip reorder `pinnedLeftOrder`; drags inside the
+   *  unpinned strip reorder `columnOrder`. Cross-strip drags snap back
+   *  silently — the user has to explicitly toggle pin via the column
+   *  picker if they want to change the partition. This keeps pin /
+   *  reorder as two orthogonal concepts the user can reason about
+   *  independently. */
+  const handleDragEnd = useCallback(
+    (event: DragEndEvent) => {
+      const { active, over } = event;
+      if (!over || active.id === over.id) return;
+      const fromId = String(active.id);
+      const toId = String(over.id);
+      const fromIsPinned = pinnedLeftSet.has(fromId);
+      const toIsPinned = pinnedLeftSet.has(toId);
+      if (fromIsPinned !== toIsPinned) return;
+
+      if (fromIsPinned) {
+        const list = pinnedLeftOrder.slice();
+        const fi = list.indexOf(fromId);
+        const ti = list.indexOf(toId);
+        if (fi < 0 || ti < 0) return;
+        list.splice(fi, 1);
+        list.splice(ti, 0, fromId);
+        setPinnedLeftOrder(list);
+      } else {
+        // Snapshot the *currently displayed* unpinned order — the stored
+        // `columnOrder` may be partial, but what's visible is what the
+        // user is reordering, so persist the full sequence to avoid
+        // surprising fall-through behaviour for never-dragged columns.
+        const unpinnedIds: string[] = [];
+        for (const c of columns) {
+          const id = columnId(c);
+          if (!pinnedLeftSet.has(id)) unpinnedIds.push(id);
+        }
+        const fi = unpinnedIds.indexOf(fromId);
+        const ti = unpinnedIds.indexOf(toId);
+        if (fi < 0 || ti < 0) return;
+        unpinnedIds.splice(fi, 1);
+        unpinnedIds.splice(ti, 0, fromId);
+        setColumnOrder(unpinnedIds);
+      }
+    },
+    [columns, pinnedLeftOrder, pinnedLeftSet, setColumnOrder, setPinnedLeftOrder],
+  );
+
   const handleSort = (key: keyof CrawlUrlRow) => {
     if (sortBy === key) {
       setSortDir(sortDir === 'asc' ? 'desc' : 'asc');
@@ -653,9 +909,27 @@ export function UrlsTab() {
           >
             <Network className="h-3.5 w-3.5" />
           </button>
+          {activeTab === 'duplicates' && (
+            <button
+              type="button"
+              onClick={() => setViewMode('cluster')}
+              className={clsx(
+                'inline-flex items-center justify-center border-l border-surface-700 px-2 py-1 text-[11px] transition',
+                viewMode === 'cluster'
+                  ? 'bg-accent-500/15 text-accent-300'
+                  : 'text-surface-300 hover:bg-surface-800',
+              )}
+              title={t('urlsTab.clusterView', {
+                defaultValue: 'Cluster view (grouped by near-duplicate clusters)',
+              })}
+            >
+              <Layers className="h-3.5 w-3.5" />
+            </button>
+          )}
         </div>
         <div className="relative">
           <button
+            ref={columnsAnchorRef}
             type="button"
             data-columns-anchor="1"
             onClick={() => setColumnPickerOpen((v) => !v)}
@@ -686,9 +960,12 @@ export function UrlsTab() {
           </button>
           {columnPickerOpen && (
             <ColumnPickerPopover
+              anchorRef={columnsAnchorRef}
               allColumns={allColumns}
               hiddenColumns={hiddenColumns}
+              pinnedLeft={pinnedLeftSet}
               onChange={setHiddenColumns}
+              onTogglePin={togglePinned}
               onClose={() => setColumnPickerOpen(false)}
             />
           )}
@@ -728,6 +1005,8 @@ export function UrlsTab() {
             }
           }}
         />
+      ) : viewMode === 'cluster' ? (
+        <DuplicatesGroupedView />
       ) : (
       <div ref={scrollRef} className="relative flex-1 select-none overflow-auto">
         <div style={{ minWidth: totalWidth, width: '100%' }}>
@@ -742,6 +1021,14 @@ export function UrlsTab() {
                 width: rowNumWidth,
                 minWidth: rowNumWidth,
                 flex: `0 0 ${rowNumWidth}px`,
+                // Row-number cell is always pinned at the table's left
+                // edge so the index never scrolls out of view. Background
+                // matches the header so cells scrolling underneath are
+                // masked.
+                position: 'sticky',
+                left: 0,
+                zIndex: 13,
+                background: 'rgb(23 23 23)',
               }}
               title="Row number"
             >
@@ -762,86 +1049,49 @@ export function UrlsTab() {
                 title="Drag to resize · double-click to reset"
               />
             </div>
-            {columns.map((c, colIdx) => {
-              const w = getWidth(c);
-              const id = columnId(c);
-              const isColSelected = selectedColumns.has(colIdx);
-              const isSortCol = sortBy === c.key;
-              return (
-                <div
-                  key={id}
-                  className={clsx(
-                    'group/header relative flex select-none items-center gap-1 border-b border-r border-surface-800 pl-2 pr-1 font-medium',
-                    isColSelected
-                      ? 'bg-accent-500/25 text-surface-50'
-                      : 'text-surface-300 hover:text-surface-100',
-                  )}
-                  style={{ width: w, minWidth: w, flex: `0 0 ${w}px` }}
-                  onMouseDown={(e) => beginColumnDrag(colIdx, e)}
-                  onMouseEnter={() => {
-                    if (dragRef.current?.kind === 'column') applyColumnDrag(colIdx);
-                  }}
-                  onContextMenu={(e) => {
-                    e.preventDefault();
-                    e.stopPropagation();
-                    const next = new Set(hiddenColumns);
-                    next.add(columnId(c));
-                    setHiddenColumns(next);
-                  }}
-                  title={translateLabel(c.header, lang) + ' ' + t('urlsTab.columnHeaderHint', { defaultValue: '(click to select · drag to multi-select · right-click to hide)' })}
-                >
-                  <span className="cursor-pointer truncate">{translateLabel(c.header, lang)}</span>
-                  {(c.info || c.example) && (
-                    <span
-                      className="shrink-0"
-                      onMouseDown={(e) => e.stopPropagation()}
-                      onClick={(e) => e.stopPropagation()}
-                    >
-                      <InfoTip info={c.info} example={c.example} />
-                    </span>
-                  )}
-                  <button
-                    type="button"
-                    className={clsx(
-                      'ml-auto flex shrink-0 items-center rounded px-0.5 hover:bg-surface-800',
-                      isSortCol ? 'text-accent-300' : 'text-surface-600 hover:text-surface-300',
-                    )}
-                    onClick={(e) => {
-                      e.stopPropagation();
-                      handleSort(c.key);
-                    }}
-                    title={
-                      isSortCol
-                        ? `Sorted ${sortDir === 'asc' ? 'ascending' : 'descending'} — click to flip`
-                        : 'Sort by this column'
-                    }
-                  >
-                    {isSortCol ? (
-                      <span className="text-[10px]">
-                        {sortDir === 'asc' ? '▲' : '▼'}
-                      </span>
-                    ) : (
-                      <ChevronsUpDown className="h-3 w-3" />
-                    )}
-                  </button>
-                  <div
-                    className="absolute -right-1 top-0 bottom-0 z-20 w-2 cursor-col-resize hover:bg-accent-500/40"
-                    onMouseDown={(e) => {
-                      e.preventDefault();
-                      e.stopPropagation();
-                      startResize(id, w, e.clientX);
-                    }}
-                    onDoubleClick={(e) => {
-                      e.preventDefault();
-                      e.stopPropagation();
-                      resetColumn(id);
-                    }}
-                    onClick={(e) => e.stopPropagation()}
-                    title="Drag to resize · double-click to reset"
-                  />
-                </div>
-              );
-            })}
+            <DndContext
+              sensors={dndSensors}
+              collisionDetection={closestCenter}
+              onDragEnd={handleDragEnd}
+            >
+              <SortableContext
+                items={columns.map((c) => columnId(c))}
+                strategy={horizontalListSortingStrategy}
+              >
+                {columns.map((c, colIdx) => {
+                  const id = columnId(c);
+                  const stickyLeft = pinnedLeftOffsets.get(id);
+                  return (
+                    <SortableHeaderCell
+                      key={id}
+                      id={id}
+                      spec={c}
+                      colIdx={colIdx}
+                      width={getWidth(c)}
+                      lang={lang}
+                      isColSelected={selectedColumns.has(colIdx)}
+                      isSortCol={sortBy === c.key}
+                      sortDir={sortDir}
+                      stickyLeft={stickyLeft}
+                      isPinned={stickyLeft !== undefined}
+                      isLastPinned={stickyLeft !== undefined && colIdx + 1 === pinnedCount}
+                      onSort={handleSort}
+                      onBeginColumnDrag={beginColumnDrag}
+                      onColumnMouseEnter={() => {
+                        if (dragRef.current?.kind === 'column') applyColumnDrag(colIdx);
+                      }}
+                      onContextHide={(hid) => {
+                        const next = new Set(hiddenColumns);
+                        next.add(hid);
+                        setHiddenColumns(next);
+                      }}
+                      onStartResize={startResize}
+                      onResetColumn={resetColumn}
+                    />
+                  );
+                })}
+              </SortableContext>
+            </DndContext>
             {/* Filler cell — extends the header bar across the remaining
                 empty space so the table doesn't look truncated when the
                 viewport is wider than the summed column widths. */}
@@ -914,6 +1164,20 @@ export function UrlsTab() {
                       minWidth: rowNumWidth,
                       flex: `0 0 ${rowNumWidth}px`,
                       height: '100%',
+                      // Always-pinned row index. Solid background masks
+                      // cells scrolling underneath; selection state still
+                      // wins (the conditional bg-accent-500/30 class
+                      // overrides the inline background via class
+                      // ordering, but Tailwind's accent class won't beat
+                      // an inline style, so paint the body-default and
+                      // let the selected highlight come from the cell's
+                      // outer ring + text colour).
+                      position: 'sticky',
+                      left: 0,
+                      zIndex: 9,
+                      background: rowSelected
+                        ? 'rgb(37 99 235 / 0.30)'
+                        : 'rgb(10 10 10)',
                     }}
                     onMouseDown={(e) => {
                       if (row) beginRowDrag(row.id, vi.index, e);
@@ -927,13 +1191,17 @@ export function UrlsTab() {
                   </div>
                   {columns.map((c, colIdx) => {
                     const w = getWidth(c);
+                    const id = columnId(c);
                     const cellSel =
                       row !== null &&
                       (selectedCells.has(`${row.id}:${colIdx}`) ||
                         selectedColumns.has(colIdx));
+                    const stickyLeft = pinnedLeftOffsets.get(id);
+                    const isPinned = stickyLeft !== undefined;
+                    const isLastPinned = isPinned && colIdx + 1 === pinnedCount;
                     return (
                       <div
-                        key={columnId(c)}
+                        key={id}
                         className={clsx(
                           'flex cursor-cell items-center overflow-hidden border-r border-surface-900 px-2',
                           cellSel
@@ -945,6 +1213,19 @@ export function UrlsTab() {
                           minWidth: w,
                           flex: `0 0 ${w}px`,
                           height: '100%',
+                          ...(isPinned && {
+                            position: 'sticky' as const,
+                            left: stickyLeft,
+                            zIndex: 8,
+                            background: cellSel
+                              ? 'rgb(37 99 235 / 0.30)'
+                              : rowSelected
+                                ? 'rgb(37 99 235 / 0.15)'
+                                : 'rgb(10 10 10)',
+                            boxShadow: isLastPinned
+                              ? '2px 0 4px rgba(0,0,0,0.4)'
+                              : undefined,
+                          }),
                         }}
                         onMouseDown={(e) => {
                           if (row) beginCellDrag(row.id, vi.index, colIdx, e);
@@ -1043,20 +1324,231 @@ export function UrlsTab() {
   );
 }
 
+/**
+ * One header cell wrapped in {@link useSortable} so the whole row of
+ * headers can be reordered by dragging the {@link GripVertical} handle
+ * on the left. The handle is the only drag-trigger: clicking anywhere
+ * else on the header preserves the existing semantics — left-click for
+ * column-selection drag, the sort-arrow button for sort, the resize
+ * handle on the right edge for resize, right-click to hide. Activation
+ * distance of 5 px (set on the parent's PointerSensor) makes the handle
+ * forgiving — a quick click doesn't slip into a drag.
+ *
+ * Sticky-left pin styling is preserved during drag (the cell stays
+ * pinned visually as the user drags it around), and the drag transform
+ * is composed on top so the cell follows the pointer.
+ */
+function SortableHeaderCell(props: {
+  id: string;
+  spec: ColumnSpec;
+  colIdx: number;
+  width: number;
+  lang: string;
+  isColSelected: boolean;
+  isSortCol: boolean;
+  sortDir: 'asc' | 'desc';
+  stickyLeft: number | undefined;
+  isPinned: boolean;
+  isLastPinned: boolean;
+  onSort: (key: keyof CrawlUrlRow) => void;
+  onBeginColumnDrag: (colIdx: number, e: React.MouseEvent) => void;
+  onColumnMouseEnter: () => void;
+  onContextHide: (id: string) => void;
+  onStartResize: (id: string, w: number, clientX: number) => void;
+  onResetColumn: (id: string) => void;
+}) {
+  const { t } = useTranslation();
+  const {
+    id,
+    spec: c,
+    colIdx,
+    width: w,
+    lang,
+    isColSelected,
+    isSortCol,
+    sortDir,
+    stickyLeft,
+    isPinned,
+    isLastPinned,
+    onSort,
+    onBeginColumnDrag,
+    onColumnMouseEnter,
+    onContextHide,
+    onStartResize,
+    onResetColumn,
+  } = props;
+
+  const {
+    attributes,
+    listeners,
+    setNodeRef,
+    transform,
+    transition,
+    isDragging,
+  } = useSortable({ id });
+
+  const style: React.CSSProperties = {
+    width: w,
+    minWidth: w,
+    flex: `0 0 ${w}px`,
+    ...(isPinned && {
+      position: 'sticky' as const,
+      left: stickyLeft,
+      zIndex: 12,
+      background: 'rgb(23 23 23)',
+      boxShadow: isLastPinned ? '2px 0 4px rgba(0,0,0,0.4)' : undefined,
+    }),
+    transform: CSS.Transform.toString(transform),
+    transition,
+    opacity: isDragging ? 0.5 : undefined,
+  };
+
+  return (
+    <div
+      ref={setNodeRef}
+      className={clsx(
+        'group/header relative flex select-none items-center gap-1 border-b border-r border-surface-800 pl-1 pr-1 font-medium',
+        isColSelected
+          ? 'bg-accent-500/25 text-surface-50'
+          : 'text-surface-300 hover:text-surface-100',
+      )}
+      style={style}
+      onMouseDown={(e) => onBeginColumnDrag(colIdx, e)}
+      onMouseEnter={onColumnMouseEnter}
+      onContextMenu={(e) => {
+        e.preventDefault();
+        e.stopPropagation();
+        onContextHide(id);
+      }}
+      title={
+        translateLabel(c.header, lang) +
+        ' ' +
+        t('urlsTab.columnHeaderHint', {
+          defaultValue:
+            '(click to select · drag to multi-select · right-click to hide)',
+        })
+      }
+    >
+      <button
+        type="button"
+        {...attributes}
+        {...listeners}
+        // Drag-handle isolation — don't let the grip's mousedown bubble
+        // up to the cell's column-selection drag handler, and don't let
+        // a click fall through to anything else.
+        onMouseDown={(e) => e.stopPropagation()}
+        onClick={(e) => e.stopPropagation()}
+        className="flex shrink-0 cursor-grab items-center text-surface-600 opacity-0 transition-opacity hover:text-surface-300 group-hover/header:opacity-100 active:cursor-grabbing"
+        title={t('urlsTab.reorderColumn', { defaultValue: 'Drag to reorder column' })}
+      >
+        <GripVertical className="h-3 w-3" />
+      </button>
+      <span className="cursor-pointer truncate">{translateLabel(c.header, lang)}</span>
+      {(c.info || c.example) && (
+        <span
+          className="shrink-0"
+          onMouseDown={(e) => e.stopPropagation()}
+          onClick={(e) => e.stopPropagation()}
+        >
+          <InfoTip info={c.info} example={c.example} />
+        </span>
+      )}
+      <button
+        type="button"
+        className={clsx(
+          'ml-auto flex shrink-0 items-center rounded px-0.5 hover:bg-surface-800',
+          isSortCol ? 'text-accent-300' : 'text-surface-600 hover:text-surface-300',
+        )}
+        onClick={(e) => {
+          e.stopPropagation();
+          onSort(c.key);
+        }}
+        onMouseDown={(e) => e.stopPropagation()}
+        title={
+          isSortCol
+            ? `Sorted ${sortDir === 'asc' ? 'ascending' : 'descending'} — click to flip`
+            : 'Sort by this column'
+        }
+      >
+        {isSortCol ? (
+          <span className="text-[10px]">{sortDir === 'asc' ? '▲' : '▼'}</span>
+        ) : (
+          <ChevronsUpDown className="h-3 w-3" />
+        )}
+      </button>
+      <div
+        className="absolute -right-1 top-0 bottom-0 z-20 w-2 cursor-col-resize hover:bg-accent-500/40"
+        onMouseDown={(e) => {
+          e.preventDefault();
+          e.stopPropagation();
+          onStartResize(id, w, e.clientX);
+        }}
+        onDoubleClick={(e) => {
+          e.preventDefault();
+          e.stopPropagation();
+          onResetColumn(id);
+        }}
+        onClick={(e) => e.stopPropagation()}
+        title="Drag to resize · double-click to reset"
+      />
+    </div>
+  );
+}
+
 function ColumnPickerPopover({
+  anchorRef,
   allColumns,
   hiddenColumns,
+  pinnedLeft,
   onChange,
+  onTogglePin,
   onClose,
 }: {
+  anchorRef: React.RefObject<HTMLButtonElement | null>;
   allColumns: ColumnSpec[];
   hiddenColumns: Set<string>;
+  pinnedLeft: Set<string>;
   onChange: (next: Set<string>) => void;
+  onTogglePin: (id: string) => void;
   onClose: () => void;
 }) {
   const { t, i18n } = useTranslation();
   const lang = i18n.language;
   const popRef = useRef<HTMLDivElement | null>(null);
+  /** Viewport-relative position derived from the anchor button's
+   *  bounding rect. Recomputed on every open + on window resize. The
+   *  popover is rendered into document.body via a portal so it isn't
+   *  clipped by the parent's overflow region or any sibling stacking
+   *  context (the bottom detail panel) — both of which previously hid
+   *  the lower checkboxes. */
+  const [pos, setPos] = useState<{
+    top: number;
+    right: number;
+    maxHeight: number;
+  } | null>(null);
+
+  useLayoutEffect(() => {
+    const updatePos = () => {
+      const anchor = anchorRef.current;
+      if (!anchor) return;
+      const rect = anchor.getBoundingClientRect();
+      const margin = 8;
+      const top = rect.bottom + 4;
+      const right = window.innerWidth - rect.right;
+      // Reserve `margin` px above the viewport bottom so the popover
+      // doesn't bleed off-screen on small viewports. Internal scroll
+      // (`overflow-y-auto` on the list region) takes over past that.
+      const maxHeight = Math.max(120, window.innerHeight - top - margin);
+      setPos({ top, right, maxHeight });
+    };
+    updatePos();
+    window.addEventListener('resize', updatePos);
+    window.addEventListener('scroll', updatePos, true);
+    return () => {
+      window.removeEventListener('resize', updatePos);
+      window.removeEventListener('scroll', updatePos, true);
+    };
+  }, [anchorRef]);
   // Outside-click closes the popover. We attach to mousedown so the
   // listener fires before any click handler inside the table can fire
   // and reopen us in the same gesture. Clicks on the anchoring "Columns"
@@ -1094,13 +1586,33 @@ function ColumnPickerPopover({
 
   const visibleCount = allColumns.length - hiddenColumns.size;
 
-  return (
+  if (!pos) return null;
+
+  const headerHeight = 33; /* matches py-2 + line-height of the top strip */
+
+  return createPortal(
     <div
       ref={popRef}
-      className="absolute right-0 top-full z-30 mt-1 w-80 rounded-md border border-surface-700 bg-surface-900 shadow-2xl"
+      className="w-80 rounded-md border border-surface-700 bg-surface-900 shadow-2xl"
+      style={{
+        position: 'fixed',
+        top: pos.top,
+        right: pos.right,
+        // Cap to whatever room is left between the anchor and the bottom
+        // of the viewport. The internal list takes overflow-y-auto.
+        maxHeight: pos.maxHeight,
+        display: 'flex',
+        flexDirection: 'column',
+        // Beat the bottom detail panel + StatsBar + any toolbar overlays
+        // — both live in sibling stacking contexts.
+        zIndex: 100,
+      }}
       onClick={(e) => e.stopPropagation()}
     >
-      <div className="flex items-center justify-between border-b border-surface-800 px-3 py-2">
+      <div
+        className="flex shrink-0 items-center justify-between border-b border-surface-800 px-3 py-2"
+        style={{ minHeight: headerHeight }}
+      >
         <div className="text-[12px] font-semibold text-surface-100">
           {t('urlsTab.columns', { defaultValue: 'Columns' })} ({visibleCount}/{allColumns.length})
         </div>
@@ -1123,27 +1635,55 @@ function ColumnPickerPopover({
           </button>
         </div>
       </div>
-      <div className="max-h-[60vh] overflow-y-auto py-1">
+      <div className="min-h-0 flex-1 overflow-y-auto py-1">
         {allColumns.map((c) => {
           const id = columnId(c);
           const visible = !hiddenColumns.has(id);
+          const isPinned = pinnedLeft.has(id);
           return (
-            <label
+            <div
               key={id}
-              className="flex cursor-pointer items-center gap-2 px-3 py-1 text-[11px] text-surface-200 hover:bg-surface-800"
+              className="flex items-center gap-2 px-3 py-1 text-[11px] text-surface-200 hover:bg-surface-800"
             >
-              <input
-                type="checkbox"
-                checked={visible}
-                onChange={() => toggle(id)}
-                className="h-3 w-3 accent-blue-500"
-              />
-              <span className="truncate">{translateLabel(c.header, lang)}</span>
-            </label>
+              <label className="flex flex-1 cursor-pointer items-center gap-2 truncate">
+                <input
+                  type="checkbox"
+                  checked={visible}
+                  onChange={() => toggle(id)}
+                  className="h-3 w-3 accent-blue-500"
+                />
+                <span className="truncate">{translateLabel(c.header, lang)}</span>
+              </label>
+              <button
+                type="button"
+                onClick={(e) => {
+                  e.stopPropagation();
+                  onTogglePin(id);
+                }}
+                className={clsx(
+                  'flex shrink-0 items-center rounded p-0.5 transition',
+                  isPinned
+                    ? 'text-accent-300 hover:bg-accent-500/20'
+                    : 'text-surface-600 hover:bg-surface-700 hover:text-surface-300',
+                )}
+                title={
+                  isPinned
+                    ? t('urlsTab.unpinColumn', { defaultValue: 'Unpin this column' })
+                    : t('urlsTab.pinColumn', {
+                        defaultValue: 'Pin this column to the left edge',
+                      })
+                }
+              >
+                <Pin
+                  className={clsx('h-3 w-3', isPinned && 'rotate-45 fill-current')}
+                />
+              </button>
+            </div>
           );
         })}
       </div>
-    </div>
+    </div>,
+    document.body,
   );
 }
 

apps/mcp-server/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/mcp-server",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "type": "module",
   "bin": {

apps/mcp-server/src/tools.ts

@@ -7,7 +7,11 @@
  */
 
 import type { ProjectDb } from '@freecrawl/db';
-import type { UrlCategory } from '@freecrawl/shared-types';
+import type {
+  AiProvider,
+  SeoProvider,
+  UrlCategory,
+} from '@freecrawl/shared-types';
 import { bridgeRequest } from './desktop-bridge.js';
 
 export interface Tool {
@@ -93,6 +97,32 @@ function projectFromRow(
   return out;
 }
 
+/**
+ * Every per-URL detail tool accepts either an `id` (preferred — cheaper
+ * lookup) or a `url` string and resolves to a numeric DB id. Throws when
+ * neither is supplied or the URL isn't crawled in the active project.
+ */
+function resolveUrlId(
+  args: Record<string, unknown>,
+  db: ProjectDb,
+): number {
+  if (typeof args.id === 'number') return args.id;
+  if (typeof args.url === 'string') {
+    const found = db.getUrlIdByUrl(args.url);
+    if (found === null) {
+      throw new Error(`URL not found in active project: ${args.url}`);
+    }
+    return found;
+  }
+  throw new Error('Tool requires either `id` (number) or `url` (string).');
+}
+
+/** Sanity-cap a numeric param without falling back to undefined. */
+function clamp(n: unknown, min: number, max: number, fallback: number): number {
+  const v = typeof n === 'number' && Number.isFinite(n) ? n : fallback;
+  return Math.max(min, Math.min(max, Math.floor(v)));
+}
+
 export function buildTools(): Tool[] {
   return [
     {
@@ -247,7 +277,7 @@ export function buildTools(): Tool[] {
       requiresDb: false,
       name: 'start_crawl',
       description:
-        'Start a crawl in the running FreeCrawl desktop app. The desktop app must be open — this drives the SAME crawler the UI uses, so progress shows up in the app as it runs. `startUrl` is required unless the user has already run a crawl in this session (the last-used start URL is reused). Other config overrides are optional and layered on top of the desktop\'s saved settings.',
+        'Start a crawl in the running FreeCrawl desktop app. The desktop app must be open — this drives the SAME crawler the UI uses, so progress shows up in the app as it runs. `startUrl` is required unless the user has already run a crawl in this session (the last-used start URL is reused). Other config overrides are optional and layered on top of the desktop\'s saved settings. NOTE: when the previous crawl with the same start URL already completed, the crawler treats this as a resume and exits immediately because every URL is already in the DB. To force a fresh BFS from the seed, call `clear_crawl` first.',
       inputSchema: {
         type: 'object',
         properties: {
@@ -319,6 +349,15 @@ export function buildTools(): Tool[] {
       handler: async () => bridgeRequest<{ ok: true }>('POST', '/v1/crawl/resume'),
     },
 
+    {
+      requiresDb: false,
+      name: 'clear_crawl',
+      description:
+        'Wipe every URL / link / image / header / cookie / sitemap entry from the active project — the same primitive the desktop "Clear" button calls. Stops any running crawl first. Use this before `start_crawl` when you want a fresh BFS from the seed and the previous crawl with that seed already completed (the crawler treats same-seed re-starts as a resume, so without clearing the new start would no-op). DESTRUCTIVE: there is no undo; the project file ends up empty until the new crawl populates it.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: async () => bridgeRequest<{ ok: true }>('POST', '/v1/crawl/clear'),
+    },
+
     {
       requiresDb: false,
       name: 'get_crawl_progress',
@@ -369,5 +408,564 @@ export function buildTools(): Tool[] {
         return { issues: ranked };
       },
     },
+
+    // =================================================================
+    // V2 Faz 0.5 Increment 1 — Per-URL detail sub-tab read tools.
+    //
+    // Most "sub-tabs" in the desktop UI are derivations of two stored
+    // primitives: the URL row (column-level aggregates like
+    // `schema_block_count`) and the raw page body (`url_sources.body`).
+    // The renderer parses the body client-side to surface Resources /
+    // Cookies / Structured Data / Outline. For MCP we expose the
+    // primitives — an LLM agent can reach into the same body to derive
+    // the rest, and we keep the wire surface compact.
+    // =================================================================
+
+    {
+      name: 'get_url_source',
+      description:
+        'Raw HTML body (and JS-rendered DOM when JS rendering was on) plus screenshot paths for one URL. Backs the renderer\'s View Source / View Rendered / Screenshot / Resources / Outline / Cookies / Structured Data sub-tabs — those are all derivations of this body. Truncated to 8 MB by the writer-worker; `truncated: true` flags when that cap fired.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer', description: 'URL id from query_urls (preferred).' },
+          url: { type: 'string', description: 'Exact URL string.' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        const source = db.getUrlSource(id);
+        if (!source) {
+          throw new Error(`No stored body for URL id ${id} — body capture may be disabled or this URL was not HTML.`);
+        }
+        return source;
+      },
+    },
+
+    {
+      name: 'get_url_inlinks',
+      description:
+        'Pages that link TO this URL — the reverse-link index. Per-row: from_url, anchor text, rel attribute, position (navigation/header/footer/sidebar/content), link kind (text/image/iframe/...). Useful for "who is linking to this 404?" or "what anchors point to my homepage?".',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+          limit: { type: 'integer', minimum: 1, maximum: 5000, description: 'Default 500.' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        const row = db.getUrlRowById(id);
+        if (!row) throw new Error(`URL id ${id} not found.`);
+        const limit = clamp(args.limit, 1, 5000, 500);
+        return db.getInlinks(row.url, limit);
+      },
+    },
+
+    {
+      name: 'get_url_outlinks',
+      description:
+        'Links FROM this page (both internal and external). Same per-row shape as get_url_inlinks but from the source-page perspective. Use this to audit a page\'s outbound link strategy or find broken outlinks.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+          limit: { type: 'integer', minimum: 1, maximum: 5000, description: 'Default 500.' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        const limit = clamp(args.limit, 1, 5000, 500);
+        return db.getOutlinks(id, limit);
+      },
+    },
+
+    {
+      name: 'get_url_images',
+      description:
+        'Every <img> referenced by this page — src, alt, intrinsic width/height, byte_size (from the post-crawl HEAD probe), is_internal flag. Use to audit per-page image weight or missing-alt distribution.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+          limit: { type: 'integer', minimum: 1, maximum: 10_000, description: 'Default 5000.' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        const limit = clamp(args.limit, 1, 10_000, 5000);
+        return { rows: db.pageImagesDetailed(id, limit) };
+      },
+    },
+
+    {
+      name: 'get_url_headers',
+      description:
+        'Every response header captured for this URL. Same data the Detail panel\'s "HTTP Headers" sub-tab shows. Sorted alphabetically; multi-valued headers appear as separate rows.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        return { rows: db.getUrlHeaders(id) };
+      },
+    },
+
+    {
+      name: 'get_url_duplicates',
+      description:
+        'Other URLs in the same near-duplicate SimHash cluster as the queried URL, sorted by Hamming distance ascending (closest matches first). Empty result means the URL is a singleton OR Crawl Analysis has not run yet — run it from the desktop first.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        return { rows: db.urlClusterMembers(id) };
+      },
+    },
+
+    {
+      name: 'get_url_analytics',
+      description:
+        'GSC + GA4 + PageSpeed + AI roll-up for one URL: clicks/impressions/CTR/position (GSC), sessions/users/bounceRate (GA4), lab scores + Core Web Vitals (PageSpeed), provider-specific AI verdict. Returns null fields for integrations that have not been fetched yet.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        const row = db.getUrlRowById(id);
+        if (!row) throw new Error(`URL id ${id} not found.`);
+        return db.getUrlAnalytics(row.url);
+      },
+    },
+
+    {
+      name: 'get_url_cert',
+      description:
+        'TLS certificate details for the host of the given URL (from the per-host cert probe). Issuer, subject, validity dates, days-until-expiry, signature algorithm, chain length. Returns null when the host was not probed (HTTP-only or probe failed).',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          id: { type: 'integer' },
+          url: { type: 'string' },
+        },
+      },
+      handler: (args, db) => {
+        const id = resolveUrlId(args, db);
+        return db.getHostCertForUrl(id);
+      },
+    },
+
+    // =================================================================
+    // V2 Faz 0.5 Increment 1 — Integration row query tools. Each mirrors
+    // a desktop tab (Search Console / GA4 / PageSpeed / AI / SEO
+    // Authority). Pagination follows the same shape as query_urls so
+    // an agent can scan large result sets in batches.
+    // =================================================================
+
+    {
+      name: 'query_gsc',
+      description:
+        'Search Console rows: clicks, impressions, CTR, average position per crawled page. Returns null per-URL when the URL has not been pulled from GSC yet (configure the integration in Settings → Integrations → Search Console).',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          search: { type: 'string' },
+          filter: {
+            type: 'string',
+            enum: ['all', 'with-data', 'without-data'],
+            description: '`with-data` = pages that have GSC rows; `without-data` = pages NOT yet pulled from GSC. Default "all".',
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 1000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) =>
+        db.queryGsc({
+          limit: clamp(args.limit, 1, 1000, 50),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          filter: args.filter as 'all' | 'with-data' | 'without-data' | undefined,
+        }),
+    },
+
+    {
+      name: 'query_ga4',
+      description:
+        'GA4 rows: sessions, users, bounceRate, engagementRate per crawled page. Configure the integration in Settings → Integrations → GA4 to populate.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          search: { type: 'string' },
+          filter: {
+            type: 'string',
+            enum: ['all', 'with-data', 'without-data'],
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 1000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) =>
+        db.queryGa4({
+          limit: clamp(args.limit, 1, 1000, 50),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          filter: args.filter as 'all' | 'with-data' | 'without-data' | undefined,
+        }),
+    },
+
+    {
+      name: 'query_pagespeed',
+      description:
+        'PageSpeed Insights rows: lab perf/SEO/a11y scores, Core Web Vitals (LCP/CLS/INP/FCP/TTFB), opportunities. Populate via Settings → Integrations → PageSpeed Insights, then run a batch from the desktop.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          search: { type: 'string' },
+          filter: {
+            type: 'string',
+            enum: ['all', 'tested', 'untested'],
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 1000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) =>
+        db.queryPagespeed({
+          limit: clamp(args.limit, 1, 1000, 50),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          filter: args.filter as 'all' | 'tested' | 'untested' | undefined,
+        }),
+    },
+
+    {
+      name: 'query_ai',
+      description:
+        'AI batch results per URL for a single provider (openai / anthropic / ollama). Configure provider + run a batch from Settings → AI to populate.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          provider: {
+            type: 'string',
+            enum: ['openai', 'anthropic', 'ollama'],
+          },
+          search: { type: 'string' },
+          filter: {
+            type: 'string',
+            enum: ['all', 'with-data', 'without-data', 'error'],
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 1000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+        required: ['provider'],
+      },
+      handler: (args, db) =>
+        db.queryAi({
+          provider: args.provider as AiProvider,
+          limit: clamp(args.limit, 1, 1000, 50),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          filter: args.filter as 'all' | 'with-data' | 'without-data' | 'error' | undefined,
+        }),
+    },
+
+    {
+      name: 'query_seo',
+      description:
+        'SEO Authority rows from one provider (moz / ahrefs / semrush etc). Per-URL domain authority, backlink count, referring domains.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          provider: {
+            type: 'string',
+            description: 'SEO authority provider — must be configured in Settings → SEO Authority.',
+          },
+          search: { type: 'string' },
+          filter: {
+            type: 'string',
+            enum: ['all', 'with-data', 'without-data', 'error'],
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 1000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+        required: ['provider'],
+      },
+      handler: (args, db) =>
+        db.querySeo({
+          provider: args.provider as SeoProvider,
+          limit: clamp(args.limit, 1, 1000, 50),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          filter: args.filter as 'all' | 'with-data' | 'without-data' | 'error' | undefined,
+        }),
+    },
+
+    // =================================================================
+    // V2 Faz 0.5 Increment 1 — Specialised entity queries (images,
+    // broken links, duplicate clusters) and reports.
+    // =================================================================
+
+    {
+      name: 'query_images',
+      description:
+        'Project-wide image catalogue — every <img> the crawler saw, with src, alt, intrinsic dimensions, byte_size, is_internal. Backs the Images tab.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          search: {
+            type: 'string',
+            description: 'Substring match against src or alt.',
+          },
+          missingAltOnly: { type: 'boolean' },
+          internalOnly: { type: 'boolean' },
+          limit: { type: 'integer', minimum: 1, maximum: 5000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) =>
+        db.queryImages({
+          limit: clamp(args.limit, 1, 5000, 100),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          missingAltOnly: args.missingAltOnly === true,
+          internalOnly: args.internalOnly === true,
+        }),
+    },
+
+    {
+      name: 'query_broken_links',
+      description:
+        'Every 4xx/5xx link the crawler followed — from_url + to_url + anchor + status_code + is_internal. Use to triage broken-link debt site-wide.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          scope: {
+            type: 'string',
+            enum: ['all', 'internal', 'external'],
+            description: 'Default "all".',
+          },
+          search: { type: 'string' },
+          limit: { type: 'integer', minimum: 1, maximum: 5000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) =>
+        db.queryBrokenLinks({
+          limit: clamp(args.limit, 1, 5000, 100),
+          offset: clamp(args.offset, 0, 10_000_000, 0),
+          search: typeof args.search === 'string' ? args.search : undefined,
+          internal:
+            (args.scope as 'all' | 'internal' | 'external' | undefined) ?? 'all',
+        }),
+    },
+
+    {
+      name: 'list_duplicate_clusters',
+      description:
+        'Paginated view of near-duplicate clusters with each cluster\'s members — same data the Cluster view on the Duplicates tab renders. Members are pre-grouped: rows sharing a cluster_id form one cluster, sorted by cluster_size DESC then alphabetic. Hamming distance from the cluster representative is included per row.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          limit: { type: 'integer', minimum: 1, maximum: 5000 },
+          offset: { type: 'integer', minimum: 0 },
+        },
+      },
+      handler: (args, db) => {
+        const limit = clamp(args.limit, 1, 5000, 500);
+        const offset = clamp(args.offset, 0, 10_000_000, 0);
+        return {
+          rows: db.listDuplicateClusters(offset, limit),
+          total: db.countDuplicateClusterMembers(),
+          limit,
+          offset,
+        };
+      },
+    },
+
+    // ----- Reports (no per-URL data — aggregates the desktop Reports
+    // menu exposes). Each maps 1:1 to a method on ProjectDb.
+    {
+      name: 'report_status_code_histogram',
+      description: 'Per-status-code URL count distribution (200/301/404/500/...). Useful baseline for a crawl health snapshot.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.getStatusCodeHistogram() }),
+    },
+    {
+      name: 'report_indexability_distribution',
+      description: 'Indexability breakdown — indexable vs noindex / canonicalised / blocked / redirect / 4xx / 5xx.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.getIndexabilityDistribution() }),
+    },
+    {
+      name: 'report_content_kind_distribution',
+      description: 'URL count by content kind (html / js / css / image / pdf / font / other).',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.getContentKindDistribution() }),
+    },
+    {
+      name: 'report_depth_histogram',
+      description: 'Per-depth URL count — depth = shortest BFS path length from the seed.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.getDepthHistogram() }),
+    },
+    {
+      name: 'report_response_time_histogram',
+      description: 'Response-time bucket histogram (0–100 ms, 100–500 ms, 500 ms–1 s, 1–3 s, 3 s+).',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.getResponseTimeHistogram() }),
+    },
+    {
+      name: 'report_inlinks_histogram',
+      description: 'How many pages have N inlinks — surface orphan / weakly-linked / hub pages at a glance.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.inlinksHistogram() }),
+    },
+    {
+      name: 'report_word_count_histogram',
+      description: 'Word-count distribution across indexable HTML pages.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.wordCountHistogram() }),
+    },
+    {
+      name: 'report_url_length_histogram',
+      description: 'URL-length distribution (≤75 / 76–115 / 116–200 / 201–500 / 501–2048 / >2048 chars).',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.urlLengthHistogram() }),
+    },
+    {
+      name: 'report_top_urls_by',
+      description: 'Top-N URLs by a numeric column (response_time_ms / depth / outlinks / inlinks / content_length). Direction defaults to desc; asc excludes zero-value rows so you get the weakest non-orphans not the dead links.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          column: {
+            type: 'string',
+            enum: ['response_time_ms', 'depth', 'outlinks', 'inlinks', 'content_length'],
+          },
+          limit: { type: 'integer', minimum: 1, maximum: 500 },
+          direction: { type: 'string', enum: ['asc', 'desc'] },
+        },
+        required: ['column'],
+      },
+      handler: (args, db) => ({
+        rows: db.topUrlsBy(
+          args.column as 'response_time_ms' | 'depth' | 'outlinks' | 'inlinks' | 'content_length',
+          clamp(args.limit, 1, 500, 25),
+          (args.direction as 'asc' | 'desc' | undefined) ?? 'desc',
+        ),
+      }),
+    },
+    {
+      name: 'report_top_anchor_texts',
+      description: 'Most-used internal anchor texts across the whole site. Heuristic for "what does the site call itself in its own links".',
+      inputSchema: {
+        type: 'object',
+        properties: { limit: { type: 'integer', minimum: 1, maximum: 1000 } },
+      },
+      handler: (args, db) => ({
+        rows: db.topAnchorTexts(clamp(args.limit, 1, 1000, 200)),
+      }),
+    },
+    {
+      name: 'report_external_domain_health',
+      description: 'External-link health per outbound domain — total / success / errors / avg response time / error-rate percent. Sorted by errorCount DESC so worst offenders top.',
+      inputSchema: {
+        type: 'object',
+        properties: { limit: { type: 'integer', minimum: 1, maximum: 1000 } },
+      },
+      handler: (args, db) => ({
+        rows: db.externalDomainHealth(clamp(args.limit, 1, 1000, 100)),
+      }),
+    },
+    {
+      name: 'report_image_weight_per_page',
+      description: 'Per-page image weight roll-up — total bytes of images on the page + image count. Top-N by total weight DESC.',
+      inputSchema: {
+        type: 'object',
+        properties: { limit: { type: 'integer', minimum: 1, maximum: 500 } },
+      },
+      handler: (args, db) => ({
+        rows: db.imageWeightPerPage(clamp(args.limit, 1, 500, 25)),
+      }),
+    },
+    {
+      name: 'report_link_position_breakdown',
+      description: 'Where internal links live — navigation / header / footer / sidebar / content. Mirrors the "Internal Link Positions" report.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.linkPositionBreakdown() }),
+    },
+    {
+      name: 'report_pages_per_directory',
+      description: 'Per-directory page count + share. `depth` controls how many path segments to bucket by (1–4).',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          depth: { type: 'integer', minimum: 1, maximum: 4 },
+          limit: { type: 'integer', minimum: 1, maximum: 5000 },
+        },
+      },
+      handler: (args, db) => ({
+        rows: db.getPagesPerDirectory({
+          depth: clamp(args.depth, 1, 4, 1),
+          limit: clamp(args.limit, 1, 5000, 1000),
+        }),
+      }),
+    },
+    {
+      name: 'report_word_count_per_directory',
+      description: 'Average word count + page count per directory. Same `depth` semantics as report_pages_per_directory. Surfaces thin-content directories.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          depth: { type: 'integer', minimum: 1, maximum: 4 },
+          limit: { type: 'integer', minimum: 1, maximum: 2000 },
+        },
+      },
+      handler: (args, db) => ({
+        rows: db.wordCountPerDirectory({
+          depth: clamp(args.depth, 1, 4, 1),
+          limit: clamp(args.limit, 1, 2000, 500),
+        }),
+      }),
+    },
+    {
+      name: 'report_sitemap_orphans',
+      description: 'URLs declared in the sitemap but never reached by the crawl. Each row carries lastmod + sourceSitemap so you can tell stale entries from genuine orphans.',
+      inputSchema: {
+        type: 'object',
+        properties: { limit: { type: 'integer', minimum: 1, maximum: 10_000 } },
+      },
+      handler: (args, db) => ({
+        rows: db.sitemapOrphans(clamp(args.limit, 1, 10_000, 1000)),
+      }),
+    },
+    {
+      name: 'report_analytics_coverage',
+      description: 'Which third-party analytics trackers (GA4 / GTM / Hotjar / Clarity / Mixpanel / Plausible / ...) appear on which fraction of indexable pages. Reveals coverage gaps and ID inconsistencies (multiple GA4 properties tagged across the site, etc).',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.analyticsCoverage() }),
+    },
+    {
+      name: 'report_server_header_breakdown',
+      description: 'Distinct `Server` response header values and how many URLs each. Surfaces mixed-stack deployments and CDN attribution.',
+      inputSchema: { type: 'object', properties: {} },
+      handler: (_args, db) => ({ rows: db.serverHeaderBreakdown() }),
+    },
   ];
 }

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "freecrawl-seo",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "freecrawl-seo",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "hasInstallScript": true,
       "license": "MIT",
       "workspaces": [
@@ -26,7 +26,7 @@
     },
     "apps/cli": {
       "name": "@freecrawl/cli",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "dependencies": {
         "@freecrawl/core": "*",
         "@freecrawl/db": "*",
@@ -43,8 +43,11 @@
     },
     "apps/desktop": {
       "name": "@freecrawl/desktop",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "dependencies": {
+        "@dnd-kit/core": "^6.3.1",
+        "@dnd-kit/sortable": "^10.0.0",
+        "@dnd-kit/utilities": "^3.2.2",
         "@freecrawl/core": "*",
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*",
@@ -87,7 +90,7 @@
     },
     "apps/mcp-server": {
       "name": "@freecrawl/mcp-server",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*"
@@ -460,6 +463,60 @@
         "url": "https://opencollective.com/webpack"
       }
     },
+    "node_modules/@dnd-kit/accessibility": {
+      "version": "3.1.1",
+      "resolved": "https://registry.npmjs.org/@dnd-kit/accessibility/-/accessibility-3.1.1.tgz",
+      "integrity": "sha512-2P+YgaXF+gRsIihwwY1gCsQSYnu9Zyj2py8kY5fFvUM1qm2WA2u639R6YNVfU4GWr+ZM5mqEsfHZZLoRONbemw==",
+      "license": "MIT",
+      "dependencies": {
+        "tslib": "^2.0.0"
+      },
+      "peerDependencies": {
+        "react": ">=16.8.0"
+      }
+    },
+    "node_modules/@dnd-kit/core": {
+      "version": "6.3.1",
+      "resolved": "https://registry.npmjs.org/@dnd-kit/core/-/core-6.3.1.tgz",
+      "integrity": "sha512-xkGBRQQab4RLwgXxoqETICr6S5JlogafbhNsidmrkVv2YRs5MLwpjoF2qpiGjQt8S9AoxtIV603s0GIUpY5eYQ==",
+      "license": "MIT",
+      "peer": true,
+      "dependencies": {
+        "@dnd-kit/accessibility": "^3.1.1",
+        "@dnd-kit/utilities": "^3.2.2",
+        "tslib": "^2.0.0"
+      },
+      "peerDependencies": {
+        "react": ">=16.8.0",
+        "react-dom": ">=16.8.0"
+      }
+    },
+    "node_modules/@dnd-kit/sortable": {
+      "version": "10.0.0",
+      "resolved": "https://registry.npmjs.org/@dnd-kit/sortable/-/sortable-10.0.0.tgz",
+      "integrity": "sha512-+xqhmIIzvAYMGfBYYnbKuNicfSsk4RksY2XdmJhT+HAC01nix6fHCztU68jooFiMUB01Ky3F0FyOvhG/BZrWkg==",
+      "license": "MIT",
+      "dependencies": {
+        "@dnd-kit/utilities": "^3.2.2",
+        "tslib": "^2.0.0"
+      },
+      "peerDependencies": {
+        "@dnd-kit/core": "^6.3.0",
+        "react": ">=16.8.0"
+      }
+    },
+    "node_modules/@dnd-kit/utilities": {
+      "version": "3.2.2",
+      "resolved": "https://registry.npmjs.org/@dnd-kit/utilities/-/utilities-3.2.2.tgz",
+      "integrity": "sha512-+MKAJEOfaBe5SmV6t34p80MMKhjvUz0vRrvVJbPT0WElzaOJ/1xs+D+KDv+tD/NE5ujfrChEcshd4fLn0wpiqg==",
+      "license": "MIT",
+      "dependencies": {
+        "tslib": "^2.0.0"
+      },
+      "peerDependencies": {
+        "react": ">=16.8.0"
+      }
+    },
     "node_modules/@electron/asar": {
       "version": "3.4.1",
       "resolved": "https://registry.npmjs.org/@electron/asar/-/asar-3.4.1.tgz",
@@ -7946,6 +8003,12 @@
       "dev": true,
       "license": "Apache-2.0"
     },
+    "node_modules/tslib": {
+      "version": "2.8.1",
+      "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
+      "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
+      "license": "0BSD"
+    },
     "node_modules/type-check": {
       "version": "0.4.0",
       "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz",
@@ -8871,7 +8934,7 @@
     },
     "packages/core": {
       "name": "@freecrawl/core",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "dependencies": {
         "@freecrawl/db": "*",
         "@freecrawl/shared-types": "*",
@@ -8898,7 +8961,7 @@
     },
     "packages/db": {
       "name": "@freecrawl/db",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "dependencies": {
         "@freecrawl/shared-types": "*"
       },
@@ -8910,7 +8973,7 @@
     },
     "packages/shared-types": {
       "name": "@freecrawl/shared-types",
-      "version": "0.6.5",
+      "version": "0.6.6",
       "devDependencies": {
         "rimraf": "^6.0.1",
         "typescript": "^5.7.0"

package.json

@@ -1,6 +1,6 @@
 {
   "name": "freecrawl-seo",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "description": "Open-source desktop SEO crawler — a free alternative to Screaming Frog",
   "license": "MIT",

packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/core",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",

packages/core/src/crawler.ts

@@ -2326,6 +2326,10 @@ export class Crawler extends EventEmitter {
           hreflangs: parsed.hreflangs.length > 0 ? JSON.stringify(parsed.hreflangs) : null,
           hreflangCount: parsed.hreflangs.length,
           amphtml: parsed.amphtml,
+          ampPage: parsed.ampPage,
+          ampValidationErrors: parsed.ampValidationErrors.length > 0
+            ? JSON.stringify(parsed.ampValidationErrors)
+            : null,
           favicon: parsed.favicon,
           appleTouchIcon: parsed.appleTouchIcon,
           androidIcon: parsed.androidIcon,

packages/core/src/extraction.ts

@@ -1,3 +1,4 @@
+import * as cheerio from 'cheerio';
 import type { CheerioAPI } from 'cheerio';
 import type { CustomExtractionRule } from '@freecrawl/shared-types';
 
@@ -54,6 +55,44 @@ export function runExtractionRules(
   return anyMatch ? out : null;
 }
 
+/**
+ * Live-preview variant of {@link runExtractionRules} — runs every rule
+ * against a single page and returns per-rule outcomes WITHOUT
+ * swallowing errors. The crawler-side variant silently nulls failed
+ * rules so a misconfigured selector doesn't poison the whole map; the
+ * preview UI needs the opposite contract because the whole point of
+ * the dialog is to surface broken selectors.
+ *
+ * Loads cheerio internally so the caller doesn't need its own parser.
+ * Same per-rule semantics for CSS vs regex; `null` value distinguishes
+ * "no match" while a populated `error` distinguishes "rule failed to
+ * compile / execute".
+ */
+export function previewExtractionRules(
+  rawHtml: string,
+  rules: ReadonlyArray<CustomExtractionRule>,
+): Array<{ name: string; value: unknown; error?: string }> {
+  if (rules.length === 0) return [];
+  const $ = cheerio.load(rawHtml);
+  const out: Array<{ name: string; value: unknown; error?: string }> = [];
+  for (const rule of rules) {
+    const name = (rule.name ?? '').trim() || '(unnamed)';
+    try {
+      let value: unknown = null;
+      if (rule.type === 'css') {
+        value = runCssRule($, rule);
+      } else if (rule.type === 'regex') {
+        value = runRegexRule(rawHtml, rule);
+      }
+      out.push({ name, value });
+    } catch (e) {
+      const msg = e instanceof Error ? e.message : String(e);
+      out.push({ name, value: null, error: msg });
+    }
+  }
+  return out;
+}
+
 function runCssRule($: CheerioAPI, rule: CustomExtractionRule): unknown {
   const $els = $(rule.selector);
   const total = $els.length;

packages/core/src/html-parser.ts

@@ -124,6 +124,15 @@ export interface ParsedPage {
   hreflangs: HreflangEntry[];
   /** `<link rel="amphtml" href>` if present, else null. */
   amphtml: string | null;
+  /** True when the page declares itself AMP via `<html ⚡>` /
+   *  `<html amp>` (also covers ⚡4ads / ⚡4email variants). */
+  ampPage: boolean;
+  /** Short, machine-readable error codes from the AMP smoke validator.
+   *  Empty when the page isn't AMP or when it passes every check.
+   *  Codes are stable identifiers (e.g. `missing-charset-meta`,
+   *  `forbidden-style-tag`) so the UI can localise the human-readable
+   *  copy without coupling to the parser. */
+  ampValidationErrors: string[];
   /** Resolved favicon URL from `<link rel="icon">` / `shortcut icon`, else null. */
   favicon: string | null;
   /** Resolved `<link rel="apple-touch-icon">` URL, else null. */
@@ -624,6 +633,17 @@ export function parseHtml(
   const amphtmlRaw = ($('link[rel="amphtml"]').attr('href') ?? '').trim();
   const amphtml = amphtmlRaw ? normalizeUrl(amphtmlRaw, pageUrl, opts.urlRewrites) : null;
 
+  // AMP smoke validation — pages that declare themselves AMP via
+  // `<html ⚡>` / `<html amp>` get a hand-rolled subset of Google's
+  // validator (boilerplate, runtime, charset, viewport, canonical,
+  // forbidden script/style/legacy tags). The full validator is a
+  // 10+ MB bundle from cdn.ampproject.org which is wildly out of
+  // proportion for a desktop tool — these checks catch the most
+  // common AMP authoring mistakes (forgotten boilerplate, missing
+  // runtime, third-party tracker scripts).
+  const { isAmpPage: ampPage, errors: ampValidationErrors } =
+    detectAndValidateAmp($);
+
   // Favicon — prefer modern `rel="icon"`, fall back to legacy
   // `rel="shortcut icon"`. We don't fabricate a default `/favicon.ico`;
   // only what the page actually declares.
@@ -1252,6 +1272,8 @@ export function parseHtml(
     paginationPrev,
     hreflangs,
     amphtml,
+    ampPage,
+    ampValidationErrors,
     favicon,
     appleTouchIcon,
     androidIcon,
@@ -1748,6 +1770,134 @@ function buildLinkPath(el: unknown): string | null {
   return parts.slice(-8).join(' > ');
 }
 
+/**
+ * Detect whether the page declares itself AMP and, if so, run the
+ * smoke-validation subset of Google's spec. Catches the most common
+ * authoring mistakes (forgotten boilerplate / runtime / canonical,
+ * forbidden inline scripts or style blocks, banned legacy tags) without
+ * pulling in the 10+ MB official validator.js bundle.
+ *
+ * AMP marker variants checked:
+ *  - `<html amp>` — the ASCII spelling
+ *  - `<html ⚡>` — the official lightning-bolt spelling
+ *  - `<html ⚡4ads>` / `<html ⚡4email>` — AMP-for-Ads / AMP-for-Email
+ *
+ * Returns short stable error codes so the UI can localise the
+ * human-readable copy without coupling to the parser.
+ */
+function detectAndValidateAmp(
+  $: ReturnType<typeof cheerio.load>,
+): { isAmpPage: boolean; errors: string[] } {
+  const htmlEl = $('html').first();
+  if (htmlEl.length === 0) return { isAmpPage: false, errors: [] };
+  const attribs =
+    ((htmlEl[0] as { attribs?: Record<string, string> } | undefined)
+      ?.attribs ?? {});
+  const attribKeys = Object.keys(attribs);
+  const isAmpPage =
+    'amp' in attribs ||
+    attribKeys.some((k) => k === '⚡' || k.startsWith('⚡'));
+  if (!isAmpPage) return { isAmpPage: false, errors: [] };
+
+  const errors: string[] = [];
+
+  // 1. <meta charset="utf-8"> required (any case for utf-8 / UTF-8).
+  const charsetMeta = $('head meta[charset]').first();
+  if (charsetMeta.length === 0) {
+    errors.push('missing-charset-meta');
+  } else if ((charsetMeta.attr('charset') ?? '').toLowerCase() !== 'utf-8') {
+    errors.push('non-utf8-charset');
+  }
+
+  // 2. <meta name="viewport"> with width=device-width.
+  const viewportMeta = $('head meta[name="viewport"]').first();
+  if (viewportMeta.length === 0) {
+    errors.push('missing-viewport');
+  } else {
+    const content = (viewportMeta.attr('content') ?? '').toLowerCase();
+    if (!content.includes('width=device-width')) {
+      errors.push('viewport-missing-device-width');
+    }
+  }
+
+  // 3. <link rel="canonical"> required.
+  if ($('head link[rel="canonical"][href]').length === 0) {
+    errors.push('missing-canonical');
+  }
+
+  // 4. AMP runtime — async script from cdn.ampproject.org.
+  const runtime = $('head script[async]').filter((_, el) => {
+    const src = ($(el).attr('src') ?? '').trim();
+    return (
+      src.startsWith('https://cdn.ampproject.org/v0.js') ||
+      src.startsWith('https://cdn.ampproject.org/v0.mjs') ||
+      src.startsWith('https://ampjs.org/v0.js')
+    );
+  });
+  if (runtime.length === 0) {
+    errors.push('missing-amp-runtime');
+  }
+
+  // 5. AMP boilerplate — `<style amp-boilerplate>` + matching
+  // `<noscript><style amp-boilerplate>`. Both halves are required by
+  // the spec; missing either makes the validator fail.
+  if ($('head style[amp-boilerplate]').length === 0) {
+    errors.push('missing-boilerplate-style');
+  }
+  if ($('head noscript style[amp-boilerplate]').length === 0) {
+    errors.push('missing-noscript-boilerplate');
+  }
+
+  // 6. Forbidden <script> tags. Allow-list:
+  //  - JSON-LD / generic JSON / amp-mustache templates (type-based)
+  //  - Async + src points to cdn.ampproject.org / ampjs.org
+  // `function () { $(this) }` form sidesteps cheerio's tricky generic
+  // `.each((i, el) => ...)` inference when `$` arrives as a parameter
+  // (vs a `cheerio.load()` local elsewhere in this file).
+  let forbiddenScript = false;
+  $('script').each(function (this: unknown) {
+    if (forbiddenScript) return;
+    const $el = $(this as Parameters<typeof $>[0]);
+    const type = ($el.attr('type') ?? '').toLowerCase();
+    if (
+      type === 'application/ld+json' ||
+      type === 'application/json' ||
+      type === 'text/template'
+    ) {
+      return;
+    }
+    const src = ($el.attr('src') ?? '').trim();
+    const isAsync = $el.is('[async]');
+    if (
+      isAsync &&
+      (src.startsWith('https://cdn.ampproject.org/') ||
+        src.startsWith('https://ampjs.org/'))
+    ) {
+      return;
+    }
+    forbiddenScript = true;
+  });
+  if (forbiddenScript) errors.push('forbidden-script-tag');
+
+  // 7. Forbidden <style> tags. Allow-list: amp-boilerplate + amp-custom.
+  let forbiddenStyle = false;
+  $('style').each(function (this: unknown) {
+    if (forbiddenStyle) return;
+    const $el = $(this as Parameters<typeof $>[0]);
+    if ($el.is('[amp-boilerplate]') || $el.is('[amp-custom]')) return;
+    forbiddenStyle = true;
+  });
+  if (forbiddenStyle) errors.push('forbidden-style-tag');
+
+  // 8. Legacy / forbidden tags banned by the AMP spec.
+  if ($('base').length > 0) errors.push('forbidden-base-tag');
+  if ($('frame, frameset, embed, object, iframe[src*="javascript:"]').length > 0) {
+    errors.push('forbidden-legacy-tag');
+  }
+
+  return { isAmpPage: true, errors: [...new Set(errors)] };
+}
+
 /**
  * Infer the page region a link lives in based on its ancestor landmark
  * elements. Walks up the parent chain and returns the first match.

packages/core/src/html-report.ts

@@ -126,6 +126,7 @@ const ISSUES: IssueDef[] = [
   { key: 'hreflangTargetIssues', label: 'Hreflang Target Issues', severity: 'error' },
   { key: 'faviconMissing', label: 'Favicon Missing', severity: 'info' },
   { key: 'charsetMissing', label: 'Charset Missing', severity: 'warn' },
+  { key: 'ampValidationErrors', label: 'AMP Validation Errors', severity: 'warn' },
   { key: 'nonIndexableInSitemap', label: 'Non-Indexable in Sitemap', severity: 'error' },
   { key: 'non200InSitemap', label: 'Non-200 in Sitemap', severity: 'error' },
   { key: 'redirectInSitemap', label: 'Redirect in Sitemap', severity: 'warn' },

packages/core/src/index.ts

@@ -60,6 +60,7 @@ export {
   type SitemapVariant,
   type SitemapExportResult,
 } from './sitemap-export.js';
+export { previewExtractionRules } from './extraction.js';
 export { exportHtmlReport, type HtmlReportOptions } from './html-report.js';
 export {
   analyseCookies,

packages/db/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/db",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",

packages/db/src/migrations.ts

@@ -1499,6 +1499,33 @@ const MIGRATIONS: Migration[] = [
       }
     },
   },
+  {
+    version: 64,
+    name: 'add_amp_validation_columns',
+    // V2 Faz 16 — AMP smoke validation. Pages that declare themselves
+    // AMP via `<html ⚡>` / `<html amp>` are validated against a
+    // hand-rolled subset of the AMP spec (boilerplate, runtime, charset,
+    // viewport, canonical, forbidden tags) — full spec needs Google's
+    // 10+ MB validator.js which is overkill for a desktop tool.
+    //
+    // `amp_page` is 0 when the page isn't AMP and 1 when the `<html>`
+    // tag carries the AMP marker. `amp_validation_errors` is a JSON
+    // array of short error codes (e.g. `["missing-boilerplate",
+    // "forbidden-style-tag"]`); empty array means clean AMP page.
+    up: (db) => {
+      const cols = db
+        .prepare('PRAGMA table_info(urls)')
+        .all() as unknown as { name: string }[];
+      if (!cols.some((c) => c.name === 'amp_page')) {
+        db.exec(
+          'ALTER TABLE urls ADD COLUMN amp_page INTEGER NOT NULL DEFAULT 0',
+        );
+      }
+      if (!cols.some((c) => c.name === 'amp_validation_errors')) {
+        db.exec('ALTER TABLE urls ADD COLUMN amp_validation_errors TEXT');
+      }
+    },
+  },
 ];
 
 export function runMigrations(db: DatabaseSync): void {

packages/db/src/project-db.ts

@@ -96,6 +96,11 @@ interface UrlRowDb {
   hreflangs: string | null;
   hreflang_count: number;
   amphtml: string | null;
+  /** 1 when the page declares `<html ⚡>` / `<html amp>`, else 0. */
+  amp_page: number;
+  /** JSON string array of AMP smoke-validator error codes. Empty when
+   *  not an AMP page or when validation passes. */
+  amp_validation_errors: string | null;
   favicon: string | null;
   mixed_content_count: number;
   mixed_content_active: number;
@@ -257,6 +262,9 @@ export interface UpsertUrlInput {
   hreflangs?: string | null;
   hreflangCount?: number;
   amphtml?: string | null;
+  ampPage?: boolean;
+  /** JSON-stringified array of AMP validator error codes. */
+  ampValidationErrors?: string | null;
   favicon?: string | null;
   mixedContentCount?: number;
   /** Active mixed content (blocked by browser) — script/iframe/object/embed/stylesheet over HTTP. */
@@ -379,7 +387,8 @@ const UPSERT_URL_SQL = `
     url_malformed, og_type, og_url, og_site_name, og_locale, android_icon,
     landmark_main, skip_link_present, aria_invalid_roles,
     schema_duplicate_ids, schema_unknown_types, schema_missing_required,
-    heading_order_violations, subresource_request_count
+    heading_order_violations, subresource_request_count,
+    amp_page, amp_validation_errors
   ) VALUES (
     :url, :content_kind, :status_code, :status_text, :indexability, :indexability_reason,
     :title, :title_length, :meta_description, :meta_description_length,
@@ -418,7 +427,8 @@ const UPSERT_URL_SQL = `
     :url_malformed, :og_type, :og_url, :og_site_name, :og_locale, :android_icon,
     :landmark_main, :skip_link_present, :aria_invalid_roles,
     :schema_duplicate_ids, :schema_unknown_types, :schema_missing_required,
-    :heading_order_violations, :subresource_request_count
+    :heading_order_violations, :subresource_request_count,
+    :amp_page, :amp_validation_errors
   )
   ON CONFLICT(url) DO UPDATE SET
     content_kind = excluded.content_kind,
@@ -547,6 +557,8 @@ const UPSERT_URL_SQL = `
     schema_missing_required = excluded.schema_missing_required,
     heading_order_violations = excluded.heading_order_violations,
     subresource_request_count = excluded.subresource_request_count,
+    amp_page = excluded.amp_page,
+    amp_validation_errors = excluded.amp_validation_errors,
     crawled_at = CURRENT_TIMESTAMP
   RETURNING id
 `;
@@ -1282,6 +1294,8 @@ export class ProjectDb {
       hreflangs: input.hreflangs ?? null,
       hreflang_count: input.hreflangCount ?? 0,
       amphtml: input.amphtml ?? null,
+      amp_page: input.ampPage ? 1 : 0,
+      amp_validation_errors: input.ampValidationErrors ?? null,
       favicon: input.favicon ?? null,
       mixed_content_count: input.mixedContentCount ?? 0,
       mixed_content_active: input.mixedContentActive ?? 0,
@@ -2001,7 +2015,7 @@ export class ProjectDb {
   listDuplicateClusters(offset: number, limit: number): DuplicateClusterRow[] {
     const rows = this.db
       .prepare(
-        `SELECT u.url, u.status_code, u.indexability, u.title, u.word_count,
+        `SELECT u.id, u.url, u.status_code, u.indexability, u.title, u.word_count,
                 u.inlinks, u.cluster_id, u.cluster_size, u.simhash
            FROM urls u
           WHERE u.is_external = 0 AND u.content_kind = 'html'
@@ -2010,6 +2024,7 @@ export class ProjectDb {
           LIMIT ? OFFSET ?`,
       )
       .all(limit, offset) as {
+      id: number;
       url: string;
       status_code: number | null;
       indexability: Indexability;
@@ -2059,6 +2074,7 @@ export class ProjectDb {
         hammingFromRep = popcount(BigInt('0x' + repHash) ^ BigInt('0x' + r.simhash));
       }
       return {
+        urlId: r.id,
         url: r.url,
         statusCode: r.status_code,
         indexability: r.indexability,
@@ -2073,6 +2089,7 @@ export class ProjectDb {
     });
   }
 
+
   /**
    * Returns OTHER URLs in the same near-duplicate cluster as `urlId`.
    * Powers the URL Details panel's "Duplicates" sub-tab — the user
@@ -3793,6 +3810,12 @@ export class ProjectDb {
         `${html} AND status_code >= 200 AND status_code < 300
          AND (charset IS NULL OR charset = '')`,
       ),
+      ampValidationErrors: countWhere(
+        `${html} AND amp_page = 1
+         AND amp_validation_errors IS NOT NULL
+         AND amp_validation_errors != ''
+         AND amp_validation_errors != '[]'`,
+      ),
       brokenLinksInternal: this.countBrokenLinks('internal'),
       brokenLinksExternal: this.countBrokenLinks('external'),
       nearDuplicate: countWhere(`${html} AND cluster_id > 0 AND cluster_size > 1`),
@@ -5824,6 +5847,8 @@ export class ProjectDb {
     hreflangs: r.hreflangs,
     hreflangCount: r.hreflang_count,
     amphtml: r.amphtml,
+    ampPage: r.amp_page === 1,
+    ampValidationErrors: r.amp_validation_errors,
     favicon: r.favicon,
     mixedContentCount: r.mixed_content_count,
     mixedContentActive: r.mixed_content_active ?? 0,
@@ -6531,6 +6556,18 @@ function categoryWhereClause(cat: UrlCategory): string | null {
       return `is_external = 0 AND content_kind = 'html'
               AND status_code >= 200 AND status_code < 300
               AND (charset IS NULL OR charset = '')`;
+    case 'issues:amp-validation-errors':
+      // V2 Faz 16 — AMP smoke validation. Page declares itself AMP via
+      // `<html ⚡>` / `<html amp>` but trips one or more of the
+      // hand-rolled spec checks (missing boilerplate, missing runtime,
+      // forbidden script/style tags, etc). The JSON column contains
+      // the error code list; presence of any non-empty value other
+      // than "[]" means at least one rule failed.
+      return `is_external = 0 AND content_kind = 'html'
+              AND amp_page = 1
+              AND amp_validation_errors IS NOT NULL
+              AND amp_validation_errors != ''
+              AND amp_validation_errors != '[]'`;
     // Broken-link categories drive the BrokenLinksTab view; they never
     // filter the URL table itself.
     case 'issues:broken-links-all':

packages/shared-types/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@freecrawl/shared-types",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "private": true,
   "type": "module",
   "main": "./dist/index.js",

packages/shared-types/src/crawl.ts

@@ -101,6 +101,7 @@ export type UrlCategory =
   | 'issues:image-empty-alt'
   | 'issues:meta-refresh-used'
   | 'issues:charset-missing'
+  | 'issues:amp-validation-errors'
   | 'issues:broken-links-all'
   | 'issues:broken-links-internal'
   | 'issues:broken-links-external'
@@ -446,6 +447,13 @@ export interface CrawlUrlRow {
   hreflangs: string | null;
   hreflangCount: number;
   amphtml: string | null;
+  /** True when the page declares `<html ⚡>` / `<html amp>`. AMP-for-Ads
+   *  / AMP-for-Email variants (`⚡4ads`, `⚡4email`) count too. */
+  ampPage: boolean;
+  /** JSON-stringified array of AMP smoke-validator error codes (e.g.
+   *  `["missing-boilerplate","forbidden-script-tag"]`), or null when
+   *  not an AMP page / clean validation. */
+  ampValidationErrors: string | null;
   favicon: string | null;
   /** Resolved `<link rel="apple-touch-icon">` URL, else null. */
   appleTouchIcon: string | null;
@@ -535,6 +543,10 @@ export interface CrawlUrlRow {
  * the cluster size for the dedicated "Duplicates" tab.
  */
 export interface DuplicateClusterRow {
+  /** Internal URL row id — required by the grouped Duplicates view to
+   *  pivot the selection into the URL Details panel without a round-trip
+   *  URL → id lookup. */
+  urlId: number;
   url: string;
   statusCode: number | null;
   indexability: Indexability;
@@ -549,6 +561,12 @@ export interface DuplicateClusterRow {
   hammingFromRep: number;
 }
 
+/** Input for {@link FreeCrawlApi.duplicateClustersList}. */
+export interface DuplicateClustersListInput {
+  offset: number;
+  limit: number;
+}
+
 export interface CrawlConfig {
   mode: CrawlMode;
   /** When `mode === 'list'`, URLs to fetch (one per entry). Ignored in spider mode. */
@@ -1010,6 +1028,75 @@ export interface CustomExtractionRule {
   multi: 'first' | 'last' | 'all' | 'concat' | 'count';
 }
 
+/**
+ * Input for the live "Preview" button in Settings → Custom Extraction.
+ * Lets the user test selectors against a real URL without running a
+ * full crawl — the response is fetched once with a single GET, parsed
+ * via cheerio, and every rule is run against it. Output mirrors what
+ * the crawler stores per URL so the preview is faithful to production
+ * behaviour.
+ */
+export interface ExtractionPreviewInput {
+  url: string;
+  rules: CustomExtractionRule[];
+  /** Optional User-Agent override; falls back to the default FreeCrawl
+   *  preview UA when omitted. */
+  userAgent?: string;
+  /** Optional Accept-Language; mirrors crawl config. */
+  acceptLanguage?: string;
+}
+
+export interface ExtractionPreviewRuleResult {
+  /** Mirrors `CustomExtractionRule.name` so the renderer can correlate
+   *  result rows back to rule rows. */
+  name: string;
+  /** JSON-serialisable extracted value. `null` when the rule matched
+   *  nothing (selector missed, regex no match). */
+  value: unknown;
+  /** Populated when the rule itself failed (e.g. invalid regex, invalid
+   *  CSS selector). Distinct from a clean "no match" — the user wants
+   *  to know which selectors are broken. */
+  error?: string;
+}
+
+export interface ExtractionPreviewResult {
+  ok: boolean;
+  /** HTTP status code when the fetch resolved. */
+  statusCode?: number;
+  /** Final URL after redirects (empty if not followed). */
+  finalUrl?: string;
+  /** Response Content-Type header. */
+  contentType?: string;
+  /** Total response body size in bytes. */
+  byteSize?: number;
+  /** Wall-clock time of the fetch in milliseconds. */
+  fetchMs?: number;
+  /** Per-rule results in submission order. */
+  results?: ExtractionPreviewRuleResult[];
+  /** Fetch-level error (DNS, timeout, network). When set, `results` is
+   *  undefined. */
+  error?: string;
+}
+
+/** Result of the "Export Rules" button in Settings → Custom Extraction.
+ *  `filePath` empty + `bytesWritten` 0 when the user cancels the save
+ *  dialog (the renderer just dismisses the action). */
+export interface ExtractionRulesExportResult {
+  filePath: string;
+  bytesWritten: number;
+}
+
+/** Result of the "Import Rules" button. `rules` is the validated list
+ *  (rejected entries excluded); `skippedCount` is how many entries in
+ *  the file failed shape validation so the renderer can warn the user.
+ *  All-empty on cancel. */
+export interface ExtractionRulesImportResult {
+  filePath: string;
+  rules: CustomExtractionRule[];
+  skippedCount: number;
+  error?: string;
+}
+
 export interface OverviewCounts {
   summary: {
     totalInternalUrls: number;
@@ -1100,6 +1187,7 @@ export interface OverviewCounts {
     imageMissingAlt: number;
     metaRefreshUsed: number;
     charsetMissing: number;
+    ampValidationErrors: number;
     brokenLinksInternal: number;
     brokenLinksExternal: number;
     nearDuplicate: number;

packages/shared-types/src/ipc.ts

@@ -5,6 +5,13 @@ import type {
   CrawlProgress,
   CrawlSummary,
   CrawlUrlRow,
+  DuplicateClusterRow,
+  DuplicateClustersListInput,
+  CustomExtractionRule,
+  ExtractionPreviewInput,
+  ExtractionPreviewResult,
+  ExtractionRulesExportResult,
+  ExtractionRulesImportResult,
   ImageRow,
   Indexability,
   OverviewCounts,
@@ -136,6 +143,27 @@ export const IPC = {
   sitemapValidate: 'sitemap:validate',
   urlRewritePreview: 'url:rewrite-preview',
   urlClusterMembers: 'urls:cluster-members',
+  /** Faz 7 — grouped Duplicates tab view: paginates near-duplicate cluster
+   *  members ordered by cluster size DESC. Cluster headers / hamming
+   *  distance to the representative are derived in {@link
+   *  packages/db/src/project-db.ts#listDuplicateClusters}. */
+  duplicateClustersList: 'duplicates:clusters-list',
+  /** Faz 7 — total cluster-member row count, used by the grouped view to
+   *  stop scrolling once every member has been fetched. */
+  duplicateClustersCount: 'duplicates:clusters-count',
+  /** Faz 11 — live preview of custom extraction rules against a single
+   *  URL. Backs the "Preview" button in Settings → Custom Extraction so
+   *  users can validate selectors / regex without committing to a full
+   *  crawl. */
+  extractionPreview: 'extraction:preview',
+  /** Faz 11 — serialize the current rule set to a JSON file. Wraps the
+   *  rules in a versioned envelope so future imports can detect schema
+   *  drift. */
+  extractionRulesExport: 'extraction:rules-export',
+  /** Faz 11 — read a JSON file and validate its rules against the
+   *  CustomExtractionRule schema. Returns the cleaned-up list plus a
+   *  skipped-count so the renderer can warn about rejected entries. */
+  extractionRulesImport: 'extraction:rules-import',
   reportsPagesPerDirectory: 'reports:pages-per-directory',
   reportsStatusCodeHistogram: 'reports:status-code-histogram',
   reportsIndexabilityDistribution: 'reports:indexability-distribution',
@@ -1362,6 +1390,25 @@ export interface FreeCrawlApi {
   sitemapValidate(input: SitemapValidateInput): Promise<SitemapValidateResult>;
   urlRewritePreview(input: UrlRewritePreviewInput): Promise<UrlRewritePreviewResult>;
   urlClusterMembers(urlId: number): Promise<UrlClusterMember[]>;
+  /** Paginates near-duplicate cluster members for the dedicated grouped
+   *  Duplicates view. Pages are ordered by cluster size DESC, cluster id
+   *  ASC, URL ASC so the largest clusters come first. */
+  duplicateClustersList(input: DuplicateClustersListInput): Promise<DuplicateClusterRow[]>;
+  /** Total cluster-member row count (cluster_id > 0). */
+  duplicateClustersCount(): Promise<number>;
+  /** Faz 11 — fetch the URL once, parse HTML, run every supplied rule
+   *  against it and return the per-rule results so the user can spot
+   *  broken selectors / regex before committing to a full crawl. */
+  extractionPreview(input: ExtractionPreviewInput): Promise<ExtractionPreviewResult>;
+  /** Faz 11 — opens a save dialog and writes `rules` as a versioned
+   *  JSON envelope. `filePath` is empty when the user cancels. */
+  extractionRulesExport(
+    rules: CustomExtractionRule[],
+  ): Promise<ExtractionRulesExportResult>;
+  /** Faz 11 — opens an open dialog, parses + validates the JSON, and
+   *  returns the cleaned-up rule list. `filePath` is empty when the
+   *  user cancels. */
+  extractionRulesImport(): Promise<ExtractionRulesImportResult>;
   reportsPagesPerDirectory(input: PagesPerDirectoryInput): Promise<PagesPerDirectoryRow[]>;
   reportsStatusCodeHistogram(): Promise<StatusCodeHistogramRow[]>;
   reportsIndexabilityDistribution(): Promise<IndexabilityDistributionRow[]>;