feat: internal link score + RAM storage mode + traffic-ranked orphans

2026-10-18 00:39:38 +02:00 · 2026-07-06 02:25:30 +03:00
1 parent 9037e5139f
commit 90f3dbcfff
@@ -31,9 +31,9 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M

 FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 30 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), and a 10-rule custom extraction engine — **CSS selectors, XPath, regex, and JSONPath (for JSON API responses)** — with a **live Preview dialog and JSON import / export** for sharing rule sets.

-The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 16-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a **By-LCP above-the-fold colour overlay**, and a click-to-trace **Crawl Path Report** that highlights the shortest discovery path from the homepage to any page), a **standalone Log File Analyzer window** (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a `.seoproject` file association for OS double-click.
+The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 16-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a **By-LCP above-the-fold colour overlay**, a **By-Link-Score (internal PageRank) colour overlay** that shows where internal link equity concentrates, and a click-to-trace **Crawl Path Report** that highlights the shortest discovery path from the homepage to any page), a **standalone Log File Analyzer window** (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a `.seoproject` file association for OS double-click.

-Integrations cover **Google Search Console** (clicks, impressions, CTR, position + URL Inspection coverage), **Google Analytics 4** (sessions, users, bounce, engagement), **PageSpeed Insights** Lighthouse audits, custom **AI prompts via OpenAI / Anthropic Claude / local Ollama** with `{url}`/`{title}`/`{description}`/`{h1}`/`{body}` variables, and **SEO authority providers** (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an **in-app step-by-step Setup Guide modal** in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified **Export Crawl Data dialog** (Excel `.xlsx` / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to **Google Sheets** and **BigQuery**. Project files can be saved as **password-protected encrypted snapshots** (`.seoproject.enc`, AES-256-GCM + PBKDF2). The **MCP server** exposes **90 tools** that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL **Duplicates** view, and a custom-CSS theme override round out the suite. Everything runs **fully local** — no telemetry, no cloud, MIT-style license.
+Integrations cover **Google Search Console** (clicks, impressions, CTR, position + URL Inspection coverage), **Google Analytics 4** (sessions, users, bounce, engagement), **PageSpeed Insights** Lighthouse audits, custom **AI prompts via OpenAI / Anthropic Claude / local Ollama** with `{url}`/`{title}`/`{description}`/`{h1}`/`{body}` variables, and **SEO authority providers** (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an **in-app step-by-step Setup Guide modal** in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified **Export Crawl Data dialog** (Excel `.xlsx` / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to **Google Sheets** and **BigQuery**. Project files can be saved as **password-protected encrypted snapshots** (`.seoproject.enc`, AES-256-GCM + PBKDF2). The **MCP server** exposes **91 tools** that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL **Duplicates** view, and a custom-CSS theme override round out the suite. Everything runs **fully local** — no telemetry, no cloud, MIT-style license.

 <br />

@@ -205,7 +205,7 @@ FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the acti
 1. **Read-only data access** to the SQLite project — runs alongside the desktop app without contention (WAL allows concurrent readers).
 2. **Live crawl control** — when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to `<userData>/mcp-bridge.json` on app launch).

-**90 tools** — every action a human can take in the desktop is callable from an agent. Representative groups:
+**91 tools** — every action a human can take in the desktop is callable from an agent. Representative groups:

 | Group | Tools |
 | :--- | :--- |
@@ -213,7 +213,7 @@ FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the acti
 | 🔬 **Per-URL detail sub-tabs** | `get_url_source` (raw + rendered + screenshot paths), `get_url_inlinks`, `get_url_outlinks`, `get_url_images`, `get_url_headers`, `get_url_duplicates`, `get_url_analytics`, `get_url_cert` |
 | 🧩 **Integration rows** | `query_gsc`, `query_ga4`, `query_pagespeed`, `query_ai`, `query_seo` |
 | 🔎 **Specialised queries** | `query_images`, `query_broken_links`, `list_duplicate_clusters` |
-| 📈 **Reports** | `report_status_code_histogram`, `report_indexability_distribution`, `report_content_kind_distribution`, `report_depth_histogram`, `report_response_time_histogram`, `report_inlinks_histogram`, `report_word_count_histogram`, `report_url_length_histogram`, `report_top_urls_by`, `report_top_anchor_texts`, `report_external_domain_health`, `report_image_weight_per_page`, `report_link_position_breakdown`, `report_pages_per_directory`, `report_word_count_per_directory`, `report_sitemap_orphans`, `report_analytics_coverage`, `report_server_header_breakdown`, `report_query_string_variants`, `report_sitemap_priority_mismatch`, `report_top_words` |
+| 📈 **Reports** | `report_status_code_histogram`, `report_indexability_distribution`, `report_content_kind_distribution`, `report_depth_histogram`, `report_response_time_histogram`, `report_inlinks_histogram`, `report_word_count_histogram`, `report_url_length_histogram`, `report_top_urls_by`, `report_top_anchor_texts`, `report_external_domain_health`, `report_image_weight_per_page`, `report_link_position_breakdown`, `report_pages_per_directory`, `report_word_count_per_directory`, `report_sitemap_orphans`, `report_orphan_cross_source`, `report_analytics_coverage`, `report_server_header_breakdown`, `report_query_string_variants`, `report_sitemap_priority_mismatch`, `report_top_words` |
 | 📤 **Exports & mutations** | `export_csv`, `export_json`, `export_xml`, `export_html_report`, `export_tabular`, `export_bulk`, `export_broken_links`, `export_images`, `sitemap_generate`, `sitemap_validate`, `robots_test`, `robots_validate`, `compare_load`, `crawl_add_url`, `respider_urls`, `remove_urls`, `data_delete_by_domain`, `graph_snapshot`, `get_crawl_path`, `url_rewrite_preview`, `extraction_preview` |
 | 🔐 **Project files** | `project_save_as`, `project_save_encrypted`, `project_open_encrypted` |
 | 🔌 **Integration fetch** | `google_auth_status`, `gsc_list_sites`, `gsc_fetch`, `ga4_list_properties`, `ga4_fetch`, `pagespeed_run`, `ai_run`, `seo_run`, `get_fetch_progress`, `cancel_fetch` |
@@ -398,7 +398,7 @@ graph LR
 ## 📈 Status

 > [!NOTE]
-> **Active development.** All 29 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, **Cytoscape visualization (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP above-the-fold overlay + click-to-trace Crawl Path Report)**, **Basic / Digest / Bearer + form-based login (HTTP multi-step + Playwright browser-driven SPA login)**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **cross-source orphan detection**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **standalone Log File Analyzer (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, crawl × log orphan detection, 40+ bot detection with reverse-DNS verification, per-bot URL filter, CSV / Excel export, and a headless `analyze-logs` CLI)**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
+> **Active development.** All 29 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, **Cytoscape visualization (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP above-the-fold overlay + internal-PageRank link-score overlay + click-to-trace Crawl Path Report)**, **Basic / Digest / Bearer + form-based login (HTTP multi-step + Playwright browser-driven SPA login)**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **traffic-ranked cross-source orphan detection (sitemap + GSC + GA4)**, **internal PageRank link scoring**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, **RAM-only (in-memory) storage mode**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **standalone Log File Analyzer (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, crawl × log orphan detection, 40+ bot detection with reverse-DNS verification, per-bot URL filter, CSV / Excel export, and a headless `analyze-logs` CLI)**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
 >
 > **Upcoming (V2):** Plugin system, Light theme, Multi-window, Code-signing + auto-update.

@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/cli",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "type": "module",
  "bin": {
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/desktop",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "description": "FreeCrawl SEO â€” desktop application",
  "homepage": "https://github.com/kemalai/FreeCrawl-SEO-Tool",
@@ -82,6 +82,7 @@ const ALLOWED_METHODS = new Set<string>([
  'updateExternalProbe',
  'recomputeInlinks',
  'recomputeRedirectChains',
+  'recomputeLinkScore',
  'recomputeHreflangAnalysis',
  'recomputeHreflangInconsistent',
  'recomputeDuplicateClusters',
@@ -624,6 +624,11 @@ function maybeShowDiagnosticDialog(msg: string): void {
 /** Currently-open project file path (empty when using the default scratch DB). */
 let currentProjectPath = '';

+/** Storage mode the active DB connection was opened in. Resolved from the
+ *  `storageMode` pref at first open; `ram` means the project lives in an
+ *  in-memory database and the worker pools are disabled. */
+let storageModeActive: 'disk' | 'ram' = 'disk';
+
 /**
 * Dispatch HTML parsing to the worker pool when ready, otherwise
 * fall back to the inline parser on the main thread. This wrapper is
@@ -775,10 +780,20 @@ async function dbCallViaPool<T>(method: string, args: unknown[]): Promise<T> {

 function getDb(): ProjectDb {
  if (!db) {
+    loadPrefs();
+    // Storage mode (Settings → Storage). RAM-only keeps the whole project
+    // in an in-memory SQLite database (`:memory:`) — fastest, nothing
+    // touches disk until Save As serialises it out. A `:memory:` DB is
+    // private to a single connection, so the reader/writer worker pools are
+    // disabled in this mode and every query runs on the main-thread
+    // connection (the pool-call wrappers already fall back to it when the
+    // pools aren't ready). Applies at first open, i.e. on the next launch.
+    const ramMode = prefsCache['storageMode'] === 'ram';
+    storageModeActive = ramMode ? 'ram' : 'disk';
    const dataDir = join(app.getPath('userData'), 'projects');
    mkdirSync(dataDir, { recursive: true });
    const defaultPath = join(dataDir, 'default.seoproject');
-    db = new ProjectDb(defaultPath);
+    db = new ProjectDb(ramMode ? ':memory:' : defaultPath);
    currentProjectPath = '';
    // Wave 6 — Snapshot the previous session's crash-recovery state
    // BEFORE the reset wipes the project tables. We capture the
@@ -835,28 +850,39 @@ function getDb(): ProjectDb {
    // Fresh start on every app launch — clear any data carried over from
    // the previous session. Explicit Save Project will be added later.
    db.reset();
-    // Spawn the read-only worker pointed at the same file. Migrations
-    // already ran above (the writer owns them) so the worker just
-    // attaches to the WAL and starts serving SELECTs concurrently.
-    try {
-      // Pass the freeze-watchdog SAB so the reader worker can publish
-      // its own heartbeat + current op into the same diagnostic file.
-      dbReaderPool.init(defaultPath, freezeWatchdog.sharedBuffer);
-    } catch (err) {
+    if (ramMode) {
+      // In-memory DB can't be shared across worker connections, so keep
+      // everything on the main-thread connection. Save As (`VACUUM INTO`)
+      // serialises it to disk and switches back to a pooled disk project.
      logger.log(
-        'warn',
+        'info',
        'main',
-        `db-reader pool init failed: ${err instanceof Error ? err.message : String(err)} — falling back to main-thread queries.`,
-      );
-    }
-    try {
-      dbWriterPool.init(defaultPath, freezeWatchdog.sharedBuffer);
-    } catch (err) {
-      logger.log(
-        'warn',
-        'main',
-        `db-writer pool init failed: ${err instanceof Error ? err.message : String(err)} — writes fall back to main thread.`,
+        'Storage mode: RAM-only — project held in an in-memory database; worker pools disabled. Save As writes it to disk.',
      );
+    } else {
+      // Spawn the read-only worker pointed at the same file. Migrations
+      // already ran above (the writer owns them) so the worker just
+      // attaches to the WAL and starts serving SELECTs concurrently.
+      try {
+        // Pass the freeze-watchdog SAB so the reader worker can publish
+        // its own heartbeat + current op into the same diagnostic file.
+        dbReaderPool.init(defaultPath, freezeWatchdog.sharedBuffer);
+      } catch (err) {
+        logger.log(
+          'warn',
+          'main',
+          `db-reader pool init failed: ${err instanceof Error ? err.message : String(err)} — falling back to main-thread queries.`,
+        );
+      }
+      try {
+        dbWriterPool.init(defaultPath, freezeWatchdog.sharedBuffer);
+      } catch (err) {
+        logger.log(
+          'warn',
+          'main',
+          `db-writer pool init failed: ${err instanceof Error ? err.message : String(err)} — writes fall back to main thread.`,
+        );
+      }
    }
  }
  return db;
@@ -882,6 +908,10 @@ function openProjectAtPath(filePath: string): void {
  }
  db = new ProjectDb(filePath);
  currentProjectPath = filePath;
+  // Opening a real `.seoproject` file always lands us in disk mode, even
+  // when the previous project was an in-memory (RAM-only) scratch DB — the
+  // pool `swap` below spins the worker pools up against the disk file.
+  storageModeActive = 'disk';
  // Re-point the read-only worker at the new file. `swap` cancels any
  // in-flight requests with `reader-swapped`; the next IPC call will
  // hit the freshly opened worker.
@@ -2714,6 +2744,12 @@ function registerIpc(): void {
    return app.getPath('documents');
  });

+  ipcMain.handle(IPC.storageModeActive, (): 'disk' | 'ram' => {
+    // Ensure the DB has been resolved at least once so the mode is set.
+    getDb();
+    return storageModeActive;
+  });
+
  // V1 Faz 7 — integration credential store. Secrets are encrypted at
  // rest with safeStorage; the renderer only ever sees the redacted
  // state map (secret values blanked, non-secret values round-tripped).
@@ -343,6 +343,8 @@ const api: FreeCrawlApi = {
  pickDirectory: (input?: { title?: string; defaultPath?: string }): Promise<string | null> =>
    ipcRenderer.invoke(IPC.pickDirectory, input),
  defaultProjectDir: (): Promise<string> => ipcRenderer.invoke(IPC.defaultProjectDir),
+  storageModeActive: (): Promise<'disk' | 'ram'> =>
+    ipcRenderer.invoke(IPC.storageModeActive),
  integrationsGetAll: (): Promise<IntegrationsState> =>
    ipcRenderer.invoke(IPC.integrationsGetAll),
  integrationsSet: (
@@ -1752,6 +1752,7 @@ function NameValueView({ row }: { row: CrawlUrlRow }) {
    ['Crawl Depth', row.depth],
    ['Inlinks', row.inlinks],
    ['Outlinks', row.outlinks],
+    ['Link Score', row.linkScore],
    ['Redirect URL', row.redirectTarget],
    ['Last Crawled', row.crawledAt],
  ];
@@ -405,12 +405,19 @@ export function ReportsDialog({ open, onClose }: Props) {
              r.map((x: OrphanCrossSourceRow) => ({
                key: x.url,
                count: 1,
-                // `sources` (sitemap / gsc / ga4) goes in the secondary
-                // value column so the user can triage why each URL was
-                // missed by the crawl — sitemap-only often means a
-                // broken internal-link path; GSC + GA4 means real
-                // user-visited pages no crawl path reaches.
-                valueLabel: x.sources.join(' + '),
+                // `sources` (sitemap / gsc / ga4) plus any traffic go in the
+                // secondary value column so the user can triage why each URL
+                // was missed and how much it matters — sitemap-only with no
+                // traffic is often a dead entry to drop; GSC clicks / GA4
+                // sessions mean real user-visited pages no crawl path reaches.
+                valueLabel: [
+                  x.sources.join(' + '),
+                  x.gscClicks != null ? `${x.gscClicks.toLocaleString()} GSC clicks` : null,
+                  x.gscImpressions != null ? `${x.gscImpressions.toLocaleString()} impr` : null,
+                  x.ga4Sessions != null ? `${x.ga4Sessions.toLocaleString()} GA4 sessions` : null,
+                ]
+                  .filter(Boolean)
+                  .join(' · '),
              })),
            );
        } else if (kind === 'server-headers') {
@@ -131,6 +131,7 @@ interface FormState {
  // Wave 8 additions
  // crawl analysis (per-pass post-crawl toggles, Wave 6)
  analyseInlinks: boolean;
+  analyseLinkScore: boolean;
  analyseRedirectChains: boolean;
  analyseHreflang: boolean;
  analyseDuplicates: boolean;
@@ -456,6 +457,7 @@ function configToForm(c: CrawlConfig): FormState {
    excludeExtensionsText: (c.excludeExtensions ?? []).join(', '),
    maxRedirects: String(c.maxRedirects ?? 10),
    analyseInlinks: c.analyseInlinks ?? true,
+    analyseLinkScore: c.analyseLinkScore ?? true,
    analyseRedirectChains: c.analyseRedirectChains ?? true,
    analyseHreflang: c.analyseHreflang ?? true,
    analyseDuplicates: c.analyseDuplicates ?? true,
@@ -690,6 +692,7 @@ export function SettingsDialog({ open, onClose }: Props) {
        .filter(Boolean),
      maxRedirects: Math.max(0, num(form.maxRedirects, config.maxRedirects)),
      analyseInlinks: form.analyseInlinks,
+      analyseLinkScore: form.analyseLinkScore,
      analyseRedirectChains: form.analyseRedirectChains,
      analyseHreflang: form.analyseHreflang,
      analyseDuplicates: form.analyseDuplicates,
@@ -2872,6 +2875,12 @@ function CrawlAnalysisPanel({ form, update }: PanelProps) {
          onChange={(v) => update('analyseInlinks', v)}
          info="Counts how many internal pages link to each URL. Drives the Most-Linked URLs report and the per-row Inlinks column."
        />
+        <Bool
+          label={t('settingsPanels.crawlAnalysis.linkScore', { defaultValue: 'Compute link score (internal PageRank)' })}
+          checked={form.analyseLinkScore}
+          onChange={(v) => update('analyseLinkScore', v)}
+          info="Runs iterative PageRank (damping 0.85) over the internal link graph and normalises it to a 0–100 Link Score per page. Drives the Link Score column and the 'By Link Score' visualization colour mode."
+        />
        <Bool
          label={t('settingsPanels.crawlAnalysis.recomputeRedirectChains', { defaultValue: 'Recompute redirect chains' })}
          checked={form.analyseRedirectChains}
@@ -3954,6 +3963,10 @@ function StoragePanel() {
    return typeof raw === 'string' ? raw : '';
  });
  const [resolved, setResolved] = useState<string>('');
+  const [mode, setMode] = useState<'disk' | 'ram'>(() =>
+    window.freecrawl.prefsGet('storageMode') === 'ram' ? 'ram' : 'disk',
+  );
+  const [activeMode, setActiveMode] = useState<'disk' | 'ram' | null>(null);

  useEffect(() => {
    let cancelled = false;
@@ -3965,6 +3978,24 @@ function StoragePanel() {
    };
  }, [override]);

+  useEffect(() => {
+    let cancelled = false;
+    void window.freecrawl.storageModeActive().then((m) => {
+      if (!cancelled) setActiveMode(m);
+    });
+    return () => {
+      cancelled = true;
+    };
+  }, []);
+
+  function pickMode(m: 'disk' | 'ram') {
+    setMode(m);
+    // Disk is the default, so a `ram` pick persists the pref and a `disk`
+    // pick clears it back to the default rather than storing a redundant key.
+    if (m === 'ram') window.freecrawl.prefsSet('storageMode', 'ram');
+    else window.freecrawl.prefsDelete('storageMode');
+  }
+
  async function browse() {
    const chosen = await window.freecrawl.pickDirectory({
      title: 'Choose Default Project Folder',
@@ -3988,6 +4019,54 @@ function StoragePanel() {
        {t('settings.storage.intro', { defaultValue: "Default folder for new .seoproject files. Save As dialogs open here by default; the Open dialog is not constrained." })}
      </p>

+      <div className="mb-4 rounded border border-surface-800 bg-surface-950/40 p-3">
+        <div className="mb-2 text-[10px] font-semibold uppercase tracking-wider text-surface-400">
+          {t('settings.storage.modeLabel', { defaultValue: "Storage Mode" })}
+        </div>
+        <div className="flex flex-col gap-2">
+          <label className="flex items-start gap-2 cursor-pointer">
+            <input
+              type="radio"
+              name="storage-mode"
+              className="mt-0.5"
+              checked={mode === 'disk'}
+              onChange={() => pickMode('disk')}
+            />
+            <span className="text-[12px] text-surface-100">
+              {t('settings.storage.modeDisk', { defaultValue: "Disk (SQLite file) — persistent, resumable, uses worker pools" })}
+            </span>
+          </label>
+          <label className="flex items-start gap-2 cursor-pointer">
+            <input
+              type="radio"
+              name="storage-mode"
+              className="mt-0.5"
+              checked={mode === 'ram'}
+              onChange={() => pickMode('ram')}
+            />
+            <span className="text-[12px] text-surface-100">
+              {t('settings.storage.modeRam', { defaultValue: "RAM-only (in-memory) — fastest, nothing written to disk until Save As" })}
+            </span>
+          </label>
+        </div>
+        <p className="mt-2 text-[10px] text-surface-500">
+          {activeMode && activeMode !== mode
+            ? t('settings.storage.modeRestart', {
+                defaultValue: "Applies the next time FreeCrawl starts (currently running in {{active}} mode).",
+                active: activeMode.toUpperCase(),
+              })
+            : t('settings.storage.modeActiveNote', {
+                defaultValue: "Currently running in {{active}} mode.",
+                active: (activeMode ?? mode).toUpperCase(),
+              })}
+        </p>
+        {mode === 'ram' && (
+          <p className="mt-1 text-[10px] text-amber-500/80">
+            {t('settings.storage.modeRamWarning', { defaultValue: "In-memory projects are lost on exit unless you Save As. Crash recovery is off and very large crawls are bounded by available RAM." })}
+          </p>
+        )}
+      </div>
+
      <div className="mb-4 rounded border border-surface-800 bg-surface-950/40 p-3">
        <div className="mb-2 text-[10px] font-semibold uppercase tracking-wider text-surface-400">
          {t('settings.storage.defaultSaveFolder', { defaultValue: "Default Save Folder" })}
@@ -350,6 +350,7 @@ const TR_LABELS: Record<string, string> = {
  'Size (Bytes)': 'Boyut (Bayt)',
  Inlinks: 'Gelen Linkler',
  Outlinks: 'Giden Linkler',
+  'Link Score': 'Bağlantı Puanı',
  'Title 1': 'Başlık 1',
  'Title 1 Length': 'Başlık 1 Uzunluğu',
  'Meta Description 1': 'Meta Açıklama 1',
@@ -602,6 +602,7 @@
    "crawlAnalysis": {
      "intro": "Per-pass post-crawl analysis toggles. Each pass runs after the HTTP fetch phase finishes and feeds different issue filters. Skip passes you don't need to shave wall-clock on large crawls; the related issue counters quietly read as 0 until the pass runs.",
      "recomputeInlinks": "Recompute inlinks",
+      "linkScore": "Compute link score (internal PageRank)",
      "recomputeRedirectChains": "Recompute redirect chains",
      "hreflang": "Hreflang reciprocity + inconsistent lang",
      "nearDuplicate": "Near-duplicate clustering",
@@ -923,6 +924,7 @@
    "byDepth": "By Depth",
    "byIndexability": "By Indexability",
    "byLcp": "By LCP (above-fold)",
+    "byLinkScore": "By Link Score",
    "crawlPath": "Crawl Path",
    "crawlPathTitle": "Crawl Path mode — click a page node to trace its shortest path from the crawl root",
    "crawlPathHint": "Select a page node to trace its shortest path from the crawl root.",
@@ -1238,6 +1240,12 @@
    "storage": {
      "title": "Storage",
      "intro": "Default folder for new .seoproject files. Save As dialogs open here by default; the Open dialog is not constrained.",
+      "modeLabel": "Storage Mode",
+      "modeDisk": "Disk (SQLite file) — persistent, resumable, uses worker pools",
+      "modeRam": "RAM-only (in-memory) — fastest, nothing written to disk until Save As",
+      "modeRestart": "Applies the next time FreeCrawl starts (currently running in {{active}} mode).",
+      "modeActiveNote": "Currently running in {{active}} mode.",
+      "modeRamWarning": "In-memory projects are lost on exit — and when you open or switch to another project — unless you Save As first. Crash recovery is off and very large crawls are bounded by available RAM.",
      "defaultSaveFolder": "Default Save Folder",
      "placeholder": "(System Documents folder)",
      "resolvedPath": "Resolved path",
@@ -602,6 +602,7 @@
    "crawlAnalysis": {
      "intro": "Pas-başına post-crawl analiz toggle'ları. Her pas HTTP fetch fazı bittikten sonra çalışır ve farklı issue filtrelerini besler. İhtiyacın olmayan pasları atla — ilgili issue sayaçları pas çalışana kadar 0 okur.",
      "recomputeInlinks": "Inlink'leri yeniden hesapla",
+      "linkScore": "Bağlantı puanı hesapla (dahili PageRank)",
      "recomputeRedirectChains": "Yönlendirme zincirlerini yeniden hesapla",
      "hreflang": "Hreflang karşılıklılığı + tutarsız lang",
      "nearDuplicate": "Yakın-yinelenen kümeleme",
@@ -923,6 +924,7 @@
    "byDepth": "Derinliğe Göre",
    "byIndexability": "İndekslenebilirliğe Göre",
    "byLcp": "LCP'ye Göre (fold üstü)",
+    "byLinkScore": "Bağlantı Puanına Göre",
    "crawlPath": "Tarama Yolu",
    "crawlPathTitle": "Tarama Yolu modu — bir sayfa düğümüne tıklayarak tarama kökünden en kısa yolu izleyin",
    "crawlPathHint": "Tarama kökünden en kısa yolu izlemek için bir sayfa düğümü seçin.",
@@ -1238,6 +1240,12 @@
    "storage": {
      "title": "Depolama",
      "intro": "Yeni .seoproject dosyaları için varsayılan klasör. Farklı Kaydet diyalogları varsayılan olarak burayı açar; Aç diyaloğu kısıtlı değildir.",
+      "modeLabel": "Depolama Modu",
+      "modeDisk": "Disk (SQLite dosyası) — kalıcı, sürdürülebilir, worker havuzlarını kullanır",
+      "modeRam": "Yalnızca RAM (bellek içi) — en hızlı, Farklı Kaydet'e kadar diske hiçbir şey yazılmaz",
+      "modeRestart": "FreeCrawl'ı bir sonraki açışınızda uygulanır (şu an {{active}} modunda çalışıyor).",
+      "modeActiveNote": "Şu an {{active}} modunda çalışıyor.",
+      "modeRamWarning": "Bellek içi projeler — çıkışta ve başka bir proje açtığınızda/geçtiğinizde — önce Farklı Kaydet yapmazsanız kaybolur. Çökme kurtarma kapalıdır ve çok büyük crawl'lar mevcut RAM ile sınırlıdır.",
      "defaultSaveFolder": "Varsayılan Kayıt Klasörü",
      "placeholder": "(Sistem Belgeler klasörü)",
      "resolvedPath": "Çözümlenen yol",
@@ -1729,6 +1729,19 @@ function Cell({

  if (spec.kind === 'status') {
    const code = row.statusCode;
+    // No HTTP status (DNS/TLS/connect/timeout failure): a bare "—" hides the
+    // reason the crawler already captured. Show a compact token (DNS / TLS /
+    // Timeout / …) with the full error text on hover.
+    if (code === null || code === undefined) {
+      return (
+        <span
+          className="inline-block rounded bg-rose-950/60 px-1.5 font-mono text-[10px] text-rose-300"
+          title={row.statusText ?? undefined}
+        >
+          {shortFailureLabel(row.statusText)}
+        </span>
+      );
+    }
    return (
      <span
        className={clsx(
@@ -1736,7 +1749,7 @@ function Cell({
          statusClasses(code),
        )}
      >
-        {code ?? '—'}
+        {code}
      </span>
    );
  }
@@ -1810,6 +1823,24 @@ function indexabilityStatusLabel(v: CrawlUrlRow['indexability']): string {
  }
 }

+/**
+ * Compact token for a no-HTTP-response failure, derived from the crawler's
+ * captured `statusText`. Technical protocol abbreviations (DNS/TLS/H2/…) — not
+ * localized, same convention as the numeric status codes themselves. The full
+ * diagnostic is on hover (title) and in the URL Details → HTTP Headers tab.
+ */
+function shortFailureLabel(statusText: string | null): string {
+  if (!statusText) return '—';
+  if (/ENOTFOUND|EAI_AGAIN|ENODATA|ESERVFAIL|getaddrinfo|DNS/i.test(statusText)) return 'DNS';
+  if (/UNABLE_TO_VERIFY|CERT|SSL|TLSV1|HANDSHAKE|EPROTO/i.test(statusText)) return 'TLS';
+  if (/ECONNREFUSED/i.test(statusText)) return 'Refused';
+  if (/ECONNRESET|SOCKET|EPIPE|reset/i.test(statusText)) return 'Reset';
+  if (/HTTP2|NGHTTP2|GOAWAY|PROTOCOL_ERROR/i.test(statusText)) return 'H2 err';
+  if (/CONNECT_TIMEOUT|ETIMEDOUT|HEADERS_TIMEOUT|BODY_TIMEOUT|aborted|timeout/i.test(statusText))
+    return 'Timeout';
+  return 'Failed';
+}
+
 function statusClasses(code: number | null): string {
  if (code === null) return 'bg-surface-800 text-surface-400';
  if (code >= 200 && code < 300) return 'bg-emerald-900/60 text-emerald-300';
@@ -190,7 +190,7 @@ const LAYOUTS: { key: LayoutKind; label: string; hint: string }[] = [
  { key: 'concentric', label: 'Concentric', hint: 'By inlinks (centre = most-linked)' },
 ];

-type ColorMode = 'status' | 'depth' | 'indexability' | 'lcp';
+type ColorMode = 'status' | 'depth' | 'indexability' | 'lcp' | 'linkScore';

 /** Colour a node by its Largest-Contentful-Paint candidate: grey when no
 *  render data, amber when the LCP is an image (a prime optimisation
@@ -285,6 +285,22 @@ function depthColor(d: number): string {
  return palette[Math.min(d, palette.length - 1)] ?? '#a3a3a3';
 }

+/** Colour a node by internal link score (PageRank, 0..100): grey when not
+ *  yet computed, pale green at the bottom of the graph, deep green for the
+ *  pages holding the most internal link equity. */
+function linkScoreColor(score: number | null): string {
+  if (score === null || score === undefined) return '#737373';
+  const palette = [
+    '#f0fdf4', '#dcfce7', '#bbf7d0', '#86efac', '#4ade80',
+    '#22c55e', '#16a34a', '#15803d', '#166534', '#14532d',
+  ];
+  const idx = Math.min(
+    palette.length - 1,
+    Math.max(0, Math.floor((score / 100) * palette.length)),
+  );
+  return palette[idx] ?? '#737373';
+}
+
 function indexColor(i: Indexability): string {
  if (i === 'indexable') return '#16a34a';
  if (i.startsWith('non-indexable')) return '#dc2626';
@@ -396,6 +412,7 @@ export function VisualizationTab() {
      if (colorMode === 'depth') return depthColor(n.depth);
      if (colorMode === 'indexability') return indexColor(n.indexability);
      if (colorMode === 'lcp') return lcpColor(n);
+      if (colorMode === 'linkScore') return linkScoreColor(n.linkScore);
      return statusColor(n.statusCode);
    };

@@ -778,6 +795,7 @@ export function VisualizationTab() {
              <option value="depth">{t('viz.byDepth', { defaultValue: 'By Depth' })}</option>
              <option value="indexability">{t('viz.byIndexability', { defaultValue: 'By Indexability' })}</option>
              <option value="lcp">{t('viz.byLcp', { defaultValue: 'By LCP (above-fold)' })}</option>
+              <option value="linkScore">{t('viz.byLinkScore', { defaultValue: 'By Link Score' })}</option>
            </select>
          </label>
          <label className="flex items-center gap-1 text-surface-400">
@@ -210,6 +210,14 @@ const COL = {
    info: 'Number of unique <a> links emitted from this page (internal + external).',
    example: '32',
  } as ColumnSpec,
+  linkScore: {
+    key: 'linkScore',
+    header: 'Link Score',
+    size: 92,
+    kind: 'number',
+    info: 'Internal PageRank, 0–100. Computed over the internal link graph (damping 0.85) and normalised so the most-linked page scores 100. Higher = more internal link equity.',
+    example: '73',
+  } as ColumnSpec,
  imagesCount: {
    key: 'imagesCount',
    header: 'Images',
@@ -541,6 +549,7 @@ export const COLUMN_SPECS: Record<TabKey, ColumnSpec[]> = {
    COL.depth,
    COL.inlinks,
    COL.outlinks,
+    COL.linkScore,
    COL.imagesCount,
    COL.imagesMissingAlt,
  ],
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/mcp-server",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "type": "module",
  "bin": {
@@ -941,6 +941,18 @@ export function buildTools(): Tool[] {
        rows: db.sitemapOrphans(clamp(args.limit, 1, 10_000, 1000)),
      }),
    },
+    {
+      name: 'report_orphan_cross_source',
+      description:
+        'Orphan pages across all truth sources: URLs present in the XML sitemap, Google Search Console, or Google Analytics 4 but never reached by the crawl (or crawled only as an unfetched stub). Each row lists which sources mentioned it plus its GSC clicks / impressions and GA4 sessions; sorted by traffic (GSC clicks + GA4 sessions) descending so high-value orphans surface first. Sitemap-only with no traffic is often a dead entry; GSC/GA4 orphans are real pages users reach that no internal link path does.',
+      inputSchema: {
+        type: 'object',
+        properties: { limit: { type: 'integer', minimum: 1, maximum: 10_000 } },
+      },
+      handler: (args, db) => ({
+        rows: db.orphanPagesCrossSource(clamp(args.limit, 1, 10_000, 1000)),
+      }),
+    },
    {
      name: 'report_analytics_coverage',
      description: 'Which third-party analytics trackers (GA4 / GTM / Hotjar / Clarity / Mixpanel / Plausible / ...) appear on which fraction of indexable pages. Reveals coverage gaps and ID inconsistencies (multiple GA4 properties tagged across the site, etc).',
@@ -1,6 +1,6 @@
 {
  "name": "freecrawl-seo",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "description": "Open-source desktop SEO crawler â€” a free alternative to Screaming Frog",
  "license": "MIT",
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/core",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",
@@ -1,6 +1,6 @@
 import { EventEmitter } from 'node:events';
 import * as os from 'node:os';
-import { fetch as undiciFetch } from 'undici';
+import { fetch as undiciFetch, type Dispatcher } from 'undici';
 import { load as cheerioLoad } from 'cheerio';
 import PQueue from 'p-queue';
 import type {
@@ -28,7 +28,9 @@ import { parseHtml, estimatePixelWidth, type ParsedPage } from './html-parser.js
 import { analyseCookies, extractSetCookies } from './cookies.js';
 import { loadRobots, type RobotsChecker } from './robots.js';
 import {
+  BROWSER_FALLBACK_UA,
  buildDigestAuthHeader,
+  buildLenientDispatcher,
  collectNetworkDiagnostics,
  defaultRequestHeaders,
  detectHttpProtocol,
@@ -77,6 +79,31 @@ const EXT_TO_KIND: Record<string, ContentKind> = {
  otf: 'font',
 };

+// Statuses a HEAD request often reports that differ from a real GET: many
+// CDNs/WAFs answer a HEAD with 403/405 but a GET with 200 (e.g. Netflix
+// HEAD 403 / GET 200). Retry these — plus any 5xx or transport failure —
+// with GET before recording, so the status matches what a visitor sees.
+// Module-scope so the external-probe hot path doesn't rebuild it per URL.
+const HEAD_UNRELIABLE_STATUSES = new Set([
+  400, 401, 403, 405, 406, 409, 429, 501,
+]);
+
+const CERT_VERIFY_ERROR_RE =
+  /UNABLE_TO_VERIFY_LEAF_SIGNATURE|UNABLE_TO_GET_ISSUER_CERT_LOCALLY|SELF_SIGNED_CERT_IN_CHAIN|DEPTH_ZERO_SELF_SIGNED_CERT|CERT_UNTRUSTED/;
+function isCertVerifyError(err: unknown): boolean {
+  return CERT_VERIFY_ERROR_RE.test(formatFetchError(err));
+}
+
+// DNS / TCP-connect failures are method- and UA-independent — a browser-UA
+// GET can't recover them, so retrying only doubles the wait on a dead host.
+// (HTTP/2 resets, headers-timeouts and 4xx/5xx statuses CAN differ on GET,
+// so those still get the retry.)
+const HARD_TRANSPORT_ERROR_RE =
+  /ENOTFOUND|EAI_AGAIN|ENODATA|ESERVFAIL|ECONNREFUSED|UND_ERR_CONNECT_TIMEOUT/;
+function isHardTransportError(err: unknown): boolean {
+  return HARD_TRANSPORT_ERROR_RE.test(formatFetchError(err));
+}
+
 export class Crawler extends EventEmitter {
  private config: CrawlConfig;
  private readonly db: ProjectDb;
@@ -786,6 +813,12 @@ export class Crawler extends EventEmitter {
      this.setOp('post-crawl:recompute-inlinks');
      await this.runDbPass('recomputeInlinks');
    }
+    if (this.config.analyseLinkScore) {
+      await yieldToEventLoop();
+      this.emit('info', 'Computing internal link scores…');
+      this.setOp('post-crawl:link-score');
+      await this.runDbPass('recomputeLinkScore');
+    }
    if (this.config.analyseRedirectChains) {
      await yieldToEventLoop();
      this.emit('info', 'Recomputing redirect chains…');
@@ -1635,37 +1668,81 @@ export class Crawler extends EventEmitter {
  private async probeExternal(url: string): Promise<void> {
    if (this.stopped) return;
    const t0 = Date.now();
-    const controller = new AbortController();
-    const timeout = setTimeout(() => controller.abort(), this.config.requestTimeoutMs);
-    const doFetch = async (method: 'HEAD' | 'GET') =>
-      undiciFetch(url, {
-        method,
-        headers: defaultRequestHeaders(
-          this.resolveUserAgent(url),
-          this.config.acceptLanguage,
-          this.config.customHeaders,
-          this.config.auth,
-        ),
-        redirect: this.config.followRedirects ? 'follow' : 'manual',
-        signal: controller.signal,
-      });
+    const cfgUa = this.resolveUserAgent(url);

-    try {
-      let res;
+    // One fetch attempt with its OWN timeout budget. Giving HEAD and the GET
+    // retry independent AbortControllers is what stops a hung HEAD from eating
+    // the GET's time and collapsing a specific error (connect timeout, TLS
+    // reject) into a generic "operation was aborted — raise your Timeout".
+    const attempt = async (
+      method: 'HEAD' | 'GET',
+      ua: string,
+      dispatcher?: Dispatcher,
+    ): Promise<{ res?: Awaited<ReturnType<typeof undiciFetch>>; err?: unknown }> => {
+      const controller = new AbortController();
+      const timer = setTimeout(() => controller.abort(), this.config.requestTimeoutMs);
      try {
-        res = await doFetch('HEAD');
-      } catch {
-        // Some hosts / WAFs reject HEAD — fall back to GET and discard body.
-        res = await doFetch('GET');
+        const res = await undiciFetch(url, {
+          method,
+          headers: defaultRequestHeaders(
+            ua,
+            this.config.acceptLanguage,
+            this.config.customHeaders,
+            this.config.auth,
+          ),
+          redirect: this.config.followRedirects ? 'follow' : 'manual',
+          signal: controller.signal,
+          ...(dispatcher ? { dispatcher } : {}),
+        });
+        return { res };
+      } catch (err) {
+        return { err };
+      } finally {
+        clearTimeout(timer);
      }
-      // If HEAD returned a suspicious status (405/403) try GET once to confirm
-      if (res.status === 405 || res.status === 501) {
-        try {
-          res = await doFetch('GET');
-        } catch {
-          /* keep HEAD result */
+    };
+
+    // 1) Cheap HEAD with the configured UA.
+    let { res, err } = await attempt('HEAD', cfgUa);
+
+    // 2) Retry with GET (browser UA) when HEAD is unreliable or threw a
+    //    recoverable error. The browser UA slips past UA-based bot walls; the
+    //    GET reveals the status a real visitor would get.
+    const headBlocked =
+      !!res && (res.status >= 500 || HEAD_UNRELIABLE_STATUSES.has(res.status));
+    const headRecoverableThrow = !res && !isHardTransportError(err);
+    if (headBlocked || headRecoverableThrow) {
+      try {
+        await res?.body?.cancel();
+      } catch {
+        /* ignore */
+      }
+      const got = await attempt('GET', BROWSER_FALLBACK_UA);
+      if (got.res) {
+        res = got.res;
+        err = undefined;
+      } else if (!res) {
+        // Prefer the more specific of the two transport errors.
+        err = got.err ?? err;
+      }
+    }
+
+    // 3) Last-resort TLS-relaxed retry for incomplete-chain certificates that
+    //    browsers accept (via AIA) but Node rejects. External link checking
+    //    only needs the status code. Skipped when a proxy is active.
+    if (!res && isCertVerifyError(err)) {
+      const lenient = buildLenientDispatcher();
+      if (lenient) {
+        const got = await attempt('GET', BROWSER_FALLBACK_UA, lenient);
+        if (got.res) {
+          res = got.res;
+          err = undefined;
        }
      }
+    }
+
+    const responseTimeMs = Date.now() - t0;
+    if (res) {
      try {
        await res.body?.cancel();
      } catch {
@@ -1675,22 +1752,21 @@ export class Crawler extends EventEmitter {
        url,
        {
          statusCode: res.status,
+          statusText: res.statusText || null,
          contentType: res.headers.get('content-type'),
          contentLength: parseIntSafe(res.headers.get('content-length')),
-          responseTimeMs: Date.now() - t0,
+          responseTimeMs,
        },
      ]);
-    } catch (err) {
+    } else {
      await this.dbCall<void>('updateExternalProbe', [
        url,
        {
          statusCode: null,
          statusText: formatFetchError(err),
-          responseTimeMs: Date.now() - t0,
+          responseTimeMs,
        },
      ]);
-    } finally {
-      clearTimeout(timeout);
    }
  }

@@ -9,6 +9,28 @@ let dnsInitialized = false;
 *  only rebuild when it actually changes (a second crawl can switch proxy). */
 let lastProxyKey: string | null = null;

+/** Resilient DNS lookup from the last direct-mode init, reused when building
+ *  the lenient (TLS-relaxed) dispatcher below. */
+let sharedResilientLookup: ReturnType<typeof createResilientLookup> | null = null;
+/** True while the global dispatcher is a direct Agent (no proxy). Lenient TLS
+ *  retry is only offered in this mode so we never silently bypass a proxy. */
+let directMode = false;
+/** Lazily-built dispatcher with certificate verification disabled — used ONLY
+ *  as a last-resort retry when an EXTERNAL link fails TLS chain verification
+ *  (incomplete chain that browsers complete via AIA fetching but Node does
+ *  not). Rebuilt whenever the effective proxy changes. */
+let lenientDispatcher: Agent | null = null;
+
+/**
+ * Browser-like User-Agent used as a fallback when an external-link probe with
+ * the configured (often non-browser) UA is blocked. Many WAFs/CDNs actively
+ * reject or reset non-browser UAs — e.g. Microsoft resets the HTTP/2 stream,
+ * Cloudflare Bot-Fight returns 403 — so a link that a real visitor can open is
+ * wrongly recorded as failed. See `Crawler.probeExternal`.
+ */
+export const BROWSER_FALLBACK_UA =
+  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36';
+
 /**
 * Configure the global undici dispatcher and Node DNS once per process.
 *
@@ -60,8 +82,19 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
  const key = proxyUrl || '<direct>';
  if (key === lastProxyKey) return;
  lastProxyKey = key;
+  // Effective transport changed — drop the cached lenient dispatcher so it
+  // rebuilds against the new proxy/direct state and DNS lookup. Close the
+  // stale Agent first so its keep-alive socket pool is released now instead
+  // of leaking until GC; fall back to a forceful destroy if graceful close
+  // rejects (e.g. already closing).
+  if (lenientDispatcher) {
+    const stale = lenientDispatcher;
+    void stale.close().catch(() => stale.destroy().catch(() => undefined));
+  }
+  lenientDispatcher = null;

  if (proxyUrl) {
+    directMode = false;
    const scheme = proxyUrl.slice(0, Math.max(0, proxyUrl.indexOf(':'))).toLowerCase();
    if (scheme.startsWith('socks')) {
      setGlobalDispatcher(buildSocksDispatcher(proxyUrl));
@@ -72,6 +105,8 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
  }

  const lookup = createResilientLookup({ onEvent: opts.onDnsEvent });
+  sharedResilientLookup = lookup;
+  directMode = true;
  const agent = new Agent({
    connections: 128,
    pipelining: 1,
@@ -93,6 +128,38 @@ export function initHttpClient(opts: { proxyOverride?: string; onDnsEvent?: DnsR
  setGlobalDispatcher(agent);
 }

+/**
+ * Lazily build (and cache) a dispatcher whose TLS layer does NOT verify the
+ * certificate chain, reusing the resilient DNS lookup. Used as a last-resort
+ * retry for EXTERNAL link probes that fail with an incomplete-chain error
+ * (`UNABLE_TO_VERIFY_LEAF_SIGNATURE`): browsers complete such chains via AIA
+ * fetching, Node does not, so an otherwise-reachable link is wrongly recorded
+ * as dead. External link checking only needs the status code — TLS problems
+ * are surfaced separately by the Security tab / certificate probe.
+ *
+ * Returns `null` when a proxy is active (we must not bypass a corporate proxy
+ * just to relax TLS) or before `initHttpClient` has run in direct mode.
+ */
+export function buildLenientDispatcher(): Agent | null {
+  if (!directMode || !sharedResilientLookup) return null;
+  if (lenientDispatcher) return lenientDispatcher;
+  lenientDispatcher = new Agent({
+    connections: 32,
+    pipelining: 1,
+    keepAliveTimeout: 10_000,
+    keepAliveMaxTimeout: 30_000,
+    headersTimeout: 10_000,
+    bodyTimeout: 30_000,
+    connect: {
+      lookup: sharedResilientLookup as never,
+      rejectUnauthorized: false,
+      autoSelectFamily: true,
+      autoSelectFamilyAttemptTimeout: 250,
+    },
+  });
+  return lenientDispatcher;
+}
+
 /**
 * Build an undici dispatcher that tunnels every connection through a SOCKS
 * proxy. undici's own `ProxyAgent` only speaks HTTP CONNECT, so for SOCKS
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/db",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",
@@ -1777,6 +1777,23 @@ const MIGRATIONS: Migration[] = [
      CREATE INDEX IF NOT EXISTS idx_log_url_bot_bot ON log_url_bot(bot);
    `,
  },
+  {
+    version: 74,
+    name: 'add_link_score',
+    // V2 — internal PageRank / link score. A per-URL 0..100 integer
+    // computed post-crawl over the internal link graph (damping 0.85),
+    // normalised so the most-linked page scores 100. NULL until the
+    // post-crawl `recomputeLinkScore` pass runs; 0 when the crawl has no
+    // internal edges to distribute equity through.
+    up: (db) => {
+      const cols = db
+        .prepare('PRAGMA table_info(urls)')
+        .all() as unknown as { name: string }[];
+      if (!cols.some((c) => c.name === 'link_score')) {
+        db.exec('ALTER TABLE urls ADD COLUMN link_score INTEGER');
+      }
+    },
+  },
 ];

 export function runMigrations(db: DatabaseSync): void {
@@ -99,6 +99,7 @@ interface UrlRowDb {
  depth: number;
  inlinks: number;
  outlinks: number;
+  link_score: number | null;
  redirect_target: string | null;
  crawled_at: string;
  is_external: number;
@@ -1263,6 +1264,29 @@ export class ProjectDb {
    // so the `external:html` filter stays empty. COALESCE keeps the prior
    // value when the probe returned no usable Content-Type.
    const contentKind = contentKindFromType(patch.contentType);
+    // Derive indexability from the probed status. External stubs are inserted
+    // with a placeholder 'indexable'; without recomputing here, a link that
+    // 404s, is bot-blocked, or never responded would stay reported as
+    // "indexable" (misleading, and inflates the Indexable count).
+    const status = patch.statusCode;
+    let indexability: string;
+    let indexabilityReason: string | null;
+    if (status === null) {
+      indexability = 'non-indexable:client-error';
+      indexabilityReason = 'No response';
+    } else if (status >= 500) {
+      indexability = 'non-indexable:server-error';
+      indexabilityReason = `HTTP ${status}`;
+    } else if (status >= 400) {
+      indexability = 'non-indexable:client-error';
+      indexabilityReason = `HTTP ${status}`;
+    } else if (status >= 300) {
+      indexability = 'non-indexable:redirect';
+      indexabilityReason = `HTTP ${status}`;
+    } else {
+      indexability = 'indexable';
+      indexabilityReason = null;
+    }
    this.db
      .prepare(
        `UPDATE urls SET
@@ -1271,7 +1295,9 @@ export class ProjectDb {
           content_type = :content_type,
           content_kind = COALESCE(:content_kind, content_kind),
           content_length = :content_length,
-           response_time_ms = :response_time_ms
+           response_time_ms = :response_time_ms,
+           indexability = :indexability,
+           indexability_reason = :indexability_reason
         WHERE url = :url AND is_external = 1`,
      )
      .run({
@@ -1282,6 +1308,8 @@ export class ProjectDb {
        content_kind: contentKind,
        content_length: patch.contentLength ?? null,
        response_time_ms: patch.responseTimeMs ?? null,
+        indexability,
+        indexability_reason: indexabilityReason,
      });
  }

@@ -1935,6 +1963,130 @@ export class ProjectDb {
    }
  }

+  /**
+   * Internal PageRank / "link score". Runs the classic iterative PageRank
+   * over the internal link graph (`links.is_internal = 1`), then normalises
+   * the stationary distribution so the most-linked page scores 100 and the
+   * rest scale linearly down to 0. Written to `urls.link_score` (INTEGER).
+   *
+   * Model details:
+   *  - Nodes = every internal URL (`is_external = 0`). Edges = unique
+   *    (from → to) internal links; duplicate nav/footer links to the same
+   *    target are collapsed so template chrome doesn't inflate the score.
+   *  - Damping 0.85 (Google's original constant). Dangling nodes (no
+   *    out-links) redistribute their mass uniformly, so equity is conserved.
+   *  - Converges on L1 delta < 1e-6 or `iterations` cap, whichever first.
+   *  - When the crawl has zero internal edges there is no equity to
+   *    distribute, so every page is written 0 (not a misleading uniform 100).
+   */
+  recomputeLinkScore(
+    opts: { damping?: number; iterations?: number } = {},
+  ): { pages: number; iterations: number } {
+    const damping = opts.damping ?? 0.85;
+    const maxIterations = opts.iterations ?? 50;
+
+    const nodeRows = this.db
+      .prepare('SELECT id, url FROM urls WHERE is_external = 0')
+      .all() as { id: number; url: string }[];
+    const n = nodeRows.length;
+    if (n === 0) return { pages: 0, iterations: 0 };
+
+    const indexById = new Map<number, number>();
+    const indexByUrl = new Map<string, number>();
+    const idByIndex = new Int32Array(n);
+    for (let i = 0; i < n; i++) {
+      const row = nodeRows[i]!;
+      indexById.set(row.id, i);
+      indexByUrl.set(row.url, i);
+      idByIndex[i] = row.id;
+    }
+
+    // Build the unique-edge adjacency list. `SELECT DISTINCT` collapses
+    // duplicate (from → to) links inside SQLite, so a page's repeated
+    // nav/footer chrome links to the same target don't inflate its
+    // out-degree — no per-source Set needed on the JS side. `to_url` is a
+    // string, so map it back to a node index; links to external stubs (not
+    // in `urls`) and self-loops are dropped.
+    const outTargets: number[][] = Array.from({ length: n }, () => []);
+    let edgeCount = 0;
+    const edgeRows = this.db
+      .prepare(
+        'SELECT DISTINCT from_url_id AS source, to_url FROM links WHERE is_internal = 1',
+      )
+      .all() as { source: number; to_url: string }[];
+    for (const e of edgeRows) {
+      const si = indexById.get(e.source);
+      if (si === undefined) continue;
+      const ti = indexByUrl.get(e.to_url);
+      if (ti === undefined || ti === si) continue;
+      outTargets[si]!.push(ti);
+      edgeCount++;
+    }
+
+    const upd = this.db.prepare('UPDATE urls SET link_score = ? WHERE id = ?');
+
+    // No internal link graph → no equity signal. Zero the column so stale
+    // scores from a previous crawl don't linger.
+    if (edgeCount === 0) {
+      this.db.exec('BEGIN');
+      try {
+        for (let i = 0; i < n; i++) upd.run(0, idByIndex[i]!);
+        this.db.exec('COMMIT');
+      } catch (err) {
+        this.db.exec('ROLLBACK');
+        throw err;
+      }
+      return { pages: n, iterations: 0 };
+    }
+
+    const outDeg = new Int32Array(n);
+    for (let i = 0; i < n; i++) outDeg[i] = outTargets[i]!.length;
+
+    let pr = new Float64Array(n);
+    pr.fill(1 / n);
+    let next = new Float64Array(n);
+    const base = (1 - damping) / n;
+    let iterationsRun = 0;
+    for (let iter = 0; iter < maxIterations; iter++) {
+      iterationsRun++;
+      // Dangling nodes (no out-links) leak PageRank mass; redistribute it
+      // uniformly so the total stays 1.
+      let dangling = 0;
+      for (let i = 0; i < n; i++) if (outDeg[i]! === 0) dangling += pr[i]!;
+      const danglingShare = (damping * dangling) / n;
+      for (let i = 0; i < n; i++) next[i] = base + danglingShare;
+      for (let i = 0; i < n; i++) {
+        const d = outDeg[i]!;
+        if (d === 0) continue;
+        const share = (damping * pr[i]!) / d;
+        const targets = outTargets[i]!;
+        for (let k = 0; k < d; k++) next[targets[k]!]! += share;
+      }
+      let diff = 0;
+      for (let i = 0; i < n; i++) diff += Math.abs(next[i]! - pr[i]!);
+      const tmp = pr;
+      pr = next;
+      next = tmp;
+      if (diff < 1e-6) break;
+    }
+
+    let maxPr = 0;
+    for (let i = 0; i < n; i++) if (pr[i]! > maxPr) maxPr = pr[i]!;
+    const scale = maxPr > 0 ? 100 / maxPr : 0;
+
+    this.db.exec('BEGIN');
+    try {
+      for (let i = 0; i < n; i++) {
+        upd.run(Math.round(pr[i]! * scale), idByIndex[i]!);
+      }
+      this.db.exec('COMMIT');
+    } catch (err) {
+      this.db.exec('ROLLBACK');
+      throw err;
+    }
+    return { pages: n, iterations: iterationsRun };
+  }
+
  /**
   * Cluster pages by SimHash hamming distance ≤ `thresholdBits` and
   * write `cluster_id` / `cluster_size` back to every URL.
@@ -5483,13 +5635,14 @@ export class ProjectDb {
      lcpTag: string | null;
      lcpResourceUrl: string | null;
      lcpCoverage: number | null;
+      linkScore: number | null;
    }[];
    edges: { source: number; target: number }[];
  } {
    const nodes = this.db
      .prepare(
        `SELECT id, url, status_code, depth, inlinks, indexability,
-                lcp_tag, lcp_resource_url, lcp_coverage
+                lcp_tag, lcp_resource_url, lcp_coverage, link_score
           FROM urls
          WHERE is_external = 0 AND content_kind = 'html'
          ORDER BY inlinks DESC, id ASC
@@ -5505,6 +5658,7 @@ export class ProjectDb {
      lcp_tag: string | null;
      lcp_resource_url: string | null;
      lcp_coverage: number | null;
+      link_score: number | null;
    }[];

    if (nodes.length === 0) return { nodes: [], edges: [] };
@@ -5543,6 +5697,7 @@ export class ProjectDb {
        lcpTag: n.lcp_tag,
        lcpResourceUrl: n.lcp_resource_url,
        lcpCoverage: n.lcp_coverage,
+        linkScore: n.link_score,
      })),
      edges,
    };
@@ -6784,10 +6939,20 @@ export class ProjectDb {
   * triage why it was missed (sitemap-only = drop or fix internal
   * linking; GSC + GA4 = real but unreachable from the crawl seed).
   */
-  orphanPagesCrossSource(
-    limit = 1000,
-  ): { url: string; sources: string[] }[] {
+  orphanPagesCrossSource(limit = 1000): {
+    url: string;
+    sources: string[];
+    gscClicks: number | null;
+    gscImpressions: number | null;
+    ga4Sessions: number | null;
+    lastmod: string | null;
+  }[] {
    const cap = Math.max(1, Math.min(10_000, limit));
+    // The GSC / GA4 / sitemap joins are all 1:1 (url is PK / UNIQUE in each
+    // source table), so pulling their metrics alongside the GROUP_CONCAT of
+    // sources introduces no fan-out. Ordered by traffic (GSC clicks + GA4
+    // sessions) descending so the highest-value orphans — real pages users
+    // reach that the crawl can't — float to the top.
    const rows = this.db
      .prepare(
        `WITH src AS (
@@ -6798,18 +6963,36 @@ export class ProjectDb {
           SELECT url, 'ga4' AS source FROM ga4_results
         )
         SELECT s.url AS url,
-                GROUP_CONCAT(DISTINCT s.source) AS sources
+                GROUP_CONCAT(DISTINCT s.source) AS sources,
+                g.clicks AS gscClicks,
+                g.impressions AS gscImpressions,
+                a.sessions AS ga4Sessions,
+                sm.lastmod AS lastmod
           FROM src s
           LEFT JOIN urls u ON u.url = s.url
+           LEFT JOIN gsc_results g ON g.url = s.url
+           LEFT JOIN ga4_results a ON a.url = s.url
+           LEFT JOIN sitemap_urls sm ON sm.url = s.url
          WHERE u.url IS NULL OR u.status_code IS NULL
          GROUP BY s.url
-          ORDER BY s.url
+          ORDER BY (COALESCE(g.clicks, 0) + COALESCE(a.sessions, 0)) DESC, s.url
          LIMIT ?`,
      )
-      .all(cap) as { url: string; sources: string }[];
+      .all(cap) as {
+      url: string;
+      sources: string;
+      gscClicks: number | null;
+      gscImpressions: number | null;
+      ga4Sessions: number | null;
+      lastmod: string | null;
+    }[];
    return rows.map((r) => ({
      url: r.url,
      sources: r.sources ? r.sources.split(',').sort() : [],
+      gscClicks: r.gscClicks ?? null,
+      gscImpressions: r.gscImpressions ?? null,
+      ga4Sessions: r.ga4Sessions ?? null,
+      lastmod: r.lastmod ?? null,
    }));
  }

@@ -6999,6 +7182,7 @@ export class ProjectDb {
    depth: r.depth,
    inlinks: r.inlinks,
    outlinks: r.outlinks,
+    linkScore: r.link_score,
    imagesCount: r.images_count,
    imagesMissingAlt: r.images_missing_alt,
    redirectTarget: r.redirect_target,
@@ -7148,6 +7332,7 @@ const VALID_SORT_COLUMNS = new Set([
  'depth',
  'inlinks',
  'outlinks',
+  'link_score',
  'crawled_at',
  'indexability',
  'content_kind',
@@ -1,6 +1,6 @@
 {
  "name": "@freecrawl/shared-types",
-  "version": "0.8.1",
+  "version": "0.8.2",
  "private": true,
  "type": "module",
  "main": "./dist/index.js",
@@ -277,6 +277,8 @@ export interface CrawlUrlRow {
  depth: number;
  inlinks: number;
  outlinks: number;
+  /** Internal PageRank / link score, 0..100 (100 = most-linked page). Null until the post-crawl pass runs. */
+  linkScore: number | null;
  imagesCount: number;
  imagesMissingAlt: number;
  /** Number of `<img>` tags with `alt=""` (decorative, distinct from missing alt). */
@@ -1011,6 +1013,8 @@ export interface CrawlConfig {
   */
  /** Recompute `inlinks` count per URL after the crawl. */
  analyseInlinks: boolean;
+  /** Compute internal PageRank / link score (0..100) over the link graph. */
+  analyseLinkScore: boolean;
  /** Walk redirect chains, fill `redirect_chain_length` / `redirect_loop`. */
  analyseRedirectChains: boolean;
  /** Hreflang reciprocity + invalid code + target health. */
@@ -1980,6 +1984,7 @@ export const DEFAULT_CRAWL_CONFIG: CrawlConfig = {
  followNofollow: false,
  followJsRedirects: false,
  analyseInlinks: true,
+  analyseLinkScore: true,
  analyseRedirectChains: true,
  analyseHreflang: true,
  analyseDuplicates: true,
@@ -225,6 +225,10 @@ export const IPC = {
   *  `projectSaveDir` pref when set, otherwise the OS Documents folder.
   *  Used by Settings → Storage to display the active path. */
  defaultProjectDir: 'app:default-project-dir',
+  /** Storage mode the active DB connection is actually running in
+   *  (`disk` | `ram`). Reflects what was resolved at launch, which may
+   *  differ from the `storageMode` pref until the next restart. */
+  storageModeActive: 'app:storage-mode-active',
  /** Faz 7 — Google PageSpeed Insights integration.
   *  `pagespeedQuery` lists crawled internal HTML pages joined with any
   *  stored audit results; `pagespeedRun` audits a user-selected set of
@@ -651,6 +655,10 @@ export interface GraphNode {
  lcpTag: string | null;
  lcpResourceUrl: string | null;
  lcpCoverage: number | null;
+  /** Internal PageRank / link score, 0..100 (100 = most-linked page).
+   *  Null until the post-crawl link-score pass runs. Powers the
+   *  "By Link Score" visualization colour mode. */
+  linkScore: number | null;
 }

 export interface GraphEdge {
@@ -983,6 +991,14 @@ export interface OrphanCrossSourceRow {
  url: string;
  /** Which truth sources mentioned this URL — `sitemap`, `gsc`, `ga4`. */
  sources: string[];
+  /** Search Console clicks for this URL, when it came from GSC. */
+  gscClicks: number | null;
+  /** Search Console impressions for this URL, when it came from GSC. */
+  gscImpressions: number | null;
+  /** Analytics 4 sessions for this URL, when it came from GA4. */
+  ga4Sessions: number | null;
+  /** `<lastmod>` from the sitemap, when it came from the sitemap. */
+  lastmod: string | null;
 }

 /** One row in the Server Stack report — `Server` response-header rollup. */
@@ -1513,6 +1529,9 @@ export interface FreeCrawlApi {
  pickDirectory(input?: { title?: string; defaultPath?: string }): Promise<string | null>;
  /** Resolved default save directory for new projects (Documents fallback when unset). */
  defaultProjectDir(): Promise<string>;
+  /** Storage mode the active DB connection is running in (`disk` | `ram`).
+   *  May lag the `storageMode` pref until the next launch. */
+  storageModeActive(): Promise<'disk' | 'ram'>;
  /** Faz 7 — list crawled internal HTML pages joined with their stored
   *  PageSpeed Insights audit results. */
  pagespeedQuery(input: PagespeedQueryInput): Promise<PagespeedQueryResult>;