feat: CrUX real-user vitals + LanguageTool spelling & grammar

This commit is contained in:
kemalai committed 2026-07-09 13:09:52 +03:00
1 parent 311ba28c73
commit f870767cb9
36 files changed
+3648 -29

No files matched your search

+7 -7
View File
@@ -29,11 +29,11 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
## ✨ What FreeCrawl Does
FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 30 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), and a 10-rule custom extraction engine — **CSS selectors, XPath, regex, and JSONPath (for JSON API responses)** — with a **live Preview dialog and JSON import / export** for sharing rule sets.
FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 32 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), **spelling & grammar checking via LanguageTool** (per-page findings with in-context highlighting and a custom dictionary), and a 10-rule custom extraction engine — **CSS selectors, XPath, regex, and JSONPath (for JSON API responses)** — with a **live Preview dialog and JSON import / export** for sharing rule sets.
The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 16-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a **By-LCP above-the-fold colour overlay**, a **By-Link-Score (internal PageRank) colour overlay** that shows where internal link equity concentrates, and a click-to-trace **Crawl Path Report** that highlights the shortest discovery path from the homepage to any page), a **standalone Log File Analyzer window** (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a `.seoproject` file association for OS double-click.
The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 17-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics, Spelling & Grammar), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a **By-LCP above-the-fold colour overlay**, a **By-Link-Score (internal PageRank) colour overlay** that shows where internal link equity concentrates, and a click-to-trace **Crawl Path Report** that highlights the shortest discovery path from the homepage to any page), a **standalone Log File Analyzer window** (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a `.seoproject` file association for OS double-click.
Integrations cover **Google Search Console** (clicks, impressions, CTR, position + URL Inspection coverage), **Google Analytics 4** (sessions, users, bounce, engagement), **PageSpeed Insights** Lighthouse audits, custom **AI prompts via OpenAI / Anthropic Claude / local Ollama** with `{url}`/`{title}`/`{description}`/`{h1}`/`{body}` variables, and **SEO authority providers** (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an **in-app step-by-step Setup Guide modal** in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified **Export Crawl Data dialog** (Excel `.xlsx` / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to **Google Sheets** and **BigQuery**. Project files can be saved as **password-protected encrypted snapshots** (`.seoproject.enc`, AES-256-GCM + PBKDF2). The **MCP server** exposes **91 tools** that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL **Duplicates** view, and a custom-CSS theme override round out the suite. Everything runs **fully local** — no telemetry, no cloud, MIT-style license.
Integrations cover **Google Search Console** (clicks, impressions, CTR, position + URL Inspection coverage), **Google Analytics 4** (sessions, users, bounce, engagement), **PageSpeed Insights** Lighthouse lab audits, **Chrome UX Report (CrUX)** real-user field Core Web Vitals (p75 LCP / INP / CLS / FCP / TTFB, phone + desktop), **LanguageTool** spelling & grammar checking (public API, self-hosted, or Premium), custom **AI prompts via OpenAI / Anthropic Claude / local Ollama** with `{url}`/`{title}`/`{description}`/`{h1}`/`{body}` variables, and **SEO authority providers** (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an **in-app step-by-step Setup Guide modal** in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified **Export Crawl Data dialog** (Excel `.xlsx` / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to **Google Sheets** and **BigQuery**. Project files can be saved as **password-protected encrypted snapshots** (`.seoproject.enc`, AES-256-GCM + PBKDF2). The **MCP server** exposes **96 tools** that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL **Duplicates** view, and a custom-CSS theme override round out the suite. Everything runs **fully local** — no telemetry, no cloud, MIT-style license.
<br />
@@ -205,18 +205,18 @@ FreeCrawl ships an **MCP (Model Context Protocol) server** that exposes the acti
1. **Read-only data access** to the SQLite project — runs alongside the desktop app without contention (WAL allows concurrent readers).
2. **Live crawl control** — when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to `<userData>/mcp-bridge.json` on app launch).
**91 tools** — every action a human can take in the desktop is callable from an agent. Representative groups:
**96 tools** — every action a human can take in the desktop is callable from an agent. Representative groups:
| Group | Tools |
| :--- | :--- |
| 📊 **Top-level data** | `get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail` |
| 🔬 **Per-URL detail sub-tabs** | `get_url_source` (raw + rendered + screenshot paths), `get_url_inlinks`, `get_url_outlinks`, `get_url_images`, `get_url_headers`, `get_url_duplicates`, `get_url_analytics`, `get_url_cert` |
| 🧩 **Integration rows** | `query_gsc`, `query_ga4`, `query_pagespeed`, `query_ai`, `query_seo` |
| 🧩 **Integration rows** | `query_gsc`, `query_ga4`, `query_pagespeed`, `query_crux`, `query_spelling`, `get_url_spelling`, `query_ai`, `query_seo` |
| 🔎 **Specialised queries** | `query_images`, `query_broken_links`, `list_duplicate_clusters` |
| 📈 **Reports** | `report_status_code_histogram`, `report_indexability_distribution`, `report_content_kind_distribution`, `report_depth_histogram`, `report_response_time_histogram`, `report_inlinks_histogram`, `report_word_count_histogram`, `report_url_length_histogram`, `report_top_urls_by`, `report_top_anchor_texts`, `report_external_domain_health`, `report_image_weight_per_page`, `report_link_position_breakdown`, `report_pages_per_directory`, `report_word_count_per_directory`, `report_sitemap_orphans`, `report_orphan_cross_source`, `report_analytics_coverage`, `report_server_header_breakdown`, `report_query_string_variants`, `report_sitemap_priority_mismatch`, `report_top_words` |
| 📤 **Exports & mutations** | `export_csv`, `export_json`, `export_xml`, `export_html_report`, `export_tabular`, `export_bulk`, `export_broken_links`, `export_images`, `sitemap_generate`, `sitemap_validate`, `robots_test`, `robots_validate`, `compare_load`, `crawl_add_url`, `respider_urls`, `remove_urls`, `data_delete_by_domain`, `graph_snapshot`, `get_crawl_path`, `url_rewrite_preview`, `extraction_preview` |
| 🔐 **Project files** | `project_save_as`, `project_save_encrypted`, `project_open_encrypted` |
| 🔌 **Integration fetch** | `google_auth_status`, `gsc_list_sites`, `gsc_fetch`, `ga4_list_properties`, `ga4_fetch`, `pagespeed_run`, `ai_run`, `seo_run`, `get_fetch_progress`, `cancel_fetch` |
| 🔌 **Integration fetch** | `google_auth_status`, `gsc_list_sites`, `gsc_fetch`, `ga4_list_properties`, `ga4_fetch`, `pagespeed_run`, `crux_run`, `spelling_run`, `ai_run`, `seo_run`, `get_fetch_progress`, `cancel_fetch` |
| ⚙ **Config & schedule** | `get_crawl_config`, `set_crawl_config`, `schedule_get`, `schedule_set` |
| 📁 **Project management** | `list_projects`, `set_project`, `current_project` |
| 🕷 **Crawl control** (desktop must be open) | `start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `clear_crawl`, `get_crawl_progress`, `get_desktop_project` |
@@ -398,7 +398,7 @@ graph LR
## 📈 Status
> [!NOTE]
> **Active development.** All 29 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, **Cytoscape visualization (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP above-the-fold overlay + internal-PageRank link-score overlay + click-to-trace Crawl Path Report)**, **Basic / Digest / Bearer + form-based login (HTTP multi-step + Playwright browser-driven SPA login)**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **traffic-ranked cross-source orphan detection (sitemap + GSC + GA4)**, **internal PageRank link scoring**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, **RAM-only (in-memory) storage mode**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **standalone Log File Analyzer (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, crawl × log orphan detection, 40+ bot detection with reverse-DNS verification, per-bot URL filter, CSV / Excel export, and a headless `analyze-logs` CLI)**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
> **Active development.** All 31 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, CrUX, Spelling, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, **Cytoscape visualization (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP above-the-fold overlay + internal-PageRank link-score overlay + click-to-trace Crawl Path Report)**, **Basic / Digest / Bearer + form-based login (HTTP multi-step + Playwright browser-driven SPA login)**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **Chrome UX Report (CrUX) real-user field Core Web Vitals**, **LanguageTool spelling & grammar checking (custom dictionary, picky rule level, in-context highlighting)**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **traffic-ranked cross-source orphan detection (sitemap + GSC + GA4)**, **internal PageRank link scoring**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, **RAM-only (in-memory) storage mode**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **standalone Log File Analyzer (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, crawl × log orphan detection, 40+ bot detection with reverse-DNS verification, per-bot URL filter, CSV / Excel export, and a headless `analyze-logs` CLI)**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
>
> **Upcoming (V2):** Plugin system, Light theme, Multi-window, Code-signing + auto-update.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/cli",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"type": "module",
"bin": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/desktop",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"description": "FreeCrawl SEO — desktop application",
"homepage": "https://github.com/kemalai/FreeCrawl-SEO-Tool",
+303
View File
@@ -0,0 +1,303 @@
/**
* Chrome UX Report (CrUX) API client.
*
* CrUX returns the 75th-percentile Core Web Vitals of real Chrome users
* over a rolling 28-day window (`chromeuxreport.googleapis.com`). Unlike
* PageSpeed Insights (synthetic lab data from one Lighthouse run), this is
* *field* data — how the page actually performs for visitors. It is fast
* (sub-second), requires an API key (no keyless mode), and returns HTTP
* 404 when a URL/origin has too little real-user traffic ("no data", not
* an error).
*
* Two layers, mirroring `pagespeed.ts`:
* - `fetchCruxRecord` — one URL + form factor → metrics (never throws;
* a 404 becomes a `nodata` result, any other failure an `error`).
* - `runCruxBatch` — a small concurrency pool over many (url, formFactor)
* items with progress + cooperative cancellation.
*/
import type { CruxMetrics, CruxFormFactor } from '@freecrawl/shared-types';
import { Agent, ProxyAgent, fetch as undiciFetch, type Dispatcher } from 'undici';
import * as logger from './logger.js';
const CRUX_ENDPOINT =
'https://chromeuxreport.googleapis.com/v1/records:queryRecord';
/** CrUX responds fast; cap a single request at 30 s. */
const REQUEST_TIMEOUT_MS = 30_000;
/** CrUX is quick and key-gated, so we can parallelise more than PSI. */
const CONCURRENCY = 8;
/** The CrUX metrics we request per record. */
const CRUX_METRICS = [
'largest_contentful_paint',
'cumulative_layout_shift',
'interaction_to_next_paint',
'first_contentful_paint',
'experimental_time_to_first_byte',
];
/**
* Dedicated undici dispatcher for CrUX requests. The crawler installs a
* process-global dispatcher with a 10 s `headersTimeout`; a plain fetch
* inherits it. CrUX is fast so that rarely bites, but under batch
* concurrency a slow response could still be strangled — use our own
* dispatcher (with proxy support to match the crawler) to be safe.
*/
function createCruxDispatcher(): Dispatcher {
const timeouts = {
headersTimeout: REQUEST_TIMEOUT_MS + 5_000,
bodyTimeout: REQUEST_TIMEOUT_MS + 5_000,
};
const proxy =
process.env['HTTPS_PROXY'] ??
process.env['https_proxy'] ??
process.env['HTTP_PROXY'] ??
process.env['http_proxy'] ??
null;
if (proxy) return new ProxyAgent({ uri: proxy, ...timeouts });
return new Agent({ ...timeouts, connect: { autoSelectFamily: true } });
}
const cruxDispatcher = createCruxDispatcher();
function baseMetrics(
fetchedAt: string,
status: CruxMetrics['status'],
error: string | null,
): CruxMetrics {
return {
lcp: null,
cls: null,
inp: null,
fcp: null,
ttfb: null,
status,
error,
collectionPeriod: null,
fetchedAt,
};
}
/** Pull the p75 value out of a CrUX metric object (may arrive as a string). */
function p75(
metrics: Record<string, unknown> | undefined,
key: string,
decimals = 0,
): number | null {
const m = metrics?.[key] as
| { percentiles?: { p75?: unknown } }
| undefined;
const raw = m?.percentiles?.p75;
const n = typeof raw === 'string' ? Number(raw) : raw;
if (typeof n !== 'number' || !Number.isFinite(n)) return null;
const factor = 10 ** decimals;
return Math.round(n * factor) / factor;
}
/** Format a CrUX `{year,month,day}` date part as `YYYY-MM-DD`. */
function formatDate(d: unknown): string | null {
const date = d as { year?: number; month?: number; day?: number } | undefined;
if (!date || typeof date.year !== 'number') return null;
const mm = String(date.month ?? 1).padStart(2, '0');
const dd = String(date.day ?? 1).padStart(2, '0');
return `${date.year}-${mm}-${dd}`;
}
/**
* Fetch one CrUX record for one URL + form factor. Never throws: a 404
* (insufficient real-user data) resolves to a `nodata` result and any
* other failure to an `error` result, so a single URL never aborts a batch.
*/
export async function fetchCruxRecord(
url: string,
formFactor: CruxFormFactor,
apiKey: string,
): Promise<CruxMetrics> {
const fetchedAt = new Date().toISOString();
const body = JSON.stringify({
url,
formFactor: formFactor === 'phone' ? 'PHONE' : 'DESKTOP',
metrics: CRUX_METRICS,
});
try {
const res = await undiciFetch(
`${CRUX_ENDPOINT}?key=${encodeURIComponent(apiKey)}`,
{
method: 'POST',
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body,
dispatcher: cruxDispatcher,
},
);
const json = (await res.json().catch(() => null)) as Record<
string,
unknown
> | null;
if (res.status === 404) {
// CrUX has no aggregated data for this URL at this form factor —
// expected for low-traffic pages, not a failure.
return baseMetrics(fetchedAt, 'nodata', null);
}
if (!res.ok) {
const apiErr = json?.['error'] as { message?: string } | undefined;
return baseMetrics(
fetchedAt,
'error',
apiErr?.message ?? `CrUX API returned HTTP ${res.status}`,
);
}
if (!json) {
return baseMetrics(fetchedAt, 'error', 'CrUX API returned an empty response');
}
const record = json['record'] as Record<string, unknown> | undefined;
const metrics = record?.['metrics'] as Record<string, unknown> | undefined;
if (!metrics) {
return baseMetrics(fetchedAt, 'nodata', null);
}
const period = record?.['collectionPeriod'] as
| { lastDate?: unknown }
| undefined;
return {
lcp: p75(metrics, 'largest_contentful_paint'),
cls: p75(metrics, 'cumulative_layout_shift', 3),
inp: p75(metrics, 'interaction_to_next_paint'),
fcp: p75(metrics, 'first_contentful_paint'),
ttfb: p75(metrics, 'experimental_time_to_first_byte'),
status: 'ok',
error: null,
collectionPeriod: formatDate(period?.lastDate),
fetchedAt,
};
} catch (err) {
const name = (err as { name?: string } | null)?.name;
const cause = (err as { cause?: unknown } | null)?.cause as
| { code?: string; message?: string }
| undefined;
const code = cause?.code;
const isTimeout =
name === 'TimeoutError' ||
name === 'AbortError' ||
code === 'UND_ERR_HEADERS_TIMEOUT' ||
code === 'UND_ERR_BODY_TIMEOUT' ||
code === 'UND_ERR_CONNECT_TIMEOUT';
const message = isTimeout
? `CrUX request timed out after ${REQUEST_TIMEOUT_MS / 1000}s`
: err instanceof Error
? cause?.message
? `${err.message} (${cause.message})`
: err.message
: String(err);
return baseMetrics(fetchedAt, 'error', message);
}
}
/** One (url, formFactor) fetch unit of work. */
export interface CruxBatchItem {
url: string;
formFactor: CruxFormFactor;
}
export interface CruxBatchOptions {
items: CruxBatchItem[];
apiKey: string;
isCancelled: () => boolean;
onResult: (
url: string,
formFactor: CruxFormFactor,
metrics: CruxMetrics,
) => void;
onProgress: (done: number, total: number, currentUrl: string | null) => void;
}
export interface CruxBatchResult {
completed: number;
failed: number;
cancelled: boolean;
}
/**
* Run a CrUX fetch over many items with a small concurrency pool.
* Cancellation gates new items only — records already in flight finish
* and are persisted. A `nodata` record counts as failed (nothing stored
* that the user can act on) but is not logged as an error.
*/
export async function runCruxBatch(
opts: CruxBatchOptions,
): Promise<CruxBatchResult> {
const { items, apiKey, isCancelled, onResult, onProgress } = opts;
const total = items.length;
let done = 0;
let completed = 0;
let failed = 0;
let cursor = 0;
const ERROR_LOG_LIMIT_PER_KIND = 3;
const errorCounts = new Map<string, number>();
function recordFailure(url: string, message: string): void {
const prev = errorCounts.get(message) ?? 0;
errorCounts.set(message, prev + 1);
if (prev < ERROR_LOG_LIMIT_PER_KIND) {
logger.log('warn', 'crux', `${url} — ${message}`);
}
}
onProgress(0, total, null);
const worker = async (): Promise<void> => {
for (;;) {
if (isCancelled()) return;
const index = cursor++;
if (index >= total) return;
const item = items[index];
if (!item) return;
onProgress(done, total, item.url);
const metrics = await fetchCruxRecord(item.url, item.formFactor, apiKey);
try {
onResult(item.url, item.formFactor, metrics);
} catch (err) {
logger.log(
'error',
'crux',
`failed to persist CrUX record for ${item.url}: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
done++;
if (metrics.status === 'ok') {
completed++;
} else {
failed++;
if (metrics.status === 'error') {
recordFailure(item.url, metrics.error ?? 'Unknown error');
}
}
onProgress(done, total, null);
}
};
const poolSize = Math.min(CONCURRENCY, Math.max(1, total));
await Promise.all(Array.from({ length: poolSize }, () => worker()));
const cancelled = isCancelled() && done < total;
logger.log(
'info',
'crux',
`batch finished — ${completed} with data, ${failed} no-data/failed${
cancelled ? ', cancelled early' : ''
} (${done}/${total})`,
);
if (errorCounts.size > 0) {
const sorted = [...errorCounts.entries()].sort((a, b) => b[1] - a[1]);
for (const [message, count] of sorted) {
logger.log('warn', 'crux', `failure summary — ${count}× "${message}"`);
}
}
return { completed, failed, cancelled };
}
@@ -70,6 +70,9 @@ const ALLOWED_METHODS = new Set<string>([
'queryImages',
'queryBrokenLinks',
'queryPagespeed',
'queryCrux',
'querySpelling',
'getSpellingMatches',
'queryGsc',
'queryGa4',
'queryAi',
+489 -1
View File
@@ -103,6 +103,17 @@ import {
type PagespeedRunInput,
type PagespeedRunResult,
type PagespeedStrategy,
type CruxQueryInput,
type CruxQueryResult,
type CruxRunInput,
type CruxRunResult,
type CruxFormFactor,
type SpellingQueryInput,
type SpellingQueryResult,
type SpellingRunInput,
type SpellingRunResult,
type SpellingMatchesResult,
type SpellingLevel,
type GoogleAuthState,
type GoogleAuthResult,
type GscListSitesResult,
@@ -177,6 +188,7 @@ import {
analyzeLogFile,
buildLogExportXlsx,
buildLogExportCsv,
extractProseText,
type LogAnalysisResult,
type LogExportTables,
} from '@freecrawl/core';
@@ -193,6 +205,12 @@ import {
resolveCredentials,
} from './credentials.js';
import { runPagespeedBatch, type PagespeedBatchItem } from './pagespeed.js';
import { runCruxBatch, type CruxBatchItem } from './crux.js';
import {
runSpellingBatch,
PUBLIC_LT_ENDPOINT,
type SpellingCheckOptions,
} from './languagetool.js';
import { startAuth, getAuthState, revokeAuth } from './google-oauth.js';
import { listSites, querySearchAnalytics, inspectUrl } from './gsc.js';
import { exportCategoryToSheets } from './sheets-export.js';
@@ -285,6 +303,14 @@ function idleFetchSnapshot(): FetchRunSnapshot {
let pagespeedRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
let aiRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
let seoRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
/** CrUX run guard + MCP poll snapshot — same shape as the PSI guard. */
let cruxRunActive = false;
let cruxCancelRequested = false;
let cruxRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
/** LanguageTool (spelling & grammar) run guard + MCP poll snapshot. */
let spellingRunActive = false;
let spellingCancelRequested = false;
let spellingRunSnapshot: FetchRunSnapshot = idleFetchSnapshot();
/** Faz 7 — GSC URL Inspection run guard. */
let gscInspectRunActive = false;
let gscInspectCancelRequested = false;
@@ -2921,6 +2947,361 @@ function registerIpc(): void {
}
});
// ---- Chrome UX Report (CrUX) — real-user field metrics ----
ipcMain.handle(
IPC.cruxQuery,
async (_e, input: CruxQueryInput): Promise<CruxQueryResult> => {
const params = {
limit: input.limit,
offset: input.offset,
search: input.search,
filter: input.filter,
};
return callReaderOrFallback<CruxQueryResult>(
'queryCrux',
[params],
() => getDb().queryCrux(params),
);
},
);
// Shared CrUX batch orchestrator — driven by both the renderer's
// `cruxRun` IPC and the MCP bridge's `crux-run` action. Mirrors
// `runPagespeedJob`; updates `cruxRunSnapshot` throughout.
async function runCruxJob(input: CruxRunInput): Promise<CruxRunResult> {
if (cruxRunActive) {
return { completed: 0, failed: 0, cancelled: true };
}
const urls = Array.from(
new Set(
(input.urls ?? []).filter(
(u): u is string => typeof u === 'string' && u.length > 0,
),
),
);
if (urls.length === 0) {
cruxRunSnapshot = {
...idleFetchSnapshot(),
finishedAt: new Date().toISOString(),
};
return { completed: 0, failed: 0, cancelled: false };
}
const formFactors: CruxFormFactor[] =
input.formFactor === 'both'
? ['phone', 'desktop']
: [input.formFactor];
const items: CruxBatchItem[] = [];
for (const url of urls) {
for (const formFactor of formFactors) items.push({ url, formFactor });
}
const apiKeyRaw = resolveCredentials('crux')['apiKey'];
const apiKey =
apiKeyRaw && apiKeyRaw.trim().length > 0 ? apiKeyRaw.trim() : '';
if (!apiKey) {
// CrUX has no keyless mode — surface a clear failure instead of a
// batch of 400s.
cruxRunSnapshot = {
...idleFetchSnapshot(),
finishedAt: new Date().toISOString(),
error: 'No CrUX API key configured (Settings → Integrations → Chrome UX Report).',
};
logger.log(
'warn',
'crux',
'run aborted — no API key (Settings → Integrations → Chrome UX Report)',
);
return { completed: 0, failed: items.length, cancelled: false };
}
cruxRunActive = true;
cruxCancelRequested = false;
cruxRunSnapshot = {
...idleFetchSnapshot(),
running: true,
total: items.length,
startedAt: new Date().toISOString(),
};
const runDb = getDb();
logger.log(
'info',
'crux',
`run started — ${items.length} record(s): ${urls.length} URL(s) × ${formFactors.length} form factor`,
);
try {
const result = await runCruxBatch({
items,
apiKey,
isCancelled: () => cruxCancelRequested,
onResult: (url, formFactor, metrics) => {
try {
runDb.upsertCruxResult(url, formFactor, metrics);
} catch (err) {
logger.log(
'error',
'crux',
`DB write failed for ${url}: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
if (metrics.status === 'ok') cruxRunSnapshot.completed++;
else cruxRunSnapshot.failed++;
},
onProgress: (done, total, currentUrl) => {
cruxRunSnapshot.done = done;
cruxRunSnapshot.total = total;
cruxRunSnapshot.currentUrl = currentUrl;
mainWindow?.webContents.send(IPC.cruxProgress, {
done,
total,
currentUrl,
running: true,
});
},
});
cruxRunSnapshot = {
...cruxRunSnapshot,
running: false,
done: result.completed + result.failed,
completed: result.completed,
failed: result.failed,
cancelled: result.cancelled,
currentUrl: null,
finishedAt: new Date().toISOString(),
};
mainWindow?.webContents.send(IPC.cruxProgress, {
done: result.completed + result.failed,
total: items.length,
currentUrl: null,
running: false,
});
fireDataChanged();
return result;
} catch (err) {
cruxRunSnapshot = {
...cruxRunSnapshot,
running: false,
currentUrl: null,
finishedAt: new Date().toISOString(),
error: err instanceof Error ? err.message : String(err),
};
throw err;
} finally {
cruxRunActive = false;
cruxCancelRequested = false;
}
}
ipcMain.handle(
IPC.cruxRun,
async (_e, input: CruxRunInput): Promise<CruxRunResult> =>
runCruxJob(input),
);
ipcMain.handle(IPC.cruxCancel, () => {
if (cruxRunActive) {
cruxCancelRequested = true;
logger.log('info', 'crux', 'run cancellation requested');
}
});
// ---- Spelling & Grammar (LanguageTool) ----
ipcMain.handle(
IPC.spellingQuery,
async (_e, input: SpellingQueryInput): Promise<SpellingQueryResult> => {
const params = {
limit: input.limit,
offset: input.offset,
search: input.search,
filter: input.filter,
};
return callReaderOrFallback<SpellingQueryResult>(
'querySpelling',
[params],
() => getDb().querySpelling(params),
);
},
);
ipcMain.handle(
IPC.spellingMatches,
async (_e, url: string): Promise<SpellingMatchesResult | null> =>
callReaderOrFallback<SpellingMatchesResult | null>(
'getSpellingMatches',
[url],
() => getDb().getSpellingMatches(url),
),
);
/** Resolve the LanguageTool endpoint + credentials + user preferences. */
function resolveSpellingOptions(): SpellingCheckOptions {
loadPrefs();
const creds = resolveCredentials('languagetool');
const endpointRaw = (creds['endpoint'] ?? '').trim();
const level: SpellingLevel =
prefsCache['spellingLevel'] === 'picky' ? 'picky' : 'default';
const ignoreRaw = prefsCache['spellingIgnoreWords'];
const ignoreWords = new Set<string>(
(typeof ignoreRaw === 'string' ? ignoreRaw : '')
.split(/[\s,;\n]+/)
.map((w) => w.trim().toLowerCase())
.filter((w) => w.length > 0),
);
return {
endpoint: endpointRaw.length > 0 ? endpointRaw : PUBLIC_LT_ENDPOINT,
username: (creds['username'] ?? '').trim() || undefined,
apiKey: (creds['apiKey'] ?? '').trim() || undefined,
level,
ignoreWords,
};
}
/**
* Resolve one page's prose for the checker. Prefers the crawl-time raw
* HTML snapshot (`url_sources.body`); when body snapshots were disabled
* the page is re-fetched once over HTTP. Either way the HTML is reduced
* to prose (chrome, script/style and code blocks stripped) before it
* leaves the machine.
*/
async function loadSpellingPage(
db: ProjectDb,
url: string,
): Promise<{ text: string | null; lang: string | null }> {
const stored = db.getUrlBodyAndLang(url);
const lang = stored?.lang ?? null;
if (stored?.body && stored.body.length > 0) {
return { text: extractProseText(stored.body), lang };
}
try {
const res = await undiciFetch(url, {
signal: AbortSignal.timeout(30_000),
headers: { Accept: 'text/html,application/xhtml+xml' },
});
if (!res.ok) return { text: null, lang };
const html = await res.text();
return { text: extractProseText(html), lang };
} catch {
return { text: null, lang };
}
}
// Shared LanguageTool batch orchestrator — driven by both the renderer's
// `spellingRun` IPC and the MCP bridge's `spelling-run` action.
async function runSpellingJob(
input: SpellingRunInput,
): Promise<SpellingRunResult> {
if (spellingRunActive) {
return { completed: 0, failed: 0, cancelled: true };
}
const urls = Array.from(
new Set(
(input.urls ?? []).filter(
(u): u is string => typeof u === 'string' && u.length > 0,
),
),
);
if (urls.length === 0) {
spellingRunSnapshot = {
...idleFetchSnapshot(),
finishedAt: new Date().toISOString(),
};
return { completed: 0, failed: 0, cancelled: false };
}
const check = resolveSpellingOptions();
spellingRunActive = true;
spellingCancelRequested = false;
spellingRunSnapshot = {
...idleFetchSnapshot(),
running: true,
total: urls.length,
startedAt: new Date().toISOString(),
};
const runDb = getDb();
logger.log(
'info',
'languagetool',
`run started — ${urls.length} page(s) against ${check.endpoint} (level: ${check.level}, ${check.ignoreWords.size} ignored word(s))`,
);
try {
const result = await runSpellingBatch({
urls,
check,
loadPage: (url) => loadSpellingPage(runDb, url),
isCancelled: () => spellingCancelRequested,
onResult: (url, r) => {
try {
runDb.upsertSpellingResult(url, r);
} catch (err) {
logger.log(
'error',
'languagetool',
`DB write failed for ${url}: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
if (r.status === 'ok') spellingRunSnapshot.completed++;
else spellingRunSnapshot.failed++;
},
onProgress: (done, total, currentUrl) => {
spellingRunSnapshot.done = done;
spellingRunSnapshot.total = total;
spellingRunSnapshot.currentUrl = currentUrl;
mainWindow?.webContents.send(IPC.spellingProgress, {
done,
total,
currentUrl,
running: true,
});
},
});
spellingRunSnapshot = {
...spellingRunSnapshot,
running: false,
done: result.completed + result.failed,
completed: result.completed,
failed: result.failed,
cancelled: result.cancelled,
currentUrl: null,
finishedAt: new Date().toISOString(),
};
mainWindow?.webContents.send(IPC.spellingProgress, {
done: result.completed + result.failed,
total: urls.length,
currentUrl: null,
running: false,
});
fireDataChanged();
return result;
} catch (err) {
spellingRunSnapshot = {
...spellingRunSnapshot,
running: false,
currentUrl: null,
finishedAt: new Date().toISOString(),
error: err instanceof Error ? err.message : String(err),
};
throw err;
} finally {
spellingRunActive = false;
spellingCancelRequested = false;
}
}
ipcMain.handle(
IPC.spellingRun,
async (_e, input: SpellingRunInput): Promise<SpellingRunResult> =>
runSpellingJob(input),
);
ipcMain.handle(IPC.spellingCancel, () => {
if (spellingRunActive) {
spellingCancelRequested = true;
logger.log('info', 'languagetool', 'run cancellation requested');
}
});
// V1 Faz 7 — Google OAuth keystone. The interactive consent flow runs
// entirely in the main process (loopback HTTP server + system
// browser); the renderer only ever sees the redacted GoogleAuthState.
@@ -4751,6 +5132,91 @@ function registerIpc(): void {
};
},
'crux-run': async (input) => {
const i = (input ?? {}) as {
urls?: unknown;
category?: unknown;
search?: unknown;
limit?: unknown;
formFactor?: unknown;
};
if (cruxRunActive) {
return {
started: false,
reason:
'A CrUX run is already in progress — poll get_fetch_progress or cancel_fetch first.',
state: cruxRunSnapshot,
};
}
const { urls, truncated } = resolveActionUrls(i);
if (urls.length === 0) {
return {
started: false,
reason:
'No URLs matched — pass an explicit `urls` array or a `category` that has rows in this project.',
state: cruxRunSnapshot,
};
}
const formFactor =
i.formFactor === 'desktop'
? ('desktop' as const)
: i.formFactor === 'both'
? ('both' as const)
: ('phone' as const);
void runCruxJob({ urls, formFactor }).catch((err) => {
logger.log(
'error',
'crux',
`MCP-triggered run failed: ${err instanceof Error ? err.message : String(err)}`,
);
});
return {
started: cruxRunActive,
total: cruxRunSnapshot.total,
truncated,
state: cruxRunSnapshot,
};
},
'spelling-run': async (input) => {
const i = (input ?? {}) as {
urls?: unknown;
category?: unknown;
search?: unknown;
limit?: unknown;
};
if (spellingRunActive) {
return {
started: false,
reason:
'A spelling run is already in progress — poll get_fetch_progress or cancel_fetch first.',
state: spellingRunSnapshot,
};
}
const { urls, truncated } = resolveActionUrls(i);
if (urls.length === 0) {
return {
started: false,
reason:
'No URLs matched — pass an explicit `urls` array or a `category` that has rows in this project.',
state: spellingRunSnapshot,
};
}
void runSpellingJob({ urls }).catch((err) => {
logger.log(
'error',
'languagetool',
`MCP-triggered run failed: ${err instanceof Error ? err.message : String(err)}`,
);
});
return {
started: spellingRunActive,
total: spellingRunSnapshot.total,
truncated,
state: spellingRunSnapshot,
};
},
'ai-run': async (input) => {
const i = (input ?? {}) as {
provider?: unknown;
@@ -4875,10 +5341,18 @@ function registerIpc(): void {
const kind = (input as { kind?: unknown } | null)?.kind;
const all = {
pagespeed: pagespeedRunSnapshot,
crux: cruxRunSnapshot,
spelling: spellingRunSnapshot,
ai: aiRunSnapshot,
seo: seoRunSnapshot,
};
if (kind === 'pagespeed' || kind === 'ai' || kind === 'seo') {
if (
kind === 'pagespeed' ||
kind === 'crux' ||
kind === 'spelling' ||
kind === 'ai' ||
kind === 'seo'
) {
return { [kind]: all[kind] };
}
return all;
@@ -4902,11 +5376,25 @@ function registerIpc(): void {
if (was) seoCancelRequested = true;
cancelled.seo = { wasRunning: was };
};
const cancelCrux = (): void => {
const was = cruxRunActive;
if (was) cruxCancelRequested = true;
cancelled.crux = { wasRunning: was };
};
const cancelSpelling = (): void => {
const was = spellingRunActive;
if (was) spellingCancelRequested = true;
cancelled.spelling = { wasRunning: was };
};
if (kind === 'pagespeed') cancelPagespeed();
else if (kind === 'crux') cancelCrux();
else if (kind === 'spelling') cancelSpelling();
else if (kind === 'ai') cancelAi();
else if (kind === 'seo') cancelSeo();
else {
cancelPagespeed();
cancelCrux();
cancelSpelling();
cancelAi();
cancelSeo();
}
+449
View File
@@ -0,0 +1,449 @@
/**
* LanguageTool client — spelling, grammar and style checking.
*
* Talks to a LanguageTool `/v2/check` endpoint: the free public API
* (rate-limited), a self-hosted server, or a Premium account. The
* endpoint / credentials come from the `languagetool` integration in the
* credential store and are resolved in the main process only.
*
* Two layers, mirroring `pagespeed.ts` / `crux.ts`:
* - `checkText` — one page's prose → matches (never throws; failures
* resolve to an `error` result so a single bad page can't abort a run).
* - `runSpellingBatch` — a concurrency pool over many URLs, with progress
* and cooperative cancellation. Text is supplied lazily per item by the
* caller so a 1000-page run never holds every page body in memory.
*/
import type {
SpellingLevel,
SpellingMatch,
SpellingResult,
} from '@freecrawl/shared-types';
import { Agent, ProxyAgent, fetch as undiciFetch, type Dispatcher } from 'undici';
import * as logger from './logger.js';
export const PUBLIC_LT_ENDPOINT = 'https://api.languagetool.org';
const REQUEST_TIMEOUT_MS = 60_000;
/** LanguageTool's public API rejects payloads above 20 KB of text. */
const MAX_TEXT_CHARS = 20_000;
/** Below this, a page is navigation/boilerplate — checking it is noise. */
const MIN_PROSE_CHARS = 40;
/** Bound the stored JSON blob for pathological pages. */
const MAX_STORED_MATCHES = 500;
/** Suggestions kept per match. */
const MAX_REPLACEMENTS = 5;
/** The public API is aggressively rate-limited; self-hosted is not. */
const CONCURRENCY_PUBLIC = 1;
const CONCURRENCY_SELF_HOSTED = 4;
/**
* Languages LanguageTool refuses to accept as a bare code — it demands an
* explicit regional variant. Everything else passes through as-is.
*/
const VARIANT_REQUIRED: Record<string, string> = {
en: 'en-US',
de: 'de-DE',
pt: 'pt-PT',
ca: 'ca-ES',
};
/**
* Map a page's `html[lang]` to a LanguageTool language code. Returns
* `auto` when the page declares nothing, letting LanguageTool detect it.
*/
export function resolveLanguage(pageLang: string | null): string {
const raw = (pageLang ?? '').trim().toLowerCase();
if (!raw) return 'auto';
const parts = raw.split('-');
const primary = parts[0] ?? '';
if (!primary) return 'auto';
if (parts.length >= 2 && parts[1]) {
return `${primary}-${parts[1].toUpperCase()}`;
}
return VARIANT_REQUIRED[primary] ?? primary;
}
function createLtDispatcher(): Dispatcher {
const timeouts = {
headersTimeout: REQUEST_TIMEOUT_MS + 5_000,
bodyTimeout: REQUEST_TIMEOUT_MS + 5_000,
};
const proxy =
process.env['HTTPS_PROXY'] ??
process.env['https_proxy'] ??
process.env['HTTP_PROXY'] ??
process.env['http_proxy'] ??
null;
if (proxy) return new ProxyAgent({ uri: proxy, ...timeouts });
return new Agent({ ...timeouts, connect: { autoSelectFamily: true } });
}
const ltDispatcher = createLtDispatcher();
function errorResult(fetchedAt: string, message: string): SpellingResult {
return {
language: null,
matchCount: 0,
matches: [],
status: 'error',
error: message,
fetchedAt,
};
}
/** Shape of a single LanguageTool `/v2/check` match. */
interface LtMatch {
message?: unknown;
shortMessage?: unknown;
offset?: unknown;
length?: unknown;
replacements?: unknown;
context?: { text?: unknown; offset?: unknown; length?: unknown };
rule?: {
id?: unknown;
issueType?: unknown;
category?: { name?: unknown };
};
}
function mapMatch(m: LtMatch): SpellingMatch | null {
const ctxText = typeof m.context?.text === 'string' ? m.context.text : '';
const ctxOffset =
typeof m.context?.offset === 'number' ? m.context.offset : 0;
const ctxLength =
typeof m.context?.length === 'number' ? m.context.length : 0;
const surface = ctxText.substr(ctxOffset, ctxLength);
if (!surface) return null;
const replacements = Array.isArray(m.replacements)
? (m.replacements as { value?: unknown }[])
.map((r) => (typeof r.value === 'string' ? r.value : null))
.filter((v): v is string => !!v)
.slice(0, MAX_REPLACEMENTS)
: [];
return {
message: typeof m.message === 'string' ? m.message : '',
shortMessage:
typeof m.shortMessage === 'string' && m.shortMessage.length > 0
? m.shortMessage
: null,
offset: typeof m.offset === 'number' ? m.offset : 0,
length: typeof m.length === 'number' ? m.length : surface.length,
text: surface,
context: ctxText,
contextOffset: ctxOffset,
replacements,
ruleId: typeof m.rule?.id === 'string' ? m.rule.id : '',
category:
typeof m.rule?.category?.name === 'string' ? m.rule.category.name : '',
issueType:
typeof m.rule?.issueType === 'string' ? m.rule.issueType : 'misc',
};
}
export interface SpellingCheckOptions {
/** Base URL, e.g. `https://api.languagetool.org` or a self-hosted one. */
endpoint: string;
username?: string | undefined;
apiKey?: string | undefined;
level: SpellingLevel;
/** Lower-cased surface forms to suppress (the custom dictionary). */
ignoreWords: ReadonlySet<string>;
}
/** undici's Response — inferred rather than cast to the DOM `Response`. */
type LtResponse = Awaited<ReturnType<typeof undiciFetch>>;
async function postCheck(
text: string,
language: string,
opts: SpellingCheckOptions,
): Promise<{ res: LtResponse; json: Record<string, unknown> | null }> {
const body = new URLSearchParams({
text,
language,
level: opts.level,
});
// Premium auth is a username + key pair; both must be present.
if (opts.username && opts.apiKey) {
body.set('username', opts.username);
body.set('apiKey', opts.apiKey);
}
const base = opts.endpoint.replace(/\/+$/, '');
const res = await undiciFetch(`${base}/v2/check`, {
method: 'POST',
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
Accept: 'application/json',
},
body: body.toString(),
dispatcher: ltDispatcher,
});
const json = (await res.json().catch(() => null)) as Record<
string,
unknown
> | null;
return { res, json };
}
/**
* Check one page's prose. Never throws. A page with too little prose
* resolves to `skipped`; a transport/API failure to `error`.
*
* When the declared `html[lang]` is a code LanguageTool rejects, the call
* is retried once with `auto` so a bad lang attribute doesn't silently
* cost the user the whole page.
*/
export async function checkText(
text: string,
pageLang: string | null,
opts: SpellingCheckOptions,
): Promise<SpellingResult> {
const fetchedAt = new Date().toISOString();
const prose = text.trim();
if (prose.length < MIN_PROSE_CHARS) {
return {
language: null,
matchCount: 0,
matches: [],
status: 'skipped',
error: null,
fetchedAt,
};
}
const clipped =
prose.length > MAX_TEXT_CHARS ? prose.slice(0, MAX_TEXT_CHARS) : prose;
const attempt = async (
language: string,
): Promise<{ ok: boolean; result: SpellingResult }> => {
const { res, json } = await postCheck(clipped, language, opts);
if (!res.ok) {
const apiErr =
(json?.['message'] as string | undefined) ??
(json?.['error'] as string | undefined);
return {
ok: false,
result: errorResult(
fetchedAt,
apiErr ?? `LanguageTool returned HTTP ${res.status}`,
),
};
}
if (!json) {
return {
ok: false,
result: errorResult(fetchedAt, 'LanguageTool returned an empty response'),
};
}
const langBlock = json['language'] as
| { code?: unknown; detectedLanguage?: { code?: unknown } }
| undefined;
const usedLang =
typeof langBlock?.code === 'string'
? langBlock.code
: typeof langBlock?.detectedLanguage?.code === 'string'
? langBlock.detectedLanguage.code
: null;
const rawMatches = Array.isArray(json['matches'])
? (json['matches'] as LtMatch[])
: [];
const matches = rawMatches
.map(mapMatch)
.filter((m): m is SpellingMatch => m !== null)
// Custom dictionary — drop findings on words the user whitelisted.
.filter((m) => !opts.ignoreWords.has(m.text.toLowerCase()))
.slice(0, MAX_STORED_MATCHES);
return {
ok: true,
result: {
language: usedLang,
matchCount: matches.length,
matches,
status: 'ok',
error: null,
fetchedAt,
},
};
};
try {
const language = resolveLanguage(pageLang);
const first = await attempt(language);
if (first.ok) return first.result;
// A rejected language code is the one failure worth retrying — fall
// back to LanguageTool's own detection rather than losing the page.
if (language !== 'auto' && /lang/i.test(first.result.error ?? '')) {
const retry = await attempt('auto');
if (retry.ok) return retry.result;
}
return first.result;
} catch (err) {
const name = (err as { name?: string } | null)?.name;
const cause = (err as { cause?: unknown } | null)?.cause as
| { code?: string; message?: string }
| undefined;
const isTimeout =
name === 'TimeoutError' ||
name === 'AbortError' ||
cause?.code === 'UND_ERR_HEADERS_TIMEOUT' ||
cause?.code === 'UND_ERR_BODY_TIMEOUT' ||
cause?.code === 'UND_ERR_CONNECT_TIMEOUT';
const message = isTimeout
? `LanguageTool request timed out after ${REQUEST_TIMEOUT_MS / 1000}s`
: err instanceof Error
? cause?.message
? `${err.message} (${cause.message})`
: err.message
: String(err);
return errorResult(fetchedAt, message);
}
}
export interface SpellingBatchOptions {
urls: string[];
check: SpellingCheckOptions;
/**
* Resolve one page's prose + declared language. Returning `null` text
* marks the page `skipped` without hitting the API.
*/
loadPage: (
url: string,
) => Promise<{ text: string | null; lang: string | null }>;
isCancelled: () => boolean;
onResult: (url: string, result: SpellingResult) => void;
onProgress: (done: number, total: number, currentUrl: string | null) => void;
}
export interface SpellingBatchResult {
completed: number;
failed: number;
cancelled: boolean;
}
/**
* Check many pages with a small concurrency pool. The public API allows
* roughly 20 requests/minute, so it runs strictly serially; a self-hosted
* endpoint gets real parallelism.
*/
export async function runSpellingBatch(
opts: SpellingBatchOptions,
): Promise<SpellingBatchResult> {
const { urls, check, loadPage, isCancelled, onResult, onProgress } = opts;
const total = urls.length;
let done = 0;
let completed = 0;
let failed = 0;
let cursor = 0;
const ERROR_LOG_LIMIT = 3;
const errorCounts = new Map<string, number>();
const recordFailure = (url: string, message: string): void => {
const prev = errorCounts.get(message) ?? 0;
errorCounts.set(message, prev + 1);
if (prev < ERROR_LOG_LIMIT) {
logger.log('warn', 'languagetool', `${url} — ${message}`);
}
};
onProgress(0, total, null);
const worker = async (): Promise<void> => {
for (;;) {
if (isCancelled()) return;
const index = cursor++;
if (index >= total) return;
const url = urls[index];
if (!url) return;
onProgress(done, total, url);
const fetchedAt = new Date().toISOString();
let result: SpellingResult;
try {
const page = await loadPage(url);
if (!page.text || page.text.trim().length === 0) {
result = {
language: null,
matchCount: 0,
matches: [],
status: 'skipped',
error: null,
fetchedAt,
};
} else {
result = await checkText(page.text, page.lang, check);
}
} catch (err) {
result = errorResult(
fetchedAt,
err instanceof Error ? err.message : String(err),
);
}
try {
onResult(url, result);
} catch (err) {
logger.log(
'error',
'languagetool',
`failed to persist result for ${url}: ${
err instanceof Error ? err.message : String(err)
}`,
);
}
done++;
if (result.status === 'ok') {
completed++;
} else {
failed++;
if (result.status === 'error') {
recordFailure(url, result.error ?? 'Unknown error');
}
}
onProgress(done, total, null);
}
};
const isPublic = check.endpoint
.replace(/\/+$/, '')
.startsWith(PUBLIC_LT_ENDPOINT);
const poolSize = Math.min(
isPublic ? CONCURRENCY_PUBLIC : CONCURRENCY_SELF_HOSTED,
Math.max(1, total),
);
await Promise.all(Array.from({ length: poolSize }, () => worker()));
const cancelled = isCancelled() && done < total;
logger.log(
'info',
'languagetool',
`batch finished — ${completed} checked, ${failed} skipped/failed${
cancelled ? ', cancelled early' : ''
} (${done}/${total})`,
);
if (errorCounts.size > 0) {
for (const [message, count] of [...errorCounts.entries()].sort(
(a, b) => b[1] - a[1],
)) {
logger.log(
'warn',
'languagetool',
`failure summary — ${count}× "${message}"`,
);
}
if (isPublic) {
logger.log(
'warn',
'languagetool',
'The public LanguageTool API is rate-limited (~20 requests/min). Self-host LanguageTool or add Premium credentials under Settings → Integrations for larger runs.',
);
}
}
return { completed, failed, cancelled };
}
+26
View File
@@ -100,6 +100,17 @@ import {
type PagespeedRunInput,
type PagespeedRunResult,
type PagespeedProgress,
type CruxQueryInput,
type CruxQueryResult,
type CruxRunInput,
type CruxRunResult,
type CruxProgress,
type SpellingQueryInput,
type SpellingQueryResult,
type SpellingRunInput,
type SpellingRunResult,
type SpellingProgress,
type SpellingMatchesResult,
type GoogleAuthState,
type GoogleAuthResult,
type GscListSitesResult,
@@ -360,6 +371,21 @@ const api: FreeCrawlApi = {
pagespeedCancel: (): Promise<void> => ipcRenderer.invoke(IPC.pagespeedCancel),
onPagespeedProgress: (cb) =>
subscribe<PagespeedProgress>(IPC.pagespeedProgress, cb),
cruxQuery: (input: CruxQueryInput): Promise<CruxQueryResult> =>
ipcRenderer.invoke(IPC.cruxQuery, input),
cruxRun: (input: CruxRunInput): Promise<CruxRunResult> =>
ipcRenderer.invoke(IPC.cruxRun, input),
cruxCancel: (): Promise<void> => ipcRenderer.invoke(IPC.cruxCancel),
onCruxProgress: (cb) => subscribe<CruxProgress>(IPC.cruxProgress, cb),
spellingQuery: (input: SpellingQueryInput): Promise<SpellingQueryResult> =>
ipcRenderer.invoke(IPC.spellingQuery, input),
spellingRun: (input: SpellingRunInput): Promise<SpellingRunResult> =>
ipcRenderer.invoke(IPC.spellingRun, input),
spellingCancel: (): Promise<void> => ipcRenderer.invoke(IPC.spellingCancel),
onSpellingProgress: (cb) =>
subscribe<SpellingProgress>(IPC.spellingProgress, cb),
spellingMatches: (url: string): Promise<SpellingMatchesResult | null> =>
ipcRenderer.invoke(IPC.spellingMatches, url),
googleAuthStart: (id: string): Promise<GoogleAuthResult> =>
ipcRenderer.invoke(IPC.googleAuthStart, id),
googleAuthStatus: (id: string): Promise<GoogleAuthState> =>
+6
View File
@@ -19,6 +19,8 @@ import { ImagesTab } from './tabs/ImagesTab.js';
import { BrokenLinksTab } from './tabs/BrokenLinksTab.js';
import { SerpTab } from './tabs/SerpTab.js';
import { PageSpeedTab } from './tabs/PageSpeedTab.js';
import { CruxTab } from './tabs/CruxTab.js';
import { SpellingTab } from './tabs/SpellingTab.js';
import { SearchConsoleTab } from './tabs/SearchConsoleTab.js';
import { AnalyticsTab } from './tabs/AnalyticsTab.js';
import { AiTab } from './tabs/AiTab.js';
@@ -435,6 +437,10 @@ export function App() {
<SerpTab />
) : activeTab === 'pagespeed' ? (
<PageSpeedTab />
) : activeTab === 'crux' ? (
<CruxTab />
) : activeTab === 'spelling' ? (
<SpellingTab />
) : activeTab === 'search-console' ? (
<SearchConsoleTab />
) : activeTab === 'analytics' ? (
@@ -16,6 +16,7 @@ import type {
UrlPageImageRow,
UrlSourceResult,
UrlAnalyticsDetail,
SpellingMatchesResult,
} from '@freecrawl/shared-types';
import { useAppStore } from '../store.js';
import { diagnoseStatus, type StatusDiagnosis } from '../utils/statusDiagnosis.js';
@@ -36,7 +37,8 @@ type SubTab =
| 'view-rendered'
| 'screenshot'
| 'duplicates'
| 'analytics';
| 'analytics'
| 'spelling';
const SUB_TABS: { key: SubTab; label: string; disabled?: boolean }[] = [
{ key: 'url-details', label: 'URL Details' },
@@ -55,6 +57,7 @@ const SUB_TABS: { key: SubTab; label: string; disabled?: boolean }[] = [
{ key: 'screenshot', label: 'Screenshot' },
{ key: 'duplicates', label: 'Duplicates' },
{ key: 'analytics', label: 'Analytics' },
{ key: 'spelling', label: 'Spelling & Grammar' },
];
// Maximum number of URLs we aggregate over in one go. Anything larger is
@@ -78,6 +81,7 @@ const SINGLE_URL_ONLY_TABS: ReadonlySet<SubTab> = new Set([
'screenshot',
'duplicates',
'analytics',
'spelling',
]);
export function BottomDetailPanel() {
@@ -424,11 +428,159 @@ export function BottomDetailPanel() {
{detail && subTab === 'analytics' && (
<AnalyticsView pageUrl={detail.row.url} />
)}
{detail && subTab === 'spelling' && (
<SpellingView pageUrl={detail.row.url} />
)}
</div>
</div>
);
}
/** Colour an issue type chip: misspellings are errors, style is advisory. */
function issueTypeClass(issueType: string): string {
if (issueType === 'misspelling' || issueType === 'typographical') {
return 'bg-red-900/40 text-red-300 border-red-800';
}
if (issueType === 'grammar') {
return 'bg-amber-900/40 text-amber-300 border-amber-800';
}
return 'bg-surface-800 text-surface-300 border-surface-700';
}
/**
* Spelling & Grammar sub-tab — the stored LanguageTool match list for the
* selected page. The offending span is highlighted inside its surrounding
* context so the user can see the error in situ, with suggested
* replacements listed beside it.
*/
function SpellingView({ pageUrl }: { pageUrl: string }) {
const { t } = useTranslation();
const [data, setData] = useState<SpellingMatchesResult | null>(null);
const [loaded, setLoaded] = useState(false);
const dataVersion = useAppStore((s) => s.dataVersion);
useEffect(() => {
let cancelled = false;
setLoaded(false);
void window.freecrawl.spellingMatches(pageUrl).then((res) => {
if (cancelled) return;
setData(res);
setLoaded(true);
});
return () => {
cancelled = true;
};
}, [pageUrl, dataVersion]);
if (!loaded) {
return (
<div className="p-3 text-[11px] text-surface-500">
{t('spelling.loading', { defaultValue: 'Loading…' })}
</div>
);
}
if (!data || data.status === null) {
return (
<div className="p-3 text-[11px] text-surface-500">
{t('spelling.notChecked', {
defaultValue:
'This page has not been checked yet. Select it in the Spelling tab and run a check.',
})}
</div>
);
}
if (data.status === 'error') {
return (
<div className="p-3 text-[11px] text-red-400">
{t('spelling.checkFailed', { defaultValue: 'Check failed' })}: {data.error}
</div>
);
}
if (data.status === 'skipped') {
return (
<div className="p-3 text-[11px] text-surface-500">
{t('spelling.skipped', {
defaultValue: 'Skipped — the page has too little prose to check.',
})}
</div>
);
}
if (data.matches.length === 0) {
return (
<div className="p-3 text-[11px] text-emerald-400">
{t('spelling.clean', {
defaultValue: 'No spelling or grammar issues found.',
})}
</div>
);
}
return (
<div className="flex h-full flex-col">
<div className="flex items-center gap-3 border-b border-surface-800 px-3 py-1.5 text-[10px] text-surface-400">
<span>
{t('spelling.summary', {
defaultValue: '{{n}} finding(s)',
n: data.matches.length,
})}
</span>
{data.language && (
<span>
{t('spelling.language', { defaultValue: 'Language' })}:{' '}
<span className="text-surface-200">{data.language}</span>
</span>
)}
</div>
<ol className="flex-1 overflow-y-auto text-[11px]">
{data.matches.map((m, i) => {
const before = m.context.slice(0, m.contextOffset);
const after = m.context.slice(m.contextOffset + m.text.length);
return (
<li
key={`${m.ruleId}-${m.offset}-${i}`}
className="border-b border-surface-800/60 px-3 py-2 odd:bg-surface-900/20"
>
<div className="mb-1 flex items-center gap-2">
<span
className={`rounded border px-1.5 py-px text-[9px] uppercase tracking-wide ${issueTypeClass(m.issueType)}`}
>
{m.issueType}
</span>
{m.category && (
<span className="text-[10px] text-surface-500">{m.category}</span>
)}
</div>
<div className="mb-1 text-surface-200">
{m.shortMessage || m.message}
</div>
<div className="mb-1 font-mono text-[10px] text-surface-400">
…{before}
<mark className="rounded bg-red-900/60 px-0.5 text-red-200">
{m.text}
</mark>
{after}…
</div>
{m.replacements.length > 0 && (
<div className="text-[10px] text-surface-500">
{t('spelling.suggestions', { defaultValue: 'Suggestions' })}:{' '}
{m.replacements.map((r, ri) => (
<span
key={ri}
className="mr-1 rounded bg-surface-800 px-1 py-px font-mono text-emerald-300"
>
{r}
</span>
))}
</div>
)}
</li>
);
})}
</ol>
</div>
);
}
function AnalyticsView({ pageUrl }: { pageUrl: string }) {
const { t } = useTranslation();
const [data, setData] = useState<UrlAnalyticsDetail | null>(null);
@@ -629,6 +629,12 @@ function buildTree(o: OverviewCounts | null): Node[] {
count: o.issues.contentThin,
category: 'issues:content-thin',
},
{
key: 'issues-spelling-grammar',
label: 'Spelling/Grammar Issues',
count: o.issues.spellingGrammar,
category: 'issues:spelling-grammar',
},
{
key: 'issues-page-empty',
label: 'Empty Page (<30 words)',
@@ -32,6 +32,7 @@ import {
AlertTriangle,
Wrench,
FolderOpen,
SpellCheck,
Lock,
Languages,
Plug,
@@ -212,6 +213,7 @@ type SectionKey =
| 'rendering'
| 'performance-budget'
| 'storage'
| 'spelling'
| 'privacy'
| 'language';
@@ -377,6 +379,12 @@ const SECTIONS: SectionDef[] = [
icon: FolderOpen,
keywords: 'storage save folder directory project location path documents disk',
},
{
key: 'spelling',
label: 'Spelling',
icon: SpellCheck,
keywords: 'spelling grammar languagetool dictionary ignore words picky rule level yazım dilbilgisi sözlük',
},
{
key: 'privacy',
label: 'Privacy',
@@ -945,6 +953,7 @@ export function SettingsDialog({ open, onClose }: Props) {
<PerformanceBudgetPanel form={form} update={update} />
)}
{active === 'storage' && <StoragePanel />}
{active === 'spelling' && <SpellingPanel />}
{active === 'privacy' && <PrivacyPanel />}
{active === 'language' && <LanguagePanel />}
</div>
@@ -3956,6 +3965,106 @@ function IntegrationCard({
* shows the resolved path so the user always sees where new projects
* will land, regardless of whether they've customised it.
*/
/**
* Spelling & Grammar panel — rule strictness + the custom dictionary.
* Both are app preferences read by the main process when a LanguageTool
* run starts; the endpoint and any Premium credentials live under
* Settings → Integrations → LanguageTool.
*/
function SpellingPanel() {
const { t } = useTranslation();
const [level, setLevel] = useState<'default' | 'picky'>(() =>
window.freecrawl.prefsGet('spellingLevel') === 'picky' ? 'picky' : 'default',
);
const [ignoreWords, setIgnoreWords] = useState<string>(() => {
const raw = window.freecrawl.prefsGet('spellingIgnoreWords');
return typeof raw === 'string' ? raw : '';
});
function pickLevel(l: 'default' | 'picky') {
setLevel(l);
// `default` is the default — clear the pref rather than storing it.
if (l === 'picky') window.freecrawl.prefsSet('spellingLevel', 'picky');
else window.freecrawl.prefsDelete('spellingLevel');
}
function saveIgnoreWords(value: string) {
setIgnoreWords(value);
if (value.trim().length > 0) {
window.freecrawl.prefsSet('spellingIgnoreWords', value);
} else {
window.freecrawl.prefsDelete('spellingIgnoreWords');
}
}
const wordCount = ignoreWords
.split(/[\s,;\n]+/)
.map((w) => w.trim())
.filter(Boolean).length;
return (
<>
<p className="mb-3 text-[11px] text-surface-400">
{t('settings.spelling.intro', { defaultValue: "Spelling, grammar and style checks run through LanguageTool against pages you select in the Spelling tab. The page's own html[lang] picks the language; pages that don't declare one are auto-detected. Configure the endpoint (public API or self-hosted) under Integrations → LanguageTool." })}
</p>
<div className="mb-4 rounded border border-surface-800 bg-surface-950/40 p-3">
<div className="mb-2 text-[10px] font-semibold uppercase tracking-wider text-surface-400">
{t('settings.spelling.levelLabel', { defaultValue: "Rule Level" })}
</div>
<div className="flex flex-col gap-2">
<label className="flex items-start gap-2 cursor-pointer">
<input
type="radio"
name="spelling-level"
className="mt-0.5"
checked={level === 'default'}
onChange={() => pickLevel('default')}
/>
<span className="text-[12px] text-surface-100">
{t('settings.spelling.levelDefault', { defaultValue: "Default — spelling and grammar errors only" })}
</span>
</label>
<label className="flex items-start gap-2 cursor-pointer">
<input
type="radio"
name="spelling-level"
className="mt-0.5"
checked={level === 'picky'}
onChange={() => pickLevel('picky')}
/>
<span className="text-[12px] text-surface-100">
{t('settings.spelling.levelPicky', { defaultValue: "Picky — also flags style, typography and wordiness" })}
</span>
</label>
</div>
</div>
<div className="rounded border border-surface-800 bg-surface-950/40 p-3">
<div className="mb-2 text-[10px] font-semibold uppercase tracking-wider text-surface-400">
{t('settings.spelling.dictionaryLabel', { defaultValue: "Custom Dictionary" })}
</div>
<p className="mb-2 text-[11px] text-surface-400">
{t('settings.spelling.dictionaryHint', { defaultValue: "Words to never flag — brand names, product names, industry jargon. One per line, or separated by commas. Case-insensitive." })}
</p>
<textarea
className="h-32 w-full resize-y rounded border border-surface-700 bg-surface-950 px-2 py-1 font-mono text-[11px] text-surface-100 focus:border-blue-500 focus:outline-none"
placeholder={"FreeCrawl\nhreflang\ncanonicalisation"}
value={ignoreWords}
onChange={(e) => saveIgnoreWords(e.target.value)}
spellCheck={false}
/>
<p className="mt-1 text-[10px] text-surface-500">
{t('settings.spelling.dictionaryCount', {
defaultValue: "{{n}} word(s) ignored",
n: wordCount,
})}
</p>
</div>
</>
);
}
function StoragePanel() {
const { t } = useTranslation();
const [override, setOverride] = useState<string>(() => {
@@ -351,6 +351,9 @@ const TR_LABELS: Record<string, string> = {
Inlinks: 'Gelen Linkler',
Outlinks: 'Giden Linkler',
'Link Score': 'Bağlantı Puanı',
'Spelling & Grammar': 'Yazım ve Dilbilgisi',
'Spelling/Grammar Issues': 'Yazım/Dilbilgisi Sorunları',
Spelling: 'Yazım',
'Title 1': 'Başlık 1',
'Title 1 Length': 'Başlık 1 Uzunluğu',
'Meta Description 1': 'Meta Açıklama 1',
@@ -1021,6 +1021,81 @@
"emptyNoData": "No internal HTML pages crawled yet — run a crawl, then select pages to audit.",
"emptyNoMatch": "No pages match the current filter."
},
"cruxTab": {
"title": "Chrome UX Report",
"filterPlaceholder": "Filter by URL…",
"filterAll": "All pages",
"filterTested": "Fetched only",
"filterUntested": "Not fetched only",
"formFactor": "Device:",
"devicePhone": "Phone",
"deviceDesktop": "Desktop",
"deviceBoth": "Phone + Desktop",
"selectUntested": "Select not fetched",
"run": "Fetch CrUX ({{count}})",
"runProgress": "Fetching {{done}} / {{total}}",
"cancel": "Cancel",
"confirmRun": "This will request CrUX data for {{count}} records. Continue?",
"noKeyGate": "The Chrome UX Report API needs a free Google API key (no keyless mode). Enable the \"Chrome UX Report API\" on your key.",
"addKeyCta": "Add API Key…",
"runDisabledNoKey": "Add a Chrome UX Report API key in Settings → Integrations to enable fetching.",
"colUrl": "URL",
"colLcp": "LCP",
"colInp": "INP",
"colCls": "CLS",
"colFcp": "FCP",
"colTtfb": "TTFB",
"colFetched": "Fetched",
"groupPhone": "Phone",
"groupDesktop": "Desktop",
"noData": "no data",
"noDataTip": "CrUX has too little real-user traffic for this page.",
"summary": "{{shown}} pages · {{tested}} with data · {{selected}} selected",
"firstN": "(first {{n}})",
"fetching": "Fetching: {{url}}",
"emptyNoData": "No internal HTML pages crawled yet — run a crawl, then select pages to fetch field data for.",
"emptyNoMatch": "No pages match the current filter."
},
"spellingTab": {
"title": "Spelling & Grammar",
"filterPlaceholder": "Filter by URL…",
"filterAll": "All pages",
"filterChecked": "Checked only",
"filterUnchecked": "Unchecked only",
"filterErrors": "With findings",
"selectUnchecked": "Select unchecked",
"options": "Options…",
"optionsTitle": "Rule level and custom dictionary live in Settings → Spelling.",
"run": "Check Spelling ({{count}})",
"runProgress": "Checking {{done}} / {{total}}",
"cancel": "Cancel",
"confirmRun": "This will check {{count}} pages with LanguageTool. The free public API is rate-limited (~20 requests/min), so this may take a while. Continue?",
"colUrl": "URL",
"colLang": "Lang",
"colWords": "Words",
"colIssues": "Findings",
"colStatus": "Status",
"colChecked": "Checked",
"statusOk": "ok",
"statusError": "ERR",
"statusSkipped": "skipped",
"statusSkippedTip": "Too little prose on the page to check.",
"summary": "{{shown}} pages · {{checked}} checked · {{errors}} with findings · {{selected}} selected",
"firstN": "(first {{n}})",
"checking": "Checking: {{url}}",
"emptyNoData": "No internal HTML pages crawled yet — run a crawl, then select pages to check.",
"emptyNoMatch": "No pages match the current filter."
},
"spelling": {
"loading": "Loading…",
"notChecked": "This page has not been checked yet. Select it in the Spelling tab and run a check.",
"checkFailed": "Check failed",
"skipped": "Skipped — the page has too little prose to check.",
"clean": "No spelling or grammar issues found.",
"summary": "{{n}} finding(s)",
"language": "Language",
"suggestions": "Suggestions"
},
"gscTab": {
"title": "Search Console",
"loading": "Loading…",
@@ -1237,6 +1312,16 @@
"tr": "Türkçe",
"note": "Some deep panels and dialogs may still display English while broader coverage is being added."
},
"spelling": {
"title": "Spelling",
"intro": "Spelling, grammar and style checks run through LanguageTool against pages you select in the Spelling tab. The page's own html[lang] picks the language; pages that don't declare one are auto-detected. Configure the endpoint (public API or self-hosted) under Integrations → LanguageTool.",
"levelLabel": "Rule Level",
"levelDefault": "Default — spelling and grammar errors only",
"levelPicky": "Picky — also flags style, typography and wordiness",
"dictionaryLabel": "Custom Dictionary",
"dictionaryHint": "Words to never flag — brand names, product names, industry jargon. One per line, or separated by commas. Case-insensitive.",
"dictionaryCount": "{{n}} word(s) ignored"
},
"storage": {
"title": "Storage",
"intro": "Default folder for new .seoproject files. Save As dialogs open here by default; the Open dialog is not constrained.",
@@ -1021,6 +1021,81 @@
"emptyNoData": "Henüz taranmış iç HTML sayfası yok — önce bir crawl çalıştır, sonra denetlenecek sayfaları seç.",
"emptyNoMatch": "Mevcut filtreyle eşleşen sayfa yok."
},
"cruxTab": {
"title": "Chrome UX Report",
"filterPlaceholder": "URL'ye göre filtrele…",
"filterAll": "Tüm sayfalar",
"filterTested": "Yalnızca çekilenler",
"filterUntested": "Yalnızca çekilmeyenler",
"formFactor": "Cihaz:",
"devicePhone": "Telefon",
"deviceDesktop": "Masaüstü",
"deviceBoth": "Telefon + Masaüstü",
"selectUntested": "Çekilmeyenleri seç",
"run": "CrUX Çek ({{count}})",
"runProgress": "Çekiliyor {{done}} / {{total}}",
"cancel": "İptal",
"confirmRun": "Bu işlem {{count}} kayıt için CrUX verisi isteyecek. Devam edilsin mi?",
"noKeyGate": "Chrome UX Report API için ücretsiz bir Google API anahtarı gerekir (anahtarsız mod yok). Anahtarınızda \"Chrome UX Report API\"yi etkinleştirin.",
"addKeyCta": "API Anahtarı Ekle…",
"runDisabledNoKey": "Çekmeyi etkinleştirmek için Ayarlar → Entegrasyonlar'dan bir Chrome UX Report API anahtarı ekleyin.",
"colUrl": "URL",
"colLcp": "LCP",
"colInp": "INP",
"colCls": "CLS",
"colFcp": "FCP",
"colTtfb": "TTFB",
"colFetched": "Çekildi",
"groupPhone": "Telefon",
"groupDesktop": "Masaüstü",
"noData": "veri yok",
"noDataTip": "CrUX'ta bu sayfa için yeterli gerçek-kullanıcı trafiği yok.",
"summary": "{{shown}} sayfa · {{tested}} veri var · {{selected}} seçildi",
"firstN": "(ilk {{n}})",
"fetching": "Çekiliyor: {{url}}",
"emptyNoData": "Henüz taranmış iç HTML sayfası yok — önce bir crawl çalıştır, sonra alan verisi çekilecek sayfaları seç.",
"emptyNoMatch": "Mevcut filtreyle eşleşen sayfa yok."
},
"spellingTab": {
"title": "Yazım ve Dilbilgisi",
"filterPlaceholder": "URL'ye göre filtrele…",
"filterAll": "Tüm sayfalar",
"filterChecked": "Yalnızca denetlenenler",
"filterUnchecked": "Yalnızca denetlenmeyenler",
"filterErrors": "Bulgusu olanlar",
"selectUnchecked": "Denetlenmeyenleri seç",
"options": "Seçenekler…",
"optionsTitle": "Kural seviyesi ve özel sözlük Ayarlar → Yazım altında.",
"run": "Yazım Denetle ({{count}})",
"runProgress": "Denetleniyor {{done}} / {{total}}",
"cancel": "İptal",
"confirmRun": "Bu işlem {{count}} sayfayı LanguageTool ile denetleyecek. Ücretsiz genel API hız sınırlıdır (~20 istek/dk), bu yüzden biraz sürebilir. Devam edilsin mi?",
"colUrl": "URL",
"colLang": "Dil",
"colWords": "Kelime",
"colIssues": "Bulgu",
"colStatus": "Durum",
"colChecked": "Denetlendi",
"statusOk": "tamam",
"statusError": "HATA",
"statusSkipped": "atlandı",
"statusSkippedTip": "Sayfada denetlenecek kadar düzyazı yok.",
"summary": "{{shown}} sayfa · {{checked}} denetlendi · {{errors}} bulgulu · {{selected}} seçildi",
"firstN": "(ilk {{n}})",
"checking": "Denetleniyor: {{url}}",
"emptyNoData": "Henüz taranmış iç HTML sayfası yok — önce bir crawl çalıştır, sonra denetlenecek sayfaları seç.",
"emptyNoMatch": "Mevcut filtreyle eşleşen sayfa yok."
},
"spelling": {
"loading": "Yükleniyor…",
"notChecked": "Bu sayfa henüz denetlenmedi. Yazım sekmesinden seçip denetim çalıştırın.",
"checkFailed": "Denetim başarısız",
"skipped": "Atlandı — sayfada denetlenecek kadar düzyazı yok.",
"clean": "Yazım veya dilbilgisi sorunu bulunamadı.",
"summary": "{{n}} bulgu",
"language": "Dil",
"suggestions": "Öneriler"
},
"gscTab": {
"title": "Search Console",
"loading": "Yükleniyor…",
@@ -1237,6 +1312,16 @@
"tr": "Türkçe",
"note": "Bazı derin paneller ve diyaloglar, çeviri kapsamı genişletilirken hala İngilizce görünebilir."
},
"spelling": {
"title": "Yazım",
"intro": "Yazım, dilbilgisi ve stil denetimleri, Yazım sekmesinde seçtiğiniz sayfalar üzerinde LanguageTool ile çalışır. Dil, sayfanın kendi html[lang] değerinden seçilir; bildirmeyen sayfalar otomatik algılanır. Uç noktayı (genel API veya kendi sunucunuz) Entegrasyonlar → LanguageTool altından ayarlayın.",
"levelLabel": "Kural Seviyesi",
"levelDefault": "Varsayılan — yalnızca yazım ve dilbilgisi hataları",
"levelPicky": "Titiz — stil, tipografi ve gereksiz uzunluğu da işaretler",
"dictionaryLabel": "Özel Sözlük",
"dictionaryHint": "Asla işaretlenmeyecek kelimeler — marka adları, ürün adları, sektör jargonu. Her satıra bir tane veya virgülle ayrılmış. Büyük/küçük harf duyarsız.",
"dictionaryCount": "{{n}} kelime yok sayılıyor"
},
"storage": {
"title": "Depolama",
"intro": "Yeni .seoproject dosyaları için varsayılan klasör. Farklı Kaydet diyalogları varsayılan olarak burayı açar; Aç diyaloğu kısıtlı değildir.",
+6
View File
@@ -35,6 +35,8 @@ export type TabKey =
| 'broken-links'
| 'serp'
| 'pagespeed'
| 'crux'
| 'spelling'
| 'search-console'
| 'analytics'
| 'ai'
@@ -67,6 +69,8 @@ export const TAB_ORDER: { key: TabKey; label: string }[] = [
{ key: 'broken-links', label: 'Broken Links' },
{ key: 'serp', label: 'SERP' },
{ key: 'pagespeed', label: 'PageSpeed' },
{ key: 'crux', label: 'CrUX' },
{ key: 'spelling', label: 'Spelling' },
{ key: 'search-console', label: 'Search Console' },
{ key: 'analytics', label: 'GA4' },
{ key: 'ai', label: 'AI' },
@@ -218,6 +222,7 @@ export const TAB_QUICK_FILTERS: Partial<Record<TabKey, TabQuickFilter[]>> = {
content: [
{ label: 'All', category: 'internal:html' },
{ label: 'Thin Content (<300 words)', category: 'issues:content-thin' },
{ label: 'Spelling/Grammar Issues', category: 'issues:spelling-grammar' },
{ label: 'Empty Page (<30 words)', category: 'issues:page-empty' },
{ label: 'Hard to Read (Flesch <30)', category: 'issues:flesch-very-difficult' },
{ label: 'Gunning Fog >17', category: 'issues:gunning-fog-very-high' },
@@ -522,6 +527,7 @@ function tabForCategory(cat: UrlCategory): TabKey | null {
if (cat === 'issues:multiple-canonicals') return 'canonicals';
if (
cat === 'issues:content-thin' ||
cat === 'issues:spelling-grammar' ||
cat === 'issues:page-empty' ||
cat === 'issues:high-boilerplate' ||
cat === 'issues:flesch-very-difficult' ||
@@ -0,0 +1,566 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { useVirtualizer } from '@tanstack/react-virtual';
import { useTranslation } from 'react-i18next';
import { Smartphone, Monitor, Loader2 } from 'lucide-react';
import type {
CruxMetrics,
CruxProgress,
CruxRow,
CruxRunFormFactor,
} from '@freecrawl/shared-types';
import { useAppStore } from '../store.js';
const TOOLBAR_HEIGHT = 36;
const HEADER_HEIGHT = 46;
const STATUS_BAR_HEIGHT = 22;
const ROW_HEIGHT = 30;
const PAGE_SIZE = 2000;
const POLL_MS_RUNNING = 5000;
/** Above this many records, warn before kicking off the run. */
const CONFIRM_THRESHOLD = 50;
/** Pixel widths of the five CrUX metric cells (shared by header + rows). */
const W_LCP = 58;
const W_INP = 52;
const W_CLS = 50;
const W_FCP = 58;
const W_TTFB = 58;
const GROUP_WIDTH = W_LCP + W_INP + W_CLS + W_FCP + W_TTFB;
type FilterMode = 'all' | 'tested' | 'untested';
/** Core Web Vitals thresholds → cell colour. */
function cwvClass(v: number | null, good: number, poor: number): string {
if (v === null) return 'text-surface-600';
if (v <= good) return 'text-emerald-400';
if (v <= poor) return 'text-amber-400';
return 'text-red-400';
}
/** Format a millisecond timing compactly (e.g. `1.8s`, `420ms`). */
function fmtMs(v: number | null): string {
if (v === null) return '—';
if (v >= 1000) return `${(v / 1000).toFixed(1)}s`;
return `${Math.round(v)}ms`;
}
function fmtCls(v: number | null): string {
return v === null ? '—' : v.toFixed(3);
}
/** Short relative time for the "Fetched" column. */
function relativeTime(iso: string): string {
if (!iso) return '—';
const then = new Date(iso).getTime();
if (Number.isNaN(then)) return '—';
const diff = Date.now() - then;
const min = Math.floor(diff / 60_000);
if (min < 1) return 'just now';
if (min < 60) return `${min}m ago`;
const hr = Math.floor(min / 60);
if (hr < 24) return `${hr}h ago`;
const day = Math.floor(hr / 24);
if (day < 30) return `${day}d ago`;
return new Date(iso).toLocaleDateString();
}
export function CruxTab() {
const { t } = useTranslation();
const dataVersion = useAppStore((s) => s.dataVersion);
const crawlProgress = useAppStore((s) => s.progress);
const setSettingsOpen = useAppStore((s) => s.setSettingsOpen);
const [rows, setRows] = useState<CruxRow[]>([]);
const [total, setTotal] = useState(0);
const [search, setSearch] = useState('');
const [filter, setFilter] = useState<FilterMode>('all');
const [formFactor, setFormFactor] = useState<CruxRunFormFactor>('phone');
const [selected, setSelected] = useState<Set<string>>(new Set());
const [running, setRunning] = useState(false);
const [progress, setProgress] = useState<CruxProgress | null>(null);
const [hasApiKey, setHasApiKey] = useState(true);
const scrollRef = useRef<HTMLDivElement>(null);
const reload = useCallback(async () => {
const res = await window.freecrawl.cruxQuery({
limit: PAGE_SIZE,
offset: 0,
search: search || undefined,
filter,
});
setRows(res.rows);
setTotal(res.total);
}, [search, filter]);
useEffect(() => {
let cancelled = false;
void reload();
if (!crawlProgress?.running) return;
const id = setInterval(() => {
if (!cancelled) void reload();
}, POLL_MS_RUNNING);
return () => {
cancelled = true;
clearInterval(id);
};
}, [reload, dataVersion, crawlProgress?.running]);
// CrUX has no keyless mode — surface the gate banner when unset.
useEffect(() => {
void window.freecrawl.integrationsGetAll().then((state) => {
setHasApiKey(state['crux']?.fields?.['apiKey']?.set ?? false);
});
}, []);
useEffect(() => {
const off = window.freecrawl.onCruxProgress((p) => {
setProgress(p.running ? p : null);
if (!p.currentUrl) void reload();
});
return off;
}, [reload]);
const virtualizer = useVirtualizer({
count: rows.length,
getScrollElement: () => scrollRef.current,
estimateSize: () => ROW_HEIGHT,
overscan: 16,
getItemKey: (index) => rows[index]?.url ?? `idx-${index}`,
});
const allLoadedSelected =
rows.length > 0 && rows.every((r) => selected.has(r.url));
const toggleRow = useCallback((url: string) => {
setSelected((prev) => {
const next = new Set(prev);
if (next.has(url)) next.delete(url);
else next.add(url);
return next;
});
}, []);
const toggleAll = useCallback(() => {
setSelected((prev) => {
if (rows.length > 0 && rows.every((r) => prev.has(r.url))) {
return new Set();
}
return new Set(rows.map((r) => r.url));
});
}, [rows]);
const selectUntested = useCallback(() => {
setSelected(
new Set(rows.filter((r) => !r.phone && !r.desktop).map((r) => r.url)),
);
}, [rows]);
const ffCount = formFactor === 'both' ? 2 : 1;
const recordCount = selected.size * ffCount;
const runFetch = useCallback(async () => {
const urls = [...selected];
if (urls.length === 0 || running || !hasApiKey) return;
if (
recordCount > CONFIRM_THRESHOLD &&
!window.confirm(
t('cruxTab.confirmRun', {
defaultValue:
'This will request CrUX data for {{count}} records. Continue?',
count: recordCount,
}),
)
) {
return;
}
setRunning(true);
setProgress({ done: 0, total: recordCount, currentUrl: null, running: true });
try {
await window.freecrawl.cruxRun({ urls, formFactor });
} finally {
setRunning(false);
setProgress(null);
void reload();
}
}, [selected, running, recordCount, formFactor, hasApiKey, t, reload]);
const cancelRun = useCallback(() => {
void window.freecrawl.cruxCancel();
}, []);
const testedCount = useMemo(
() => rows.filter((r) => r.phone || r.desktop).length,
[rows],
);
return (
<div className="flex h-full w-full flex-col bg-surface-950">
{/* Toolbar */}
<div
className="flex items-center gap-2 border-b border-surface-800 bg-surface-900/40 px-3"
style={{ height: TOOLBAR_HEIGHT }}
>
<div className="text-[12px] font-semibold tracking-wide text-surface-100">
{t('cruxTab.title', { defaultValue: 'Chrome UX Report' })}
</div>
<input
type="text"
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder={t('cruxTab.filterPlaceholder', {
defaultValue: 'Filter by URL…',
})}
className="h-6 w-56 rounded border border-surface-700 bg-surface-950 px-2 text-[11px] text-surface-100 placeholder-surface-500 focus:border-blue-500 focus:outline-none"
/>
<select
className="h-6 rounded border border-surface-700 bg-surface-950 px-1.5 text-[11px] text-surface-100 focus:border-blue-500 focus:outline-none"
value={filter}
onChange={(e) => setFilter(e.target.value as FilterMode)}
>
<option value="all">
{t('cruxTab.filterAll', { defaultValue: 'All pages' })}
</option>
<option value="tested">
{t('cruxTab.filterTested', { defaultValue: 'Fetched only' })}
</option>
<option value="untested">
{t('cruxTab.filterUntested', { defaultValue: 'Not fetched only' })}
</option>
</select>
<label className="flex items-center gap-1 text-[11px] text-surface-400">
{t('cruxTab.formFactor', { defaultValue: 'Device:' })}
<select
className="h-6 rounded border border-surface-700 bg-surface-950 px-1.5 text-[11px] text-surface-100 focus:border-blue-500 focus:outline-none"
value={formFactor}
onChange={(e) => setFormFactor(e.target.value as CruxRunFormFactor)}
disabled={running}
>
<option value="phone">
{t('cruxTab.devicePhone', { defaultValue: 'Phone' })}
</option>
<option value="desktop">
{t('cruxTab.deviceDesktop', { defaultValue: 'Desktop' })}
</option>
<option value="both">
{t('cruxTab.deviceBoth', { defaultValue: 'Phone + Desktop' })}
</option>
</select>
</label>
<button
type="button"
onClick={selectUntested}
disabled={running || rows.length === 0}
className="h-6 rounded border border-surface-700 bg-surface-800 px-2 text-[11px] text-surface-200 hover:bg-surface-700 disabled:cursor-not-allowed disabled:opacity-40"
>
{t('cruxTab.selectUntested', { defaultValue: 'Select not fetched' })}
</button>
<div className="ml-auto flex items-center gap-2">
{running && progress ? (
<>
<span className="flex items-center gap-1.5 text-[11px] text-surface-300">
<Loader2 size={12} className="animate-spin text-blue-400" />
{t('cruxTab.runProgress', {
defaultValue: 'Fetching {{done}} / {{total}}',
done: progress.done,
total: progress.total,
})}
</span>
<button
type="button"
onClick={cancelRun}
className="h-6 rounded border border-red-700 bg-red-900/40 px-2.5 text-[11px] font-medium text-red-200 hover:bg-red-900/70"
>
{t('cruxTab.cancel', { defaultValue: 'Cancel' })}
</button>
</>
) : (
<button
type="button"
onClick={() => void runFetch()}
disabled={selected.size === 0 || running || !hasApiKey}
title={
!hasApiKey
? t('cruxTab.runDisabledNoKey', {
defaultValue:
'Add a Chrome UX Report API key in Settings → Integrations to enable fetching.',
})
: undefined
}
className="h-6 rounded bg-blue-600 px-3 text-[11px] font-medium text-white hover:bg-blue-500 disabled:cursor-not-allowed disabled:bg-surface-700 disabled:text-surface-500"
>
{t('cruxTab.run', {
defaultValue: 'Fetch CrUX ({{count}})',
count: recordCount,
})}
</button>
)}
</div>
</div>
{/* No-key gate — CrUX has no keyless mode at all. */}
{!hasApiKey && (
<div className="flex items-center gap-2 border-b border-amber-700/50 bg-amber-900/25 px-3 py-1.5 text-[11px] text-amber-100">
<span className="font-semibold">⚠</span>
<span className="flex-1">
{t('cruxTab.noKeyGate', {
defaultValue:
'The Chrome UX Report API needs a free Google API key (no keyless mode). Enable the "Chrome UX Report API" on your key.',
})}
</span>
<button
type="button"
onClick={() => setSettingsOpen(true)}
className="rounded border border-amber-500/60 bg-amber-700/40 px-2 py-0.5 text-[10px] font-medium text-amber-50 hover:bg-amber-700/70"
>
{t('cruxTab.addKeyCta', { defaultValue: 'Add API Key…' })}
</button>
</div>
)}
{/* Column header (two-tier: device groups + metric labels) */}
<div
className="flex shrink-0 select-none items-stretch border-b border-surface-800 bg-surface-900/60 text-[10px] font-medium text-surface-400"
style={{ height: HEADER_HEIGHT }}
>
<div className="flex w-[30px] items-center justify-center border-r border-surface-800">
<input
type="checkbox"
checked={allLoadedSelected}
ref={(el) => {
if (el) {
el.indeterminate = selected.size > 0 && !allLoadedSelected;
}
}}
onChange={toggleAll}
disabled={rows.length === 0}
className="h-3 w-3 accent-blue-500"
/>
</div>
<div className="flex flex-1 flex-col justify-end border-r border-surface-800 px-2 pb-1">
{t('cruxTab.colUrl', { defaultValue: 'URL' })}
</div>
{(['phone', 'desktop'] as const).map((ff) => (
<div
key={ff}
className="flex flex-col border-r border-surface-800"
style={{ width: GROUP_WIDTH }}
>
<div className="flex h-[22px] items-center justify-center gap-1 border-b border-surface-800 bg-surface-900/80 text-surface-300">
{ff === 'phone' ? <Smartphone size={11} /> : <Monitor size={11} />}
{ff === 'phone'
? t('cruxTab.groupPhone', { defaultValue: 'Phone' })
: t('cruxTab.groupDesktop', { defaultValue: 'Desktop' })}
</div>
<div className="flex flex-1 items-end pb-1 text-center">
<div style={{ width: W_LCP }}>{t('cruxTab.colLcp', { defaultValue: 'LCP' })}</div>
<div style={{ width: W_INP }}>{t('cruxTab.colInp', { defaultValue: 'INP' })}</div>
<div style={{ width: W_CLS }}>{t('cruxTab.colCls', { defaultValue: 'CLS' })}</div>
<div style={{ width: W_FCP }}>{t('cruxTab.colFcp', { defaultValue: 'FCP' })}</div>
<div style={{ width: W_TTFB }}>{t('cruxTab.colTtfb', { defaultValue: 'TTFB' })}</div>
</div>
</div>
))}
<div className="flex w-[96px] flex-col justify-end px-2 pb-1 text-right">
{t('cruxTab.colFetched', { defaultValue: 'Fetched' })}
</div>
</div>
{/* Body */}
<div ref={scrollRef} className="flex-1 overflow-y-auto">
{rows.length === 0 ? (
<div className="flex h-full items-center justify-center px-6 text-center text-[12px] text-surface-500">
{total === 0
? t('cruxTab.emptyNoData', {
defaultValue:
'No internal HTML pages crawled yet — run a crawl, then select pages to fetch field data for.',
})
: t('cruxTab.emptyNoMatch', {
defaultValue: 'No pages match the current filter.',
})}
</div>
) : (
<div
style={{
height: virtualizer.getTotalSize(),
width: '100%',
position: 'relative',
}}
>
{virtualizer.getVirtualItems().map((vRow) => {
const row = rows[vRow.index];
if (!row) return null;
return (
<div
key={vRow.key}
style={{
position: 'absolute',
top: 0,
left: 0,
width: '100%',
height: ROW_HEIGHT,
transform: `translateY(${vRow.start}px)`,
}}
>
<CruxTableRow
row={row}
selected={selected.has(row.url)}
onToggle={() => toggleRow(row.url)}
/>
</div>
);
})}
</div>
)}
</div>
{/* Status bar */}
<div
className="flex items-center gap-3 border-t border-surface-800 bg-surface-900/40 px-3 text-[11px] text-surface-400"
style={{ height: STATUS_BAR_HEIGHT }}
>
<span>
{t('cruxTab.summary', {
defaultValue: '{{shown}} pages · {{tested}} with data · {{selected}} selected',
shown: rows.length.toLocaleString(),
tested: testedCount.toLocaleString(),
selected: selected.size.toLocaleString(),
})}
{total > rows.length &&
' ' +
t('cruxTab.firstN', {
defaultValue: '(first {{n}})',
n: PAGE_SIZE.toLocaleString(),
})}
</span>
{running && progress?.currentUrl && (
<span className="truncate text-surface-500">
{t('cruxTab.fetching', {
defaultValue: 'Fetching: {{url}}',
url: progress.currentUrl,
})}
</span>
)}
</div>
</div>
);
}
function MetricCell({
text,
cls,
width,
}: {
text: string;
cls: string;
width: number;
}) {
return (
<div
className={`shrink-0 text-center tabular-nums ${cls}`}
style={{ width }}
>
{text}
</div>
);
}
function FormFactorCells({ m }: { m: CruxMetrics | null }) {
const { t } = useTranslation();
if (!m) {
return (
<>
<MetricCell text="—" cls="text-surface-700" width={W_LCP} />
<MetricCell text="—" cls="text-surface-700" width={W_INP} />
<MetricCell text="—" cls="text-surface-700" width={W_CLS} />
<MetricCell text="—" cls="text-surface-700" width={W_FCP} />
<MetricCell text="—" cls="text-surface-700" width={W_TTFB} />
</>
);
}
if (m.status === 'nodata') {
return (
<div
className="flex shrink-0 items-center justify-center text-surface-500"
style={{ width: GROUP_WIDTH }}
title={t('cruxTab.noDataTip', {
defaultValue: 'CrUX has too little real-user traffic for this page.',
})}
>
{t('cruxTab.noData', { defaultValue: 'no data' })}
</div>
);
}
if (m.status === 'error') {
return (
<div
className="flex shrink-0 cursor-help items-center justify-center gap-1 text-red-400"
style={{ width: GROUP_WIDTH }}
title={m.error ?? 'Fetch failed'}
>
<span className="font-semibold">ERR</span>
<span className="text-[9px] text-red-300/70">ⓘ</span>
</div>
);
}
return (
<>
<MetricCell text={fmtMs(m.lcp)} cls={cwvClass(m.lcp, 2500, 4000)} width={W_LCP} />
<MetricCell text={fmtMs(m.inp)} cls={cwvClass(m.inp, 200, 500)} width={W_INP} />
<MetricCell text={fmtCls(m.cls)} cls={cwvClass(m.cls, 0.1, 0.25)} width={W_CLS} />
<MetricCell text={fmtMs(m.fcp)} cls={cwvClass(m.fcp, 1800, 3000)} width={W_FCP} />
<MetricCell text={fmtMs(m.ttfb)} cls={cwvClass(m.ttfb, 800, 1800)} width={W_TTFB} />
</>
);
}
function CruxTableRow({
row,
selected,
onToggle,
}: {
row: CruxRow;
selected: boolean;
onToggle: () => void;
}) {
const fetched = useMemo(() => {
const candidates = [row.phone?.fetchedAt, row.desktop?.fetchedAt].filter(
(v): v is string => !!v,
);
if (candidates.length === 0) return '';
return candidates.sort().reverse()[0] ?? '';
}, [row.phone, row.desktop]);
return (
<div
onClick={onToggle}
className={`flex h-full cursor-pointer items-center text-[11px] ${
selected
? 'bg-blue-900/30'
: 'odd:bg-surface-900/20 hover:bg-surface-800/40'
}`}
>
<div className="flex w-[30px] items-center justify-center">
<input
type="checkbox"
checked={selected}
onChange={onToggle}
onClick={(e) => e.stopPropagation()}
className="h-3 w-3 accent-blue-500"
/>
</div>
<div className="flex-1 truncate px-2 text-surface-200" title={row.url}>
{row.url}
</div>
<div className="flex shrink-0 items-center" style={{ width: GROUP_WIDTH }}>
<FormFactorCells m={row.phone} />
</div>
<div className="flex shrink-0 items-center" style={{ width: GROUP_WIDTH }}>
<FormFactorCells m={row.desktop} />
</div>
<div className="w-[96px] shrink-0 truncate px-2 text-right text-surface-500">
{fetched ? relativeTime(fetched) : '—'}
</div>
</div>
);
}
@@ -0,0 +1,453 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { useVirtualizer } from '@tanstack/react-virtual';
import { useTranslation } from 'react-i18next';
import { Loader2 } from 'lucide-react';
import type { SpellingProgress, SpellingRow } from '@freecrawl/shared-types';
import { useAppStore } from '../store.js';
const TOOLBAR_HEIGHT = 36;
const HEADER_HEIGHT = 26;
const STATUS_BAR_HEIGHT = 22;
const ROW_HEIGHT = 28;
const PAGE_SIZE = 2000;
const POLL_MS_RUNNING = 5000;
/** The public LanguageTool API allows ~20 requests/min — warn past this. */
const CONFIRM_THRESHOLD = 20;
type FilterMode = 'all' | 'checked' | 'unchecked' | 'errors';
/** Colour the match count by severity. */
function countClass(n: number | null): string {
if (n === null) return 'text-surface-700';
if (n === 0) return 'text-emerald-400';
if (n <= 5) return 'text-amber-400';
return 'text-red-400';
}
function relativeTime(iso: string | null): string {
if (!iso) return '—';
const then = new Date(iso).getTime();
if (Number.isNaN(then)) return '—';
const min = Math.floor((Date.now() - then) / 60_000);
if (min < 1) return 'just now';
if (min < 60) return `${min}m ago`;
const hr = Math.floor(min / 60);
if (hr < 24) return `${hr}h ago`;
const day = Math.floor(hr / 24);
if (day < 30) return `${day}d ago`;
return new Date(iso).toLocaleDateString();
}
export function SpellingTab() {
const { t } = useTranslation();
const dataVersion = useAppStore((s) => s.dataVersion);
const crawlProgress = useAppStore((s) => s.progress);
const setSettingsOpen = useAppStore((s) => s.setSettingsOpen);
const [rows, setRows] = useState<SpellingRow[]>([]);
const [total, setTotal] = useState(0);
const [search, setSearch] = useState('');
const [filter, setFilter] = useState<FilterMode>('all');
const [selected, setSelected] = useState<Set<string>>(new Set());
const [running, setRunning] = useState(false);
const [progress, setProgress] = useState<SpellingProgress | null>(null);
const scrollRef = useRef<HTMLDivElement>(null);
const reload = useCallback(async () => {
const res = await window.freecrawl.spellingQuery({
limit: PAGE_SIZE,
offset: 0,
search: search || undefined,
filter,
});
setRows(res.rows);
setTotal(res.total);
}, [search, filter]);
useEffect(() => {
let cancelled = false;
void reload();
if (!crawlProgress?.running) return;
const id = setInterval(() => {
if (!cancelled) void reload();
}, POLL_MS_RUNNING);
return () => {
cancelled = true;
clearInterval(id);
};
}, [reload, dataVersion, crawlProgress?.running]);
useEffect(() => {
const off = window.freecrawl.onSpellingProgress((p) => {
setProgress(p.running ? p : null);
if (!p.currentUrl) void reload();
});
return off;
}, [reload]);
const virtualizer = useVirtualizer({
count: rows.length,
getScrollElement: () => scrollRef.current,
estimateSize: () => ROW_HEIGHT,
overscan: 16,
getItemKey: (index) => rows[index]?.url ?? `idx-${index}`,
});
const allLoadedSelected =
rows.length > 0 && rows.every((r) => selected.has(r.url));
const toggleRow = useCallback((url: string) => {
setSelected((prev) => {
const next = new Set(prev);
if (next.has(url)) next.delete(url);
else next.add(url);
return next;
});
}, []);
const toggleAll = useCallback(() => {
setSelected((prev) => {
if (rows.length > 0 && rows.every((r) => prev.has(r.url))) {
return new Set();
}
return new Set(rows.map((r) => r.url));
});
}, [rows]);
const selectUnchecked = useCallback(() => {
setSelected(new Set(rows.filter((r) => r.status === null).map((r) => r.url)));
}, [rows]);
const runCheck = useCallback(async () => {
const urls = [...selected];
if (urls.length === 0 || running) return;
if (
urls.length > CONFIRM_THRESHOLD &&
!window.confirm(
t('spellingTab.confirmRun', {
defaultValue:
'This will check {{count}} pages with LanguageTool. The free public API is rate-limited (~20 requests/min), so this may take a while. Continue?',
count: urls.length,
}),
)
) {
return;
}
setRunning(true);
setProgress({ done: 0, total: urls.length, currentUrl: null, running: true });
try {
await window.freecrawl.spellingRun({ urls });
} finally {
setRunning(false);
setProgress(null);
void reload();
}
}, [selected, running, t, reload]);
const cancelRun = useCallback(() => {
void window.freecrawl.spellingCancel();
}, []);
const checkedCount = useMemo(
() => rows.filter((r) => r.status !== null).length,
[rows],
);
const withErrors = useMemo(
() => rows.filter((r) => (r.matchCount ?? 0) > 0).length,
[rows],
);
return (
<div className="flex h-full w-full flex-col bg-surface-950">
{/* Toolbar */}
<div
className="flex items-center gap-2 border-b border-surface-800 bg-surface-900/40 px-3"
style={{ height: TOOLBAR_HEIGHT }}
>
<div className="text-[12px] font-semibold tracking-wide text-surface-100">
{t('spellingTab.title', { defaultValue: 'Spelling & Grammar' })}
</div>
<input
type="text"
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder={t('spellingTab.filterPlaceholder', {
defaultValue: 'Filter by URL…',
})}
className="h-6 w-56 rounded border border-surface-700 bg-surface-950 px-2 text-[11px] text-surface-100 placeholder-surface-500 focus:border-blue-500 focus:outline-none"
/>
<select
className="h-6 rounded border border-surface-700 bg-surface-950 px-1.5 text-[11px] text-surface-100 focus:border-blue-500 focus:outline-none"
value={filter}
onChange={(e) => setFilter(e.target.value as FilterMode)}
>
<option value="all">
{t('spellingTab.filterAll', { defaultValue: 'All pages' })}
</option>
<option value="checked">
{t('spellingTab.filterChecked', { defaultValue: 'Checked only' })}
</option>
<option value="unchecked">
{t('spellingTab.filterUnchecked', { defaultValue: 'Unchecked only' })}
</option>
<option value="errors">
{t('spellingTab.filterErrors', { defaultValue: 'With findings' })}
</option>
</select>
<button
type="button"
onClick={selectUnchecked}
disabled={running || rows.length === 0}
className="h-6 rounded border border-surface-700 bg-surface-800 px-2 text-[11px] text-surface-200 hover:bg-surface-700 disabled:cursor-not-allowed disabled:opacity-40"
>
{t('spellingTab.selectUnchecked', { defaultValue: 'Select unchecked' })}
</button>
<button
type="button"
onClick={() => setSettingsOpen(true)}
className="h-6 rounded border border-surface-700 bg-surface-800 px-2 text-[11px] text-surface-300 hover:bg-surface-700"
title={t('spellingTab.optionsTitle', {
defaultValue: 'Rule level and custom dictionary live in Settings → Spelling.',
})}
>
{t('spellingTab.options', { defaultValue: 'Options…' })}
</button>
<div className="ml-auto flex items-center gap-2">
{running && progress ? (
<>
<span className="flex items-center gap-1.5 text-[11px] text-surface-300">
<Loader2 size={12} className="animate-spin text-blue-400" />
{t('spellingTab.runProgress', {
defaultValue: 'Checking {{done}} / {{total}}',
done: progress.done,
total: progress.total,
})}
</span>
<button
type="button"
onClick={cancelRun}
className="h-6 rounded border border-red-700 bg-red-900/40 px-2.5 text-[11px] font-medium text-red-200 hover:bg-red-900/70"
>
{t('spellingTab.cancel', { defaultValue: 'Cancel' })}
</button>
</>
) : (
<button
type="button"
onClick={() => void runCheck()}
disabled={selected.size === 0 || running}
className="h-6 rounded bg-blue-600 px-3 text-[11px] font-medium text-white hover:bg-blue-500 disabled:cursor-not-allowed disabled:bg-surface-700 disabled:text-surface-500"
>
{t('spellingTab.run', {
defaultValue: 'Check Spelling ({{count}})',
count: selected.size,
})}
</button>
)}
</div>
</div>
{/* Column header */}
<div
className="flex shrink-0 select-none items-center border-b border-surface-800 bg-surface-900/60 text-[10px] font-medium text-surface-400"
style={{ height: HEADER_HEIGHT }}
>
<div className="flex w-[30px] items-center justify-center">
<input
type="checkbox"
checked={allLoadedSelected}
ref={(el) => {
if (el) el.indeterminate = selected.size > 0 && !allLoadedSelected;
}}
onChange={toggleAll}
disabled={rows.length === 0}
className="h-3 w-3 accent-blue-500"
/>
</div>
<div className="flex-1 px-2">
{t('spellingTab.colUrl', { defaultValue: 'URL' })}
</div>
<div className="w-[60px] shrink-0 text-center">
{t('spellingTab.colLang', { defaultValue: 'Lang' })}
</div>
<div className="w-[76px] shrink-0 text-right">
{t('spellingTab.colWords', { defaultValue: 'Words' })}
</div>
<div className="w-[80px] shrink-0 text-right">
{t('spellingTab.colIssues', { defaultValue: 'Findings' })}
</div>
<div className="w-[78px] shrink-0 text-center">
{t('spellingTab.colStatus', { defaultValue: 'Status' })}
</div>
<div className="w-[92px] shrink-0 px-2 text-right">
{t('spellingTab.colChecked', { defaultValue: 'Checked' })}
</div>
</div>
{/* Body */}
<div ref={scrollRef} className="flex-1 overflow-y-auto">
{rows.length === 0 ? (
<div className="flex h-full items-center justify-center px-6 text-center text-[12px] text-surface-500">
{total === 0
? t('spellingTab.emptyNoData', {
defaultValue:
'No internal HTML pages crawled yet — run a crawl, then select pages to check.',
})
: t('spellingTab.emptyNoMatch', {
defaultValue: 'No pages match the current filter.',
})}
</div>
) : (
<div
style={{
height: virtualizer.getTotalSize(),
width: '100%',
position: 'relative',
}}
>
{virtualizer.getVirtualItems().map((vRow) => {
const row = rows[vRow.index];
if (!row) return null;
return (
<div
key={vRow.key}
style={{
position: 'absolute',
top: 0,
left: 0,
width: '100%',
height: ROW_HEIGHT,
transform: `translateY(${vRow.start}px)`,
}}
>
<SpellingTableRow
row={row}
selected={selected.has(row.url)}
onToggle={() => toggleRow(row.url)}
/>
</div>
);
})}
</div>
)}
</div>
{/* Status bar */}
<div
className="flex items-center gap-3 border-t border-surface-800 bg-surface-900/40 px-3 text-[11px] text-surface-400"
style={{ height: STATUS_BAR_HEIGHT }}
>
<span>
{t('spellingTab.summary', {
defaultValue:
'{{shown}} pages · {{checked}} checked · {{errors}} with findings · {{selected}} selected',
shown: rows.length.toLocaleString(),
checked: checkedCount.toLocaleString(),
errors: withErrors.toLocaleString(),
selected: selected.size.toLocaleString(),
})}
{total > rows.length &&
' ' +
t('spellingTab.firstN', {
defaultValue: '(first {{n}})',
n: PAGE_SIZE.toLocaleString(),
})}
</span>
{running && progress?.currentUrl && (
<span className="truncate text-surface-500">
{t('spellingTab.checking', {
defaultValue: 'Checking: {{url}}',
url: progress.currentUrl,
})}
</span>
)}
</div>
</div>
);
}
function StatusBadge({ row }: { row: SpellingRow }) {
const { t } = useTranslation();
if (row.status === null) {
return <span className="text-surface-700">—</span>;
}
if (row.status === 'error') {
return (
<span
className="cursor-help font-semibold text-red-400"
title={row.error ?? 'Check failed'}
>
{t('spellingTab.statusError', { defaultValue: 'ERR' })}
</span>
);
}
if (row.status === 'skipped') {
return (
<span
className="text-surface-500"
title={t('spellingTab.statusSkippedTip', {
defaultValue: 'Too little prose on the page to check.',
})}
>
{t('spellingTab.statusSkipped', { defaultValue: 'skipped' })}
</span>
);
}
return (
<span className="text-emerald-400">
{t('spellingTab.statusOk', { defaultValue: 'ok' })}
</span>
);
}
function SpellingTableRow({
row,
selected,
onToggle,
}: {
row: SpellingRow;
selected: boolean;
onToggle: () => void;
}) {
return (
<div
onClick={onToggle}
className={`flex h-full cursor-pointer items-center text-[11px] ${
selected
? 'bg-blue-900/30'
: 'odd:bg-surface-900/20 hover:bg-surface-800/40'
}`}
>
<div className="flex w-[30px] items-center justify-center">
<input
type="checkbox"
checked={selected}
onChange={onToggle}
onClick={(e) => e.stopPropagation()}
className="h-3 w-3 accent-blue-500"
/>
</div>
<div className="flex-1 truncate px-2 text-surface-200" title={row.url}>
{row.url}
</div>
<div className="w-[60px] shrink-0 text-center text-surface-400">
{row.language ?? row.lang ?? '—'}
</div>
<div className="w-[76px] shrink-0 text-right tabular-nums text-surface-400">
{row.wordCount?.toLocaleString() ?? '—'}
</div>
<div
className={`w-[80px] shrink-0 text-right font-mono tabular-nums ${countClass(row.matchCount)}`}
>
{row.matchCount?.toLocaleString() ?? '—'}
</div>
<div className="w-[78px] shrink-0 text-center">
<StatusBadge row={row} />
</div>
<div className="w-[92px] shrink-0 truncate px-2 text-right text-surface-500">
{relativeTime(row.fetchedAt)}
</div>
</div>
);
}
@@ -615,6 +615,8 @@ export const COLUMN_SPECS: Record<TabKey, ColumnSpec[]> = {
'broken-links': [],
serp: [],
pagespeed: [],
crux: [],
spelling: [],
'search-console': [],
analytics: [],
ai: [],
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/mcp-server",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"type": "module",
"bin": {
+144 -2
View File
@@ -639,6 +639,71 @@ export function buildTools(): Tool[] {
filter: args.filter as 'all' | 'tested' | 'untested' | undefined,
}),
},
{
name: 'query_crux',
description:
'Chrome UX Report rows: real-user (field) 75th-percentile Core Web Vitals per URL (LCP/INP/CLS/FCP/TTFB) for phone + desktop. Populate via Settings → Integrations → Chrome UX Report, then run a batch from the desktop CrUX tab (or crux_run).',
inputSchema: {
type: 'object',
properties: {
search: { type: 'string' },
filter: { type: 'string', enum: ['all', 'tested', 'untested'] },
limit: { type: 'integer', minimum: 1, maximum: 1000 },
offset: { type: 'integer', minimum: 0 },
},
},
handler: (args, db) =>
db.queryCrux({
limit: clamp(args.limit, 1, 1000, 50),
offset: clamp(args.offset, 0, 10_000_000, 0),
search: typeof args.search === 'string' ? args.search : undefined,
filter: args.filter as 'all' | 'tested' | 'untested' | undefined,
}),
},
{
name: 'query_spelling',
description:
'Spelling & grammar check summary per crawled page (LanguageTool): match count, language used, status. Use filter "errors" for pages with ≥1 finding. Populate via Settings → Integrations → LanguageTool, then run a batch from the desktop Spelling tab (or spelling_run). Call get_url_spelling for a page\'s full match list.',
inputSchema: {
type: 'object',
properties: {
search: { type: 'string' },
filter: {
type: 'string',
enum: ['all', 'checked', 'unchecked', 'errors'],
},
limit: { type: 'integer', minimum: 1, maximum: 1000 },
offset: { type: 'integer', minimum: 0 },
},
},
handler: (args, db) =>
db.querySpelling({
limit: clamp(args.limit, 1, 1000, 50),
offset: clamp(args.offset, 0, 10_000_000, 0),
search: typeof args.search === 'string' ? args.search : undefined,
filter: args.filter as
| 'all'
| 'checked'
| 'unchecked'
| 'errors'
| undefined,
}),
},
{
name: 'get_url_spelling',
description:
'Full LanguageTool match list for one URL: message, offending text, surrounding context, suggested replacements, rule id and issue type per finding.',
inputSchema: {
type: 'object',
properties: { url: { type: 'string' } },
required: ['url'],
},
handler: (args, db) => {
const url = typeof args.url === 'string' ? args.url : '';
if (!url) throw new Error('url is required');
return db.getSpellingMatches(url) ?? { url, matches: [], status: null };
},
},
{
name: 'query_ai',
@@ -1695,6 +1760,77 @@ export function buildTools(): Tool[] {
bridgeRequest<unknown>('POST', '/v1/action/pagespeed-run', args),
},
{
requiresDb: false,
name: 'crux_run',
description:
'Kick off a Chrome UX Report (real-user field data) batch fetch in the desktop app and return immediately — it does NOT block. Poll `get_fetch_progress` (kind "crux") to watch it and `cancel_fetch` to stop. Pick targets with an explicit `urls` array OR a `category` resolved server-side. Requires a Google API key with the Chrome UX Report API enabled (Settings → Integrations → Chrome UX Report). Pages with too little real-user traffic simply return no data. Results land where `query_crux` reads. Returns `{started, total, truncated, state}`.',
inputSchema: {
type: 'object',
properties: {
urls: {
type: 'array',
items: { type: 'string' },
description: 'Explicit URLs to fetch. Wins over `category`.',
},
category: {
type: 'string',
description: 'Resolve target URLs from the active project by category when `urls` is omitted.',
},
search: {
type: 'string',
description: 'Optional substring filter applied when resolving via `category`.',
},
limit: {
type: 'integer',
minimum: 1,
maximum: 5000,
description: 'Max URLs to pull when resolving via `category`. Default 100.',
},
formFactor: {
type: 'string',
enum: ['phone', 'desktop', 'both'],
description: 'Device class. `both` doubles the API calls. Default "phone".',
},
},
},
handler: async (args) =>
bridgeRequest<unknown>('POST', '/v1/action/crux-run', args),
},
{
requiresDb: false,
name: 'spelling_run',
description:
'Kick off a LanguageTool spelling/grammar check over crawled pages in the desktop app and return immediately — it does NOT block. Poll `get_fetch_progress` (kind "spelling") to watch it and `cancel_fetch` to stop. Pick targets with an explicit `urls` array OR a `category` resolved server-side. Uses the free public LanguageTool API by default (rate-limited to ~20 req/min, so runs are serial); configure a self-hosted endpoint or Premium credentials under Settings → Integrations → LanguageTool for throughput. Results land where `query_spelling` / `get_url_spelling` read.',
inputSchema: {
type: 'object',
properties: {
urls: {
type: 'array',
items: { type: 'string' },
description: 'Explicit URLs to check. Wins over `category`.',
},
category: {
type: 'string',
description: 'Resolve target URLs from the active project by category when `urls` is omitted.',
},
search: {
type: 'string',
description: 'Optional substring filter applied when resolving via `category`.',
},
limit: {
type: 'integer',
minimum: 1,
maximum: 5000,
description: 'Max URLs to pull when resolving via `category`. Default 100.',
},
},
},
handler: async (args) =>
bridgeRequest<unknown>('POST', '/v1/action/spelling-run', args),
},
{
requiresDb: false,
name: 'ai_run',
@@ -1781,7 +1917,10 @@ export function buildTools(): Tool[] {
inputSchema: {
type: 'object',
properties: {
kind: { type: 'string', enum: ['pagespeed', 'ai', 'seo'] },
kind: {
type: 'string',
enum: ['pagespeed', 'crux', 'spelling', 'ai', 'seo'],
},
},
},
handler: async (args) =>
@@ -1796,7 +1935,10 @@ export function buildTools(): Tool[] {
inputSchema: {
type: 'object',
properties: {
kind: { type: 'string', enum: ['pagespeed', 'ai', 'seo'] },
kind: {
type: 'string',
enum: ['pagespeed', 'crux', 'spelling', 'ai', 'seo'],
},
},
},
handler: async (args) =>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "freecrawl-seo",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"description": "Open-source desktop SEO crawler — a free alternative to Screaming Frog",
"license": "MIT",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/core",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"type": "module",
"main": "./dist/index.js",
+22
View File
@@ -345,6 +345,28 @@ export interface ParsedPage {
* fingerprint uses this; `wordCount` / readability stay on the full
* body so existing thin-content thresholds don't shift.
*/
/**
* Extract human-readable prose from a raw HTML document, for grammar /
* spell checking. Unlike `extractMainContentText` (which feeds the
* duplicate fingerprint and keeps whatever `<body>` yields), this strips
* everything that would produce false positives in a language checker:
* script / style / noscript payloads, `<template>` and `<svg>` innards,
* and `<code>` / `<pre>` / `<kbd>` / `<samp>` blocks (identifiers are not
* prose). Page chrome (nav / header / footer / aside) is dropped via the
* same main-content heuristic so a shared menu isn't re-flagged on every
* page. Returns whitespace-collapsed, trimmed text — empty when the
* document carries no prose.
*/
export function extractProseText(
html: string,
contentAreaSelector?: string,
): string {
if (!html || html.trim().length === 0) return '';
const $ = cheerio.load(html);
$('script, style, noscript, template, svg, code, pre, kbd, samp').remove();
return extractMainContentText($, contentAreaSelector);
}
function extractMainContentText(
$: cheerio.CheerioAPI,
userSelector?: string,
+1
View File
@@ -35,6 +35,7 @@ export type {
export {
parseHtml,
estimatePixelWidth,
extractProseText,
type AnalyticsTracker,
type HreflangEntry,
} from './html-parser.js';
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/db",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"type": "module",
"main": "./dist/index.js",
+67
View File
@@ -1794,6 +1794,73 @@ const MIGRATIONS: Migration[] = [
}
},
},
{
version: 75,
name: 'add_crux_results_table',
// V2 — Chrome UX Report (CrUX) integration. Real-user field Core Web
// Vitals live in their own table, mirroring `pagespeed_results`: CrUX
// is fetched on-demand against a user-selected subset, each URL can
// have up to two rows (phone + desktop), and it keeps the very-wide
// `urls` table from growing further.
//
// url — the page URL (joins back to `urls.url`).
// form_factor — 'phone' | 'desktop'.
// lcp/inp/fcp/ttfb — p75 timing metrics in milliseconds.
// cls — p75 Cumulative Layout Shift (unitless).
// status — 'ok' | 'nodata' (too little traffic) | 'error'.
// error — failure reason when status='error'.
// collection_period — CrUX window end date (YYYY-MM-DD) when known.
// fetched_at — ISO timestamp of the fetch.
up: (db) => {
db.exec(`
CREATE TABLE IF NOT EXISTS crux_results (
url TEXT NOT NULL,
form_factor TEXT NOT NULL,
lcp REAL,
cls REAL,
inp REAL,
fcp REAL,
ttfb REAL,
status TEXT NOT NULL,
error TEXT,
collection_period TEXT,
fetched_at TEXT NOT NULL,
PRIMARY KEY (url, form_factor)
);
`);
},
},
{
version: 76,
name: 'add_spelling_results_table',
// V2 — Spelling & Grammar (LanguageTool). Like the other on-demand
// audit integrations this lives in its own url-keyed table rather than
// widening `urls`: checks run against a user-selected subset, and the
// per-page match list is a variable-length JSON blob.
//
// url — the checked page URL (joins back to `urls.url`).
// language — language code LanguageTool used (may be auto-detected).
// match_count — findings after the ignore-dictionary filter.
// matches — JSON array of SpellingMatch objects.
// status — 'ok' | 'skipped' (too little prose) | 'error'.
// error — failure reason when status='error'.
// fetched_at — ISO timestamp of the check.
up: (db) => {
db.exec(`
CREATE TABLE IF NOT EXISTS spelling_results (
url TEXT PRIMARY KEY,
language TEXT,
match_count INTEGER NOT NULL DEFAULT 0,
matches TEXT,
status TEXT NOT NULL,
error TEXT,
fetched_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_spelling_match_count
ON spelling_results(match_count);
`);
},
},
];
export function runMigrations(db: DatabaseSync): void {
+332 -11
View File
@@ -19,6 +19,12 @@ import type {
PagespeedMetrics,
PagespeedRow,
PagespeedStrategy,
CruxMetrics,
CruxRow,
CruxFormFactor,
SpellingResult,
SpellingRow,
SpellingMatch,
GscRow,
Ga4Row,
AiProvider,
@@ -785,6 +791,8 @@ export class ProjectDb {
DELETE FROM crawl_queue;
DELETE FROM host_certs;
DELETE FROM pagespeed_results;
DELETE FROM crux_results;
DELETE FROM spelling_results;
DELETE FROM gsc_results;
DELETE FROM ga4_results;
DELETE FROM ai_results;
@@ -910,10 +918,12 @@ export class ProjectDb {
WHERE id NOT IN (SELECT DISTINCT image_id FROM image_usages)`,
);
// PageSpeed + Search Console rows are keyed by URL string (no FK
// to `urls`), so wipe them by the same host match — otherwise a
// GDPR delete leaves stale data that would re-attach if the URL
// is re-crawled later.
// The url-keyed audit tables (PageSpeed, CrUX, Search Console,
// Analytics, AI, SEO) have no FK to `urls`, so wipe them by the same
// host match — otherwise a GDPR delete leaves stale data that would
// re-attach if the URL is re-crawled later. Driven off
// `URL_KEYED_AUDIT_TABLES` so a newly-added audit table can never be
// silently missed here.
const hostMatchSql = (table: string): string =>
`DELETE FROM ${table}
WHERE LOWER(
@@ -927,12 +937,9 @@ export class ProjectDb {
END
)
) = ?`;
this.db.prepare(hostMatchSql('pagespeed_results')).run(target);
this.db.prepare(hostMatchSql('gsc_results')).run(target);
this.db.prepare(hostMatchSql('ga4_results')).run(target);
this.db.prepare(hostMatchSql('ai_results')).run(target);
this.db.prepare(hostMatchSql('seo_results')).run(target);
this.db.prepare(hostMatchSql('gsc_inspection_results')).run(target);
for (const table of ProjectDb.URL_KEYED_AUDIT_TABLES) {
this.db.prepare(hostMatchSql(table)).run(target);
}
return { urlsDeleted, linksDeleted };
});
@@ -1114,9 +1121,11 @@ export class ProjectDb {
*/
/** url-keyed audit tables with no FK to `urls` — must be wiped explicitly
* on delete (ON DELETE CASCADE doesn't reach them). Kept in sync with
* `deleteByDomain`, which clears the same six. */
* `deleteByDomain`, which clears the same set. */
private static readonly URL_KEYED_AUDIT_TABLES = [
'pagespeed_results',
'crux_results',
'spelling_results',
'gsc_results',
'ga4_results',
'ai_results',
@@ -3451,6 +3460,307 @@ export class ProjectDb {
};
}
/**
* Store one CrUX field-data record, keyed by (url, form_factor);
* re-fetching overwrites the previous row.
*/
upsertCruxResult(
url: string,
formFactor: CruxFormFactor,
m: CruxMetrics,
): void {
this.db
.prepare(
`INSERT INTO crux_results
(url, form_factor, lcp, cls, inp, fcp, ttfb,
status, error, collection_period, fetched_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(url, form_factor) DO UPDATE SET
lcp = excluded.lcp,
cls = excluded.cls,
inp = excluded.inp,
fcp = excluded.fcp,
ttfb = excluded.ttfb,
status = excluded.status,
error = excluded.error,
collection_period = excluded.collection_period,
fetched_at = excluded.fetched_at`,
)
.run(
url,
formFactor,
m.lcp,
m.cls,
m.inp,
m.fcp,
m.ttfb,
m.status,
m.error,
m.collectionPeriod,
m.fetchedAt,
);
}
/**
* Candidate query for the CrUX tab. Returns crawled internal HTML 2xx
* pages left-joined with any stored phone + desktop field records.
*/
queryCrux(params: {
limit: number;
offset: number;
search?: string;
filter?: 'all' | 'tested' | 'untested';
}): { rows: CruxRow[]; total: number } {
const where: string[] = [
"u.is_external = 0 AND u.content_kind = 'html'",
'u.status_code >= 200 AND u.status_code < 300',
];
const args: (string | number)[] = [];
if (params.search) {
where.push('u.url LIKE ?');
args.push(`%${params.search}%`);
}
const filter = params.filter ?? 'all';
if (filter === 'tested') {
where.push('(cp.url IS NOT NULL OR cd.url IS NOT NULL)');
} else if (filter === 'untested') {
where.push('cp.url IS NULL AND cd.url IS NULL');
}
const whereSql = `WHERE ${where.join(' AND ')}`;
const joinSql = `
FROM urls u
LEFT JOIN crux_results cp ON cp.url = u.url AND cp.form_factor = 'phone'
LEFT JOIN crux_results cd ON cd.url = u.url AND cd.form_factor = 'desktop'`;
const totalRow = this.db
.prepare(`SELECT COUNT(*) AS c ${joinSql} ${whereSql}`)
.get(...args) as { c: number };
const rowsDb = this.db
.prepare(
`SELECT u.url AS url, u.status_code AS status_code,
cp.lcp AS p_lcp, cp.cls AS p_cls, cp.inp AS p_inp,
cp.fcp AS p_fcp, cp.ttfb AS p_ttfb, cp.status AS p_status,
cp.error AS p_error, cp.collection_period AS p_period, cp.fetched_at AS p_fetched,
cd.lcp AS d_lcp, cd.cls AS d_cls, cd.inp AS d_inp,
cd.fcp AS d_fcp, cd.ttfb AS d_ttfb, cd.status AS d_status,
cd.error AS d_error, cd.collection_period AS d_period, cd.fetched_at AS d_fetched
${joinSql}
${whereSql}
ORDER BY u.inlinks DESC, u.url ASC
LIMIT ? OFFSET ?`,
)
.all(...args, params.limit, params.offset) as unknown as Record<
string,
unknown
>[];
const metrics = (
r: Record<string, unknown>,
p: string,
): CruxMetrics | null => {
const status = r[`${p}_status`];
if (typeof status !== 'string') return null;
return {
lcp: (r[`${p}_lcp`] as number | null) ?? null,
cls: (r[`${p}_cls`] as number | null) ?? null,
inp: (r[`${p}_inp`] as number | null) ?? null,
fcp: (r[`${p}_fcp`] as number | null) ?? null,
ttfb: (r[`${p}_ttfb`] as number | null) ?? null,
status:
status === 'ok' ? 'ok' : status === 'nodata' ? 'nodata' : 'error',
error: (r[`${p}_error`] as string | null) ?? null,
collectionPeriod: (r[`${p}_period`] as string | null) ?? null,
fetchedAt: (r[`${p}_fetched`] as string | null) ?? '',
};
};
return {
total: totalRow.c,
rows: rowsDb.map((r) => ({
url: r['url'] as string,
statusCode: (r['status_code'] as number | null) ?? null,
phone: metrics(r, 'p'),
desktop: metrics(r, 'd'),
})),
};
}
/**
* Raw stored HTML body + declared `html[lang]` for one URL. Feeds the
* spelling check, which prefers the crawl-time snapshot over re-fetching
* the page. `body` is null when body snapshots were disabled for the
* crawl, in which case the caller re-fetches.
*/
getUrlBodyAndLang(
url: string,
): { body: string | null; lang: string | null } | null {
const row = this.db
.prepare(
`SELECT u.lang AS lang, s.body AS body
FROM urls u
LEFT JOIN url_sources s ON s.url_id = u.id
WHERE u.url = ?`,
)
.get(url) as { lang: string | null; body: string | null } | undefined;
if (!row) return null;
return { body: row.body ?? null, lang: row.lang ?? null };
}
/**
* Store one LanguageTool check result, keyed by url; re-checking a page
* overwrites the previous row. `matches` is persisted as a JSON array.
*/
upsertSpellingResult(url: string, r: SpellingResult): void {
this.db
.prepare(
`INSERT INTO spelling_results
(url, language, match_count, matches, status, error, fetched_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(url) DO UPDATE SET
language = excluded.language,
match_count = excluded.match_count,
matches = excluded.matches,
status = excluded.status,
error = excluded.error,
fetched_at = excluded.fetched_at`,
)
.run(
url,
r.language,
r.matchCount,
r.matches.length > 0 ? JSON.stringify(r.matches) : null,
r.status,
r.error,
r.fetchedAt,
);
}
/**
* Candidate query for the Spelling tab. Returns crawled internal HTML
* 2xx pages left-joined with any stored check summary. The full match
* list is deliberately NOT selected here — it can be thousands of
* entries per page; the Detail panel loads it per-URL on demand.
*/
querySpelling(params: {
limit: number;
offset: number;
search?: string;
filter?: 'all' | 'checked' | 'unchecked' | 'errors';
}): { rows: SpellingRow[]; total: number } {
const where: string[] = [
"u.is_external = 0 AND u.content_kind = 'html'",
'u.status_code >= 200 AND u.status_code < 300',
];
const args: (string | number)[] = [];
if (params.search) {
where.push('u.url LIKE ?');
args.push(`%${params.search}%`);
}
const filter = params.filter ?? 'all';
if (filter === 'checked') where.push('s.url IS NOT NULL');
else if (filter === 'unchecked') where.push('s.url IS NULL');
else if (filter === 'errors') where.push('s.match_count > 0');
const whereSql = `WHERE ${where.join(' AND ')}`;
const joinSql = `
FROM urls u
LEFT JOIN spelling_results s ON s.url = u.url`;
const totalRow = this.db
.prepare(`SELECT COUNT(*) AS c ${joinSql} ${whereSql}`)
.get(...args) as { c: number };
const rowsDb = this.db
.prepare(
`SELECT u.url AS url, u.lang AS lang, u.word_count AS word_count,
s.language AS language, s.match_count AS match_count,
s.status AS status, s.error AS error, s.fetched_at AS fetched_at
${joinSql}
${whereSql}
ORDER BY s.match_count DESC, u.inlinks DESC, u.url ASC
LIMIT ? OFFSET ?`,
)
.all(...args, params.limit, params.offset) as unknown as Record<
string,
unknown
>[];
return {
total: totalRow.c,
rows: rowsDb.map((r) => {
const status = r['status'];
return {
url: r['url'] as string,
lang: (r['lang'] as string | null) ?? null,
wordCount: (r['word_count'] as number | null) ?? null,
language: (r['language'] as string | null) ?? null,
matchCount: (r['match_count'] as number | null) ?? null,
status:
status === 'ok' || status === 'skipped' || status === 'error'
? status
: null,
error: (r['error'] as string | null) ?? null,
fetchedAt: (r['fetched_at'] as string | null) ?? null,
};
}),
};
}
/**
* Full stored match list for one URL — drives the Detail panel's
* Spelling & Grammar sub-tab. Returns null when the page was never
* checked. A corrupt `matches` blob degrades to an empty list rather
* than throwing.
*/
getSpellingMatches(url: string): {
url: string;
language: string | null;
status: 'ok' | 'skipped' | 'error' | null;
error: string | null;
fetchedAt: string | null;
matches: SpellingMatch[];
} | null {
const row = this.db
.prepare(
`SELECT url, language, matches, status, error, fetched_at
FROM spelling_results WHERE url = ?`,
)
.get(url) as
| {
url: string;
language: string | null;
matches: string | null;
status: string;
error: string | null;
fetched_at: string | null;
}
| undefined;
if (!row) return null;
let matches: SpellingMatch[] = [];
if (row.matches) {
try {
const parsed = JSON.parse(row.matches) as unknown;
if (Array.isArray(parsed)) matches = parsed as SpellingMatch[];
} catch {
// Corrupt blob — surface the summary without the list.
}
}
const status = row.status;
return {
url: row.url,
language: row.language,
status:
status === 'ok' || status === 'skipped' || status === 'error'
? status
: null,
error: row.error,
fetchedAt: row.fetched_at,
matches,
};
}
/**
* Faz 7 — wholesale-replace the stored Google Search Console data.
* Each GSC pull is a fresh snapshot for a date range, so the old rows
@@ -4384,6 +4694,12 @@ export class ProjectDb {
)`,
),
contentThin: countWhere(`${html} AND word_count IS NOT NULL AND word_count < 300`),
spellingGrammar: countWhere(
`${html} AND EXISTS (
SELECT 1 FROM spelling_results s
WHERE s.url = urls.url AND s.match_count > 0
)`,
),
responseSlow: countWhere(`${html} AND response_time_ms > 1000`),
responseVerySlow: countWhere(`${html} AND response_time_ms > 3000`),
pageLarge: countWhere(`${html} AND content_length > 1048576`),
@@ -7836,6 +8152,11 @@ function categoryWhereClause(cat: UrlCategory): string | null {
)`;
case 'issues:content-thin':
return "is_external = 0 AND content_kind = 'html' AND word_count IS NOT NULL AND word_count < 300";
case 'issues:spelling-grammar':
return `is_external = 0 AND content_kind = 'html' AND EXISTS (
SELECT 1 FROM spelling_results s
WHERE s.url = urls.url AND s.match_count > 0
)`;
case 'issues:response-slow':
return "is_external = 0 AND content_kind = 'html' AND response_time_ms > 1000";
case 'issues:response-very-slow':
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@freecrawl/shared-types",
"version": "0.8.2",
"version": "0.8.5",
"private": true,
"type": "module",
"main": "./dist/index.js",
+3
View File
@@ -68,6 +68,7 @@ export type UrlCategory =
| 'issues:canonical-to-redirect'
| 'issues:canonical-to-noindex'
| 'issues:content-thin'
| 'issues:spelling-grammar'
| 'issues:response-slow'
| 'issues:response-very-slow'
| 'issues:page-large'
@@ -1392,6 +1393,8 @@ export interface OverviewCounts {
canonicalToRedirect: number;
canonicalToNoindex: number;
contentThin: number;
/** Pages whose LanguageTool check returned ≥1 spelling/grammar match. */
spellingGrammar: number;
responseSlow: number;
responseVerySlow: number;
pageLarge: number;
+63
View File
@@ -0,0 +1,63 @@
/**
* Chrome UX Report (CrUX) integration types.
*
* CrUX exposes **field** data — the 75th-percentile Core Web Vitals of
* real Chrome users over a rolling 28-day window — via the CrUX API
* (`chromeuxreport.googleapis.com`). Unlike PageSpeed Insights (which is
* synthetic *lab* data from a single Lighthouse run), CrUX answers "how
* fast is this page for actual visitors?". It is per-URL and per-origin,
* sub-second to fetch, and requires a Google API key (no keyless mode).
*
* FreeCrawl drives it on-demand against a user-selected subset of crawled
* pages and stores results per URL + form factor in the project DB. The
* key is resolved in the main process only — it never crosses into the
* renderer.
*/
/** Which device class CrUX aggregates real-user data for. */
export type CruxFormFactor = 'phone' | 'desktop';
/** A run can target one form factor or both. */
export type CruxRunFormFactor = CruxFormFactor | 'both';
/**
* Field (real-user) Core Web Vitals for one URL + form factor, each the
* 75th-percentile value across the collection window. Timing values are
* milliseconds; `cls` is the unitless Cumulative Layout Shift score. Any
* metric may be `null` when CrUX has no percentile for it.
*/
export interface CruxMetrics {
/** Largest Contentful Paint p75 (ms). */
lcp: number | null;
/** Cumulative Layout Shift p75 (unitless). */
cls: number | null;
/** Interaction to Next Paint p75 (ms). */
inp: number | null;
/** First Contentful Paint p75 (ms). */
fcp: number | null;
/** Time to First Byte p75 (ms) — experimental CrUX metric. */
ttfb: number | null;
/**
* `ok` — data returned; `nodata` — URL/origin has too little real-user
* traffic for CrUX (HTTP 404, not an error); `error` — request failed.
*/
status: 'ok' | 'nodata' | 'error';
/** Human-readable failure reason when `status` is `error`. */
error: string | null;
/** Collection-period end date (`YYYY-MM-DD`) when available. */
collectionPeriod: string | null;
/** ISO 8601 timestamp of when the record was fetched. */
fetchedAt: string;
}
/**
* One row in the CrUX tab — a crawled internal HTML page with its stored
* field metrics. `phone` / `desktop` are `null` until that form factor
* has been fetched for the URL.
*/
export interface CruxRow {
url: string;
statusCode: number | null;
phone: CruxMetrics | null;
desktop: CruxMetrics | null;
}
+2
View File
@@ -4,6 +4,8 @@ export * from './ipc.js';
export * from './project.js';
export * from './integrations.js';
export * from './pagespeed.js';
export * from './crux.js';
export * from './spelling.js';
export * from './google.js';
export * from './ai.js';
export * from './seo.js';
+46
View File
@@ -161,6 +161,52 @@ export const INTEGRATIONS: IntegrationDef[] = [
setupHint:
"A free API key is required — Google's keyless shared quota is now 0/day, so anonymous audits no longer work. Create a key in Google Cloud Console (no billing required), enable the PageSpeed Insights API, then paste the key here. Free tier: 25 000 queries/day, 240/min.",
},
{
id: 'crux',
name: 'Chrome UX Report (CrUX)',
category: 'performance',
authType: 'api-key',
description: 'Real-user field Core Web Vitals per URL (p75 LCP / INP / CLS / FCP / TTFB).',
fields: [KEY_FIELD],
helpUrl: 'https://developer.chrome.com/docs/crux/api',
setupHint:
'An API key is required (no keyless mode). Create a key in Google Cloud Console (no billing required), enable the "Chrome UX Report API", then paste the key here. A single Google API key can serve both this and PageSpeed Insights once both APIs are enabled. Pages with too little real-user traffic simply return no data.',
},
// ---- Content quality ----
{
id: 'languagetool',
name: 'LanguageTool',
category: 'seo',
authType: 'local',
description: 'Spelling, grammar and style checking per page.',
fields: [
{
key: 'endpoint',
label: 'API endpoint',
secret: false,
placeholder: 'https://api.languagetool.org',
optional: true,
},
{
key: 'username',
label: 'Premium username',
secret: false,
optional: true,
placeholder: 'Account email (Premium only)',
},
{
key: 'apiKey',
label: 'Premium API Key',
secret: true,
optional: true,
placeholder: 'Premium API key (optional)',
},
],
helpUrl: 'https://languagetool.org/http-api/',
setupHint:
'Leave everything blank to use the free public API (rate-limited to ~20 requests/min). For unlimited checks, self-host LanguageTool and point the endpoint at it (e.g. http://localhost:8081), or paste a Premium username + API key.',
},
// ---- SEO data ----
{
+134
View File
@@ -23,6 +23,8 @@ import type {
PagespeedRow,
PagespeedRunStrategy,
} from './pagespeed.js';
import type { CruxRow, CruxRunFormFactor } from './crux.js';
import type { SpellingMatch, SpellingRow } from './spelling.js';
import type {
GoogleAuthState,
GoogleAuthResult,
@@ -239,6 +241,26 @@ export const IPC = {
pagespeedCancel: 'pagespeed:cancel',
/** main → renderer: live progress of an in-flight PageSpeed run. */
pagespeedProgress: 'pagespeed:progress',
/** Chrome UX Report (CrUX) integration — real-user field metrics.
* `cruxQuery` lists crawled internal HTML pages joined with any stored
* field data; `cruxRun` fetches a user-selected set of URLs (emits
* `cruxProgress`); `cruxCancel` stops an in-flight run early. */
cruxQuery: 'crux:query',
cruxRun: 'crux:run',
cruxCancel: 'crux:cancel',
/** main → renderer: live progress of an in-flight CrUX run. */
cruxProgress: 'crux:progress',
/** Spelling & Grammar (LanguageTool) integration.
* `spellingQuery` lists crawled internal HTML pages joined with any
* stored check summary; `spellingRun` checks a user-selected set of
* URLs (emits `spellingProgress`); `spellingCancel` stops a run early;
* `spellingMatches` loads the full match list for one URL. */
spellingQuery: 'spelling:query',
spellingRun: 'spelling:run',
spellingCancel: 'spelling:cancel',
spellingMatches: 'spelling:matches',
/** main → renderer: live progress of an in-flight spelling run. */
spellingProgress: 'spelling:progress',
/** Faz 7 — Google OAuth keystone (shared by Search Console, GA4,
* Sheets). `googleAuthStart` opens the consent screen in the browser
* and catches the loopback redirect; `googleAuthStatus` reports the
@@ -1198,6 +1220,96 @@ export interface PagespeedProgress {
running: boolean;
}
/**
* Inputs for the CrUX tab's candidate query — crawled internal HTML pages
* left-joined with any stored field data.
*/
export interface CruxQueryInput {
limit: number;
offset: number;
/** Substring match against the URL. */
search?: string;
/** `all` (default), `tested` (has ≥1 stored result), `untested`. */
filter?: 'all' | 'tested' | 'untested';
}
export interface CruxQueryResult {
rows: CruxRow[];
/** Total candidate rows matching the filter (before pagination). */
total: number;
}
export interface CruxRunInput {
/** URLs to fetch — taken from the candidate rows in the tab. */
urls: string[];
/** Which form factor(s) to fetch. `both` doubles the API calls. */
formFactor: CruxRunFormFactor;
}
export interface CruxRunResult {
/** Number of (url × form-factor) records fetched with data. */
completed: number;
/** Number that returned an error or no data. */
failed: number;
/** True when the user cancelled the run before it finished. */
cancelled: boolean;
}
/** main → renderer live progress while a CrUX run is in flight. */
export interface CruxProgress {
done: number;
total: number;
currentUrl: string | null;
running: boolean;
}
/** Inputs for the Spelling tab's candidate query. */
export interface SpellingQueryInput {
limit: number;
offset: number;
/** Substring match against the URL. */
search?: string;
/** `all` (default), `checked`, `unchecked`, `errors` (matchCount > 0). */
filter?: 'all' | 'checked' | 'unchecked' | 'errors';
}
export interface SpellingQueryResult {
rows: SpellingRow[];
/** Total candidate rows matching the filter (before pagination). */
total: number;
}
export interface SpellingRunInput {
/** URLs to check — taken from the candidate rows in the tab. */
urls: string[];
}
export interface SpellingRunResult {
/** Pages checked successfully (including clean pages). */
completed: number;
/** Pages that errored or were skipped for having no prose. */
failed: number;
cancelled: boolean;
}
/** main → renderer live progress while a spelling run is in flight. */
export interface SpellingProgress {
done: number;
total: number;
currentUrl: string | null;
running: boolean;
}
/** Full stored match list for one URL (Detail panel sub-tab). */
export interface SpellingMatchesResult {
url: string;
language: string | null;
status: 'ok' | 'skipped' | 'error' | null;
error: string | null;
fetchedAt: string | null;
matches: SpellingMatch[];
}
/** Result of `gscListSites` — the connected account's GSC properties. */
export interface GscListSitesResult {
ok: boolean;
@@ -1543,6 +1655,28 @@ export interface FreeCrawlApi {
pagespeedCancel(): Promise<void>;
/** Live progress of an in-flight PageSpeed run. */
onPagespeedProgress(cb: (p: PagespeedProgress) => void): () => void;
/** List crawled internal HTML pages joined with their stored CrUX
* real-user field metrics. */
cruxQuery(input: CruxQueryInput): Promise<CruxQueryResult>;
/** Fetch CrUX field data for the given URLs. Resolves when the run
* finishes; subscribe to `onCruxProgress` for live progress. */
cruxRun(input: CruxRunInput): Promise<CruxRunResult>;
/** Request cancellation of an in-flight CrUX run. */
cruxCancel(): Promise<void>;
/** Live progress of an in-flight CrUX run. */
onCruxProgress(cb: (p: CruxProgress) => void): () => void;
/** List crawled internal HTML pages joined with their stored
* spelling / grammar check summary. */
spellingQuery(input: SpellingQueryInput): Promise<SpellingQueryResult>;
/** Run a LanguageTool check over the given URLs. Resolves when the run
* finishes; subscribe to `onSpellingProgress` for live progress. */
spellingRun(input: SpellingRunInput): Promise<SpellingRunResult>;
/** Request cancellation of an in-flight spelling run. */
spellingCancel(): Promise<void>;
/** Live progress of an in-flight spelling run. */
onSpellingProgress(cb: (p: SpellingProgress) => void): () => void;
/** Full stored match list for one URL. */
spellingMatches(url: string): Promise<SpellingMatchesResult | null>;
/** Faz 7 — start the interactive Google OAuth consent flow for one
* integration. Resolves once the user finishes (or cancels) in the
* browser. */
@@ -65,6 +65,7 @@ const WARNING: UrlCategory[] = [
'issues:canonical-missing',
'issues:canonical-to-redirect',
'issues:content-thin',
'issues:spelling-grammar',
'issues:response-slow',
'issues:page-large',
'issues:viewport-missing',
+75
View File
@@ -0,0 +1,75 @@
/**
* Spelling & Grammar integration types (LanguageTool).
*
* FreeCrawl sends each selected page's prose to a LanguageTool `/v2/check`
* endpoint — either the free public API, a self-hosted server, or a
* Premium account — and stores the returned matches per URL.
*
* Language is taken from the page's `html[lang]` when present, falling
* back to LanguageTool's own auto-detection. Results are fetched
* on-demand against a user-selected subset (checking a whole crawl would
* blow through the public API's rate limit), so this mirrors the
* PageSpeed / CrUX batch model.
*/
/** LanguageTool rule strictness. `picky` enables style/typography rules. */
export type SpellingLevel = 'default' | 'picky';
/** One spelling / grammar / style finding inside a page's prose. */
export interface SpellingMatch {
/** Full human-readable explanation from LanguageTool. */
message: string;
/** Terse variant, when LanguageTool supplies one. */
shortMessage: string | null;
/** Character offset of the finding within the checked text. */
offset: number;
/** Character length of the offending span. */
length: number;
/** The offending surface text itself. */
text: string;
/** Surrounding snippet, for display. */
context: string;
/** Offset of `text` inside `context` — used to highlight the span. */
contextOffset: number;
/** Suggested corrections, best-first (may be empty). */
replacements: string[];
/** LanguageTool rule id, e.g. `MORFOLOGIK_RULE_EN_US`. */
ruleId: string;
/** Human-readable rule category, e.g. `Possible Typo`. */
category: string;
/** LanguageTool issue type: `misspelling`, `grammar`, `style`, … */
issueType: string;
}
/** Stored check result for one URL. */
export interface SpellingResult {
/** Language code LanguageTool actually used (may be auto-detected). */
language: string | null;
/** Number of matches after the ignore-dictionary filter. */
matchCount: number;
matches: SpellingMatch[];
/**
* `ok` — checked; `skipped` — page had too little prose to check;
* `error` — the request failed.
*/
status: 'ok' | 'skipped' | 'error';
error: string | null;
fetchedAt: string;
}
/**
* One row in the Spelling tab — a crawled internal HTML page with its
* stored check summary. Result fields are `null` until the page is checked.
*/
export interface SpellingRow {
url: string;
/** `html[lang]` captured during the crawl. */
lang: string | null;
wordCount: number | null;
/** Language LanguageTool used for the check. */
language: string | null;
matchCount: number | null;
status: 'ok' | 'skipped' | 'error' | null;
error: string | null;
fetchedAt: string | null;
}