mirror of
https://github.com/kemalai/FreeCrawl-SEO-Tool.git
synced 2026-10-18 00:39:38 +02:00
feat: multi-window projects + Projects manager
This commit is contained in:
1 parent
e4df72061f
commit
72268f56ba
24 files changed
+2006
-803
No files matched your search
@@ -29,11 +29,13 @@ A high-performance website crawler built for serious SEO audits. Targets <kbd>1M
|
||||
|
||||
## ✨ What FreeCrawl Does
|
||||
|
||||
FreeCrawl is a desktop SEO crawler that scales to **1M+ URLs on a single machine** at 80–150 URL/s via undici with keep-alive, with optional **JavaScript rendering** through headless Chromium (Playwright) that captures the post-JS DOM, full-page / above-fold / mobile screenshots, LCP candidate elements, and a per-URL mobile usability audit. It runs **150+ on-page SEO checks across 32 top-level tabs** — exact and near-duplicate clustering (SimHash + LSH) with a dedicated Cluster view, full hreflang validation (reciprocity, self-reference, inconsistent lang), OpenGraph / Twitter Card / JSON-LD / Web App Manifest parsing, AMP smoke validation, structured-data validation across 17 schema types (duplicate `@id`, malformed `@type`, missing required + Google-recommended props), a configurable performance budget (response-time / page-size / LCP / CLS ceilings), WCAG accessibility checks (`<main>` landmark, skip-link, ARIA roles, heading order), security headers + SSL/TLS chain audit + active/passive mixed content split, readability scores (Flesch, Flesch–Kincaid, Gunning Fog), **spelling & grammar checking via LanguageTool** (per-page findings with in-context highlighting and a custom dictionary), and a 10-rule custom extraction engine — **CSS selectors, XPath, regex, and JSONPath (for JSON API responses)** — with a **live Preview dialog and JSON import / export** for sharing rule sets.
|
||||
FreeCrawl crawls your website and checks it for SEO problems — a free, desktop alternative to Screaming Frog. It handles very large sites (**1M+ URLs on one machine**) and streams the results into a fast, dense table while the crawl is still running.
|
||||
|
||||
The **dense dark UI** renders virtualized 1M-row tables that live-stream rows during a crawl, with **List ↔ Tree ↔ Cluster view toggles**, **column pin (sticky-left) + drag-to-reorder + show/hide**, advanced AND/OR filters across 24 fields × 12 operators, per-tab quick-filter dropdowns, a 17-tab Details panel per URL (URL Details, Outline, Inlinks, Outlinks, Images, Resources, Extracted Data, SERP Snippet, HTTP Headers, Cookies, Structured Data, View Source, View Rendered, Screenshot, Duplicates, Analytics, Spelling & Grammar), an English + Turkish UI including every Settings panel, **in-app scheduled crawls** (hourly / daily / weekly / custom), a live memory monitor, robots.txt syntax validator, URL rewriting with live preview, project-vs-project compare diff, a Cytoscape link graph in its own native window (force-directed, BFS / radial / Sugiyama tree, and directory-tree layouts, a **By-LCP above-the-fold colour overlay**, a **By-Link-Score (internal PageRank) colour overlay** that shows where internal link equity concentrates, and a click-to-trace **Crawl Path Report** that highlights the shortest discovery path from the homepage to any page), a **standalone Log File Analyzer window** (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, and crawl × log orphan detection — with 40+ bot detection, optional reverse-DNS verification, per-bot URL filtering, and one-click CSV / Excel export of every table), and a `.seoproject` file association for OS double-click.
|
||||
It runs **150+ SEO checks across 32 tabs** — page titles, meta descriptions, headings, duplicate content, broken links, redirects, canonicals, hreflang, structured data, security headers, readability, and more. It can also render JavaScript pages with a real headless browser to capture the final DOM, screenshots, and Core Web Vitals.
|
||||
|
||||
Integrations cover **Google Search Console** (clicks, impressions, CTR, position + URL Inspection coverage), **Google Analytics 4** (sessions, users, bounce, engagement), **PageSpeed Insights** Lighthouse lab audits, **Chrome UX Report (CrUX)** real-user field Core Web Vitals (p75 LCP / INP / CLS / FCP / TTFB, phone + desktop), **LanguageTool** spelling & grammar checking (public API, self-hosted, or Premium), custom **AI prompts via OpenAI / Anthropic Claude / local Ollama** with `{url}`/`{title}`/`{description}`/`{h1}`/`{body}` variables, and **SEO authority providers** (Ahrefs / Majestic / Moz / Semrush) behind a single dropdown — each integration card now ships with an **in-app step-by-step Setup Guide modal** in English and Turkish that walks you through OAuth client creation, test-user setup, API enablement, and the most common errors. Exports go through a unified **Export Crawl Data dialog** (Excel `.xlsx` / CSV UTF-8 / JSON / XML with hierarchical category selection and nested folder output) plus a standalone HTML audit report, sitemap generator (standard / image / hreflang / sharded / gzipped), and direct streaming to **Google Sheets** and **BigQuery**. Project files can be saved as **password-protected encrypted snapshots** (`.seoproject.enc`, AES-256-GCM + PBKDF2). The **MCP server** exposes **96 tools** that let Claude Code or any MCP client drive crawls live, query every UI surface, run single or bulk exports, save / open encrypted project snapshots, fetch GSC/GA4 data on demand, and modify settings — every action a human user can take in the desktop is callable from an agent. Crawl-completion webhooks, OS notifications, 22 built-in reports (histograms, top/bottom URLs, link positions, top words, cross-source orphans), per-URL **Duplicates** view, and a custom-CSS theme override round out the suite. Everything runs **fully local** — no telemetry, no cloud, MIT-style license.
|
||||
Pull in real data from **Google Search Console, Analytics 4, PageSpeed Insights, and Chrome UX Report**, check spelling and grammar with **LanguageTool**, and run your own **AI prompts** on each page (OpenAI, Claude, or Ollama). Open several projects side by side, each in its own window running its own crawl, and manage them from a Projects dialog with search, tags, and starter templates.
|
||||
|
||||
Export to **Excel, CSV, JSON, XML**, an HTML report, sitemaps, **Google Sheets**, or **BigQuery**, and save projects as **password-protected encrypted files**. A built-in **MCP server** (96 tools) lets AI agents like Claude drive crawls live and read every result — every action a human can take in the desktop is callable from an agent. Everything runs **fully local** — no telemetry, no cloud, MIT-licensed.
|
||||
|
||||
<br />
|
||||
|
||||
@@ -398,9 +400,13 @@ graph LR
|
||||
## 📈 Status
|
||||
|
||||
> [!NOTE]
|
||||
> **Active development.** All 31 analysis tabs (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP, **PageSpeed, CrUX, Spelling, Search Console, GA4, AI, SEO Authority**) plus standalone Visualization window, advanced search, **per-tab quick-filter dropdown** + **List/Tree view toggle**, 150+ issue categories, sitemap export variants, **Export Crawl Data dialog (XLSX / CSV-UTF-8 / JSON / XML with hierarchical tree picker + nested folder output)**, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, **Cytoscape visualization (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP above-the-fold overlay + internal-PageRank link-score overlay + click-to-trace Crawl Path Report)**, **Basic / Digest / Bearer + form-based login (HTTP multi-step + Playwright browser-driven SPA login)**, **HTTP / SOCKS5 proxy**, webhook, **MCP server with crawl control + live progress**, **Google PageSpeed / Search Console / URL Inspection / GA4 integrations**, **Chrome UX Report (CrUX) real-user field Core Web Vitals**, **LanguageTool spelling & grammar checking (custom dictionary, picky rule level, in-context highlighting)**, **AI per-URL prompts (OpenAI / Anthropic / Ollama)**, **SEO Authority providers (Ahrefs / Majestic / Moz / Semrush)**, **Google Sheets + BigQuery direct export**, **encrypted project snapshots (AES-256-GCM)**, **traffic-ranked cross-source orphan detection (sitemap + GSC + GA4)**, **internal PageRank link scoring**, **JavaScript rendering with Playwright (post-JS DOM, screenshot capture, LCP candidate, Mobile Usability audit)**, **memory-limit auto-pause watchdog**, **RAM-only (in-memory) storage mode**, OS notifications, **robots.txt syntax validator**, URL rewriting + preview, **status-code diagnosis banner**, **live memory monitor**, **in-app scheduled crawl**, **multi-language UI (EN + TR) with full Settings coverage**, **standalone Log File Analyzer (Apache / Nginx / IIS / custom access logs → bot hits per URL, crawl budget, response-code distribution, daily trend, crawl × log orphan detection, 40+ bot detection with reverse-DNS verification, per-bot URL filter, CSV / Excel export, and a headless `analyze-logs` CLI)**, **`.seoproject` file association**, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows `.exe` + portable, macOS `.dmg`, Linux `.AppImage` / `.deb` / `.rpm`) — **release builds ship Playwright Chromium offline so JS rendering works on first launch**. Live-streaming UX with **first row in ~1 s**.
|
||||
> **Active development.** All 31 analysis tabs are working — the core set (Internal, External, Response Codes, URL, Page Titles, Meta Description, H1, H2, Content, Images, Canonicals, Directives, Redirects, Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search, Security, Duplicates, Links, Broken Links, SERP) plus the integration tabs (**PageSpeed, CrUX, Spelling, Search Console, GA4, AI, SEO Authority**).
|
||||
>
|
||||
> **Upcoming (V2):** Plugin system, Light theme, Multi-window, Code-signing + auto-update.
|
||||
> Also shipping: 150+ issue checks, **JavaScript rendering** (Playwright — post-JS DOM, screenshots, LCP candidate, Mobile Usability audit), **multi-window projects** (independent per-window crawls), near/exact-duplicate detection, hreflang validation, **internal PageRank link scoring**, project compare, the **Cytoscape link graph** (force-directed / BFS / radial / Sugiyama / directory-tree layouts + LCP and link-score overlays + click-to-trace Crawl Path Report), **Basic / Digest / Bearer + form-based & browser-driven SPA login**, **HTTP / SOCKS5 proxy**, the **MCP server with live crawl control**, all **Google integrations** (Search Console / URL Inspection / GA4 / PageSpeed / CrUX), **LanguageTool spelling & grammar**, **AI per-URL prompts** (OpenAI / Anthropic / Ollama), **SEO Authority providers** (Ahrefs / Majestic / Moz / Semrush), **Google Sheets + BigQuery export**, **encrypted project snapshots** (AES-256-GCM), **RAM-only storage mode**, scheduled crawls, robots.txt validator, URL rewriting, EN + TR UI, and the **standalone Log File Analyzer** (Apache / Nginx / IIS access logs → bot hits, crawl budget, crawl × log orphans, 40+ bot detection, CSV / Excel export, and a headless `analyze-logs` CLI).
|
||||
>
|
||||
> Cross-platform installers ship for Windows (`.exe` + portable), macOS (`.dmg`), and Linux (`.AppImage` / `.deb` / `.rpm`), with Playwright Chromium bundled so JS rendering works on first launch. The virtualized 1M-row table streams the **first row in ~1 s**.
|
||||
>
|
||||
> **Upcoming:** Plugin system, Light theme, Code-signing + auto-update.
|
||||
|
||||
<br />
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/cli",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/desktop",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"description": "FreeCrawl SEO — desktop application",
|
||||
"homepage": "https://github.com/kemalai/FreeCrawl-SEO-Tool",
|
||||
|
||||
@@ -88,7 +88,7 @@ interface WorkerSlot {
|
||||
restartTimes: number[];
|
||||
}
|
||||
|
||||
class DbReaderPool {
|
||||
export class DbReaderPool {
|
||||
private slots: WorkerSlot[] = [];
|
||||
private dbPath: string | null = null;
|
||||
private freezeWatchdogSab: SharedArrayBuffer | null = null;
|
||||
@@ -304,9 +304,20 @@ class DbReaderPool {
|
||||
}
|
||||
}
|
||||
|
||||
// Process-wide singleton — only one worker pool ever exists.
|
||||
// Process-wide singleton — the primary window's reader pool. Multi-window
|
||||
// gives additional windows their own `DbReaderPool` instances.
|
||||
export const dbReaderPool = new DbReaderPool();
|
||||
|
||||
/** Resolves which reader pool `callReaderOrFallback` should use. Defaults
|
||||
* to the singleton; the desktop host injects a per-window resolver so a
|
||||
* read IPC call hits the calling window's pool. */
|
||||
let resolveReaderPool: () => DbReaderPool = () => dbReaderPool;
|
||||
|
||||
/** Inject a per-call reader-pool resolver (multi-window). */
|
||||
export function setReaderPoolResolver(fn: () => DbReaderPool): void {
|
||||
resolveReaderPool = fn;
|
||||
}
|
||||
|
||||
/**
|
||||
* Thin helper for IPC handlers: try the worker pool first, fall back
|
||||
* to the synchronous main-process DB on any worker error. The fallback
|
||||
@@ -318,11 +329,12 @@ export async function callReaderOrFallback<T>(
|
||||
args: unknown[],
|
||||
fallback: () => T | Promise<T>,
|
||||
): Promise<T> {
|
||||
if (!dbReaderPool.isReady()) {
|
||||
const pool = resolveReaderPool();
|
||||
if (!pool.isReady()) {
|
||||
return fallback();
|
||||
}
|
||||
try {
|
||||
return await dbReaderPool.call<T>(method, args);
|
||||
return await pool.call<T>(method, args);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
logger.log(
|
||||
|
||||
@@ -38,7 +38,7 @@ interface ResponseMessage {
|
||||
error?: string;
|
||||
}
|
||||
|
||||
class DbWriterPool {
|
||||
export class DbWriterPool {
|
||||
private worker: Worker | null = null;
|
||||
private dbPath: string | null = null;
|
||||
private freezeWatchdogSab: SharedArrayBuffer | null = null;
|
||||
@@ -177,8 +177,22 @@ class DbWriterPool {
|
||||
}
|
||||
}
|
||||
|
||||
// Process-wide singleton — the primary window's writer pool. Multi-window
|
||||
// gives additional windows their own `DbWriterPool` instances.
|
||||
export const dbWriterPool = new DbWriterPool();
|
||||
|
||||
/** Resolves which writer pool `callWriterOrFallback` should use. Defaults to
|
||||
* the singleton; the desktop host injects a per-window resolver so a write
|
||||
* hits the calling window's pool — mirrors `setReaderPoolResolver`. Without
|
||||
* this, a per-window write helper would silently target the primary window's
|
||||
* DB, the same multi-window footgun the clear/reset path once had. */
|
||||
let resolveWriterPool: () => DbWriterPool = () => dbWriterPool;
|
||||
|
||||
/** Inject a per-call writer-pool resolver (multi-window). */
|
||||
export function setWriterPoolResolver(fn: () => DbWriterPool): void {
|
||||
resolveWriterPool = fn;
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper: try the worker first, fall back to a synchronous main-thread
|
||||
* call on any worker error. The fallback keeps writes happening even
|
||||
@@ -190,11 +204,12 @@ export async function callWriterOrFallback<T>(
|
||||
args: unknown[],
|
||||
fallback: () => T | Promise<T>,
|
||||
): Promise<T> {
|
||||
if (!dbWriterPool.isReady()) {
|
||||
const pool = resolveWriterPool();
|
||||
if (!pool.isReady()) {
|
||||
return fallback();
|
||||
}
|
||||
try {
|
||||
return await dbWriterPool.call<T>(method, args);
|
||||
return await pool.call<T>(method, args);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
logger.log(
|
||||
|
||||
+1321
-768
File diff suppressed because it is too large.
Load diff
@@ -132,7 +132,13 @@ function writeToDisk(entry: LogEntry): void {
|
||||
}
|
||||
}
|
||||
|
||||
export function log(level: LogLevel, source: string, message: string, extra?: unknown): LogEntry {
|
||||
export function log(
|
||||
level: LogLevel,
|
||||
source: string,
|
||||
message: string,
|
||||
extra?: unknown,
|
||||
windowId?: number,
|
||||
): LogEntry {
|
||||
const msg =
|
||||
extra !== undefined ? `${message} ${safeStringify(extra)}` : message;
|
||||
const entry: LogEntry = {
|
||||
@@ -141,6 +147,7 @@ export function log(level: LogLevel, source: string, message: string, extra?: un
|
||||
level,
|
||||
source,
|
||||
message: msg,
|
||||
...(windowId !== undefined ? { windowId } : {}),
|
||||
};
|
||||
buffer.push(entry);
|
||||
if (buffer.length > MAX_BUFFER_ENTRIES) buffer.shift();
|
||||
|
||||
@@ -18,10 +18,12 @@ export type MenuLang = 'en' | 'tr';
|
||||
export interface MenuLabels {
|
||||
file: string;
|
||||
newProject: string;
|
||||
newProjectWindow: string;
|
||||
openProject: string;
|
||||
openRecent: string;
|
||||
clearRecent: string;
|
||||
emptyRecent: string;
|
||||
manageProjects: string;
|
||||
clearCrawlData: string;
|
||||
exportAs: string;
|
||||
generateSitemap: string;
|
||||
@@ -138,7 +140,9 @@ const MENU_EN: MenuLabels = {
|
||||
file: 'File',
|
||||
newProject: 'New Project',
|
||||
openProject: 'Open Project…',
|
||||
newProjectWindow: 'New Project Window',
|
||||
openRecent: 'Open Recent',
|
||||
manageProjects: 'Manage Projects…',
|
||||
clearRecent: 'Clear Recent',
|
||||
emptyRecent: '(empty)',
|
||||
clearCrawlData: 'Clear Crawl Data',
|
||||
@@ -264,7 +268,9 @@ const MENU_TR: MenuLabels = {
|
||||
file: 'Dosya',
|
||||
newProject: 'Yeni Proje',
|
||||
openProject: 'Proje Aç…',
|
||||
newProjectWindow: 'Yeni Proje Penceresi',
|
||||
openRecent: 'Son Açılanlar',
|
||||
manageProjects: 'Projeleri Yönet…',
|
||||
clearRecent: 'Son Açılanları Temizle',
|
||||
emptyRecent: '(boş)',
|
||||
clearCrawlData: 'Crawl Verilerini Temizle',
|
||||
|
||||
@@ -10,6 +10,8 @@ function send(event: MenuEvent): void {
|
||||
|
||||
export interface AppMenuHandlers {
|
||||
onOpenLogs: () => void;
|
||||
/** Open a second top-level window driving its own independent project. */
|
||||
onNewProjectWindow: () => void;
|
||||
/** Open an existing `.seoproject` file via dialog. */
|
||||
onOpenProject: () => void;
|
||||
/** Open a recent project by its absolute path. */
|
||||
@@ -72,12 +74,18 @@ export function buildAppMenu(handlers: AppMenuHandlers): Menu {
|
||||
label: L.file,
|
||||
submenu: [
|
||||
{ label: L.newProject, accelerator: 'CmdOrCtrl+N', click: () => send('new-project') },
|
||||
{
|
||||
label: L.newProjectWindow,
|
||||
accelerator: 'CmdOrCtrl+Shift+N',
|
||||
click: () => handlers.onNewProjectWindow(),
|
||||
},
|
||||
{
|
||||
label: L.openProject,
|
||||
accelerator: 'CmdOrCtrl+O',
|
||||
click: () => handlers.onOpenProject(),
|
||||
},
|
||||
{ label: L.openRecent, submenu: recentSubmenu },
|
||||
{ label: L.manageProjects, click: () => send('manage-projects') },
|
||||
{ label: L.clearCrawlData, click: () => send('clear-crawl') },
|
||||
{ type: 'separator' },
|
||||
{
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
IPC,
|
||||
type ConfirmClearResult,
|
||||
type CrawlConfig,
|
||||
type RecentProject,
|
||||
type CrawlProgress,
|
||||
type CrawlSummary,
|
||||
type ExportCsvInput,
|
||||
@@ -184,6 +185,20 @@ const api: FreeCrawlApi = {
|
||||
ipcRenderer.invoke(IPC.projectOpen, filePath),
|
||||
projectCurrentPath: (): Promise<string | null> =>
|
||||
ipcRenderer.invoke(IPC.projectCurrentPath),
|
||||
projectConfigGet: (): Promise<CrawlConfig | null> =>
|
||||
ipcRenderer.invoke(IPC.projectConfigGet),
|
||||
projectConfigSet: (config: CrawlConfig): Promise<void> =>
|
||||
ipcRenderer.invoke(IPC.projectConfigSet, config),
|
||||
onProjectConfigChanged: (cb: (config: CrawlConfig | null) => void) =>
|
||||
subscribe<CrawlConfig | null>(IPC.projectConfigChanged, cb),
|
||||
recentProjectsList: (): Promise<RecentProject[]> =>
|
||||
ipcRenderer.invoke(IPC.recentProjectsList),
|
||||
recentProjectSetArchived: (path: string, archived: boolean): Promise<void> =>
|
||||
ipcRenderer.invoke(IPC.recentProjectSetArchived, path, archived),
|
||||
recentProjectSetTags: (path: string, tags: string[]): Promise<void> =>
|
||||
ipcRenderer.invoke(IPC.recentProjectSetTags, path, tags),
|
||||
recentProjectRemove: (path: string): Promise<void> =>
|
||||
ipcRenderer.invoke(IPC.recentProjectRemove, path),
|
||||
projectSaveEncrypted: (
|
||||
password: string,
|
||||
): Promise<{ filePath: string; bytesWritten: number } | { error: string } | null> =>
|
||||
@@ -262,7 +277,8 @@ const api: FreeCrawlApi = {
|
||||
void ipcRenderer.invoke(IPC.prefsDelete, key);
|
||||
},
|
||||
confirmClear: (): Promise<ConfirmClearResult> => ipcRenderer.invoke(IPC.confirmClear),
|
||||
logsGetAll: (): Promise<LogEntry[]> => ipcRenderer.invoke(IPC.logsGetAll),
|
||||
logsGetAll: (ownerId?: number): Promise<LogEntry[]> =>
|
||||
ipcRenderer.invoke(IPC.logsGetAll, ownerId),
|
||||
logsClear: (): Promise<void> => ipcRenderer.invoke(IPC.logsClear),
|
||||
logsOpenWindow: (): Promise<void> => ipcRenderer.invoke(IPC.logsOpenWindow),
|
||||
openVisualizationWindow: (): Promise<void> =>
|
||||
|
||||
@@ -12,6 +12,7 @@ import { ReportsDialog } from './components/ReportsDialog.js';
|
||||
import { SettingsDialog } from './components/SettingsDialog.js';
|
||||
import { CompareDialog } from './components/CompareDialog.js';
|
||||
import { ScheduledCrawlDialog } from './components/ScheduledCrawlDialog.js';
|
||||
import { ProjectsDialog } from './components/ProjectsDialog.js';
|
||||
import { PasswordPromptDialog } from './components/PasswordPromptDialog.js';
|
||||
import { ExportDialog } from './components/ExportDialog.js';
|
||||
import { UrlsTab } from './tabs/UrlsTab.js';
|
||||
@@ -44,6 +45,7 @@ export function App() {
|
||||
const settingsOpen = useAppStore((s) => s.settingsOpen);
|
||||
const setSettingsOpen = useAppStore((s) => s.setSettingsOpen);
|
||||
const setConfig = useAppStore((s) => s.setConfig);
|
||||
const applyProjectConfig = useAppStore((s) => s.applyProjectConfig);
|
||||
const setActiveTab = useAppStore((s) => s.setActiveTab);
|
||||
const [robotsTesterOpen, setRobotsTesterOpen] = useState(false);
|
||||
const [sitemapValidatorOpen, setSitemapValidatorOpen] = useState(false);
|
||||
@@ -51,6 +53,7 @@ export function App() {
|
||||
const [reportsOpen, setReportsOpen] = useState(false);
|
||||
const [compareOpen, setCompareOpen] = useState(false);
|
||||
const [scheduleOpen, setScheduleOpen] = useState(false);
|
||||
const [projectsOpen, setProjectsOpen] = useState(false);
|
||||
const [dropFlash, setDropFlash] = useState<string | null>(null);
|
||||
const [passwordPrompt, setPasswordPrompt] = useState<
|
||||
| {
|
||||
@@ -218,6 +221,11 @@ export function App() {
|
||||
const off2 = window.freecrawl.onDone((s) => setSummary(s));
|
||||
const off3 = window.freecrawl.onError(setError);
|
||||
const offData = window.freecrawl.onDataChanged(() => bumpDataVersion());
|
||||
// Per-project settings — when a project opens, swap the UI to its saved
|
||||
// crawl config (null = project has none yet, keep current config).
|
||||
const offConfig = window.freecrawl.onProjectConfigChanged((cfg) => {
|
||||
if (cfg) applyProjectConfig(cfg);
|
||||
});
|
||||
const off4 = window.freecrawl.onMenuEvent((event: MenuEvent) => {
|
||||
switch (event) {
|
||||
case 'toggle-sidebar':
|
||||
@@ -364,6 +372,9 @@ export function App() {
|
||||
case 'compare-with-project':
|
||||
setCompareOpen(true);
|
||||
break;
|
||||
case 'manage-projects':
|
||||
setProjectsOpen(true);
|
||||
break;
|
||||
case 'save-project-as':
|
||||
void window.freecrawl.projectSaveAs();
|
||||
break;
|
||||
@@ -391,6 +402,7 @@ export function App() {
|
||||
off3();
|
||||
off4();
|
||||
offData();
|
||||
offConfig();
|
||||
};
|
||||
}, [
|
||||
t,
|
||||
@@ -403,6 +415,7 @@ export function App() {
|
||||
bumpDataVersion,
|
||||
setSettingsOpen,
|
||||
setActiveTab,
|
||||
applyProjectConfig,
|
||||
]);
|
||||
|
||||
return (
|
||||
@@ -490,6 +503,19 @@ export function App() {
|
||||
<SettingsDialog open={settingsOpen} onClose={() => setSettingsOpen(false)} />
|
||||
<CompareDialog open={compareOpen} onClose={() => setCompareOpen(false)} />
|
||||
<ScheduledCrawlDialog open={scheduleOpen} onClose={() => setScheduleOpen(false)} />
|
||||
<ProjectsDialog
|
||||
open={projectsOpen}
|
||||
onClose={() => setProjectsOpen(false)}
|
||||
onNewFromTemplate={(overrides) => {
|
||||
setProjectsOpen(false);
|
||||
void clearCrawlWithConfirm().then((didClear) => {
|
||||
if (didClear) {
|
||||
reset();
|
||||
setConfig(overrides);
|
||||
}
|
||||
});
|
||||
}}
|
||||
/>
|
||||
<ExportDialog
|
||||
open={exportAsOpen}
|
||||
onClose={() => setExportAsOpen(false)}
|
||||
|
||||
@@ -106,11 +106,19 @@ export function LogsView() {
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
void window.freecrawl.logsGetAll().then((rows) => {
|
||||
if (!cancelled) {
|
||||
setEntries(rows.length > MAX_ENTRIES ? rows.slice(rows.length - MAX_ENTRIES) : rows);
|
||||
}
|
||||
});
|
||||
// This Logs popup is scoped to the project window that opened it
|
||||
// (?owner=<webContents.id>). The main process filters both the initial
|
||||
// snapshot and the live batches to that window's crawler entries plus
|
||||
// app-global infra logs, so two windows' crawls never mix.
|
||||
const ownerParam = new URLSearchParams(window.location.search).get('owner');
|
||||
const ownerId = ownerParam !== null ? Number(ownerParam) : undefined;
|
||||
void window.freecrawl
|
||||
.logsGetAll(Number.isFinite(ownerId) ? ownerId : undefined)
|
||||
.then((rows) => {
|
||||
if (!cancelled) {
|
||||
setEntries(rows.length > MAX_ENTRIES ? rows.slice(rows.length - MAX_ENTRIES) : rows);
|
||||
}
|
||||
});
|
||||
|
||||
const flush = (): void => {
|
||||
if (isBusyRef.current) return; // window is being dragged/resized — defer
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from 'react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { X, FolderOpen, Archive, ArchiveRestore, Trash2, Plus } from 'lucide-react';
|
||||
import type { CrawlConfig, RecentProject } from '@freecrawl/shared-types';
|
||||
import { PROJECT_TEMPLATES } from '../projectTemplates.js';
|
||||
|
||||
function basename(p: string): string {
|
||||
const parts = p.split(/[\\/]/);
|
||||
return parts[parts.length - 1] || p;
|
||||
}
|
||||
|
||||
export function ProjectsDialog({
|
||||
open,
|
||||
onClose,
|
||||
onNewFromTemplate,
|
||||
}: {
|
||||
open: boolean;
|
||||
onClose: () => void;
|
||||
onNewFromTemplate: (overrides: Partial<CrawlConfig>) => void;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
const [recents, setRecents] = useState<RecentProject[]>([]);
|
||||
const [search, setSearch] = useState('');
|
||||
const [showArchived, setShowArchived] = useState(false);
|
||||
const [tagFilter, setTagFilter] = useState<string | null>(null);
|
||||
const [editingTags, setEditingTags] = useState<string | null>(null);
|
||||
const [tagDraft, setTagDraft] = useState('');
|
||||
|
||||
const reload = useCallback(async () => {
|
||||
setRecents(await window.freecrawl.recentProjectsList());
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (open) void reload();
|
||||
}, [open, reload]);
|
||||
|
||||
const allTags = useMemo(() => {
|
||||
const s = new Set<string>();
|
||||
for (const r of recents) for (const tag of r.tags ?? []) s.add(tag);
|
||||
return [...s].sort();
|
||||
}, [recents]);
|
||||
|
||||
const filtered = useMemo(() => {
|
||||
const q = search.trim().toLowerCase();
|
||||
return recents.filter((r) => {
|
||||
if (!showArchived && r.archived) return false;
|
||||
if (q && !r.path.toLowerCase().includes(q)) return false;
|
||||
if (tagFilter && !(r.tags ?? []).includes(tagFilter)) return false;
|
||||
return true;
|
||||
});
|
||||
}, [recents, search, showArchived, tagFilter]);
|
||||
|
||||
const openProject = useCallback(
|
||||
(path: string) => {
|
||||
void window.freecrawl.projectOpen(path);
|
||||
onClose();
|
||||
},
|
||||
[onClose],
|
||||
);
|
||||
|
||||
const toggleArchive = useCallback(
|
||||
async (r: RecentProject) => {
|
||||
await window.freecrawl.recentProjectSetArchived(r.path, !r.archived);
|
||||
void reload();
|
||||
},
|
||||
[reload],
|
||||
);
|
||||
|
||||
const remove = useCallback(
|
||||
async (path: string) => {
|
||||
await window.freecrawl.recentProjectRemove(path);
|
||||
void reload();
|
||||
},
|
||||
[reload],
|
||||
);
|
||||
|
||||
const saveTags = useCallback(
|
||||
async (path: string) => {
|
||||
const tags = tagDraft
|
||||
.split(/[,\n]/)
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean);
|
||||
await window.freecrawl.recentProjectSetTags(path, tags);
|
||||
setEditingTags(null);
|
||||
setTagDraft('');
|
||||
void reload();
|
||||
},
|
||||
[tagDraft, reload],
|
||||
);
|
||||
|
||||
if (!open) return null;
|
||||
|
||||
return (
|
||||
<div
|
||||
className="fixed inset-0 z-50 flex items-center justify-center bg-black/60"
|
||||
onClick={onClose}
|
||||
>
|
||||
<div
|
||||
className="flex max-h-[85vh] w-[720px] flex-col rounded-lg border border-surface-700 bg-surface-900 shadow-2xl"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
{/* Header */}
|
||||
<div className="flex items-center justify-between border-b border-surface-800 px-4 py-2.5">
|
||||
<h2 className="text-[13px] font-semibold text-surface-100">
|
||||
{t('projects.title', { defaultValue: 'Projects' })}
|
||||
</h2>
|
||||
<button
|
||||
onClick={onClose}
|
||||
className="rounded p-1 text-surface-400 hover:bg-surface-800 hover:text-surface-100"
|
||||
>
|
||||
<X size={16} />
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="flex-1 overflow-y-auto p-4">
|
||||
{/* New from template */}
|
||||
<div className="mb-5">
|
||||
<div className="mb-2 text-[10px] font-semibold uppercase tracking-wider text-surface-400">
|
||||
{t('projects.newFromTemplate', { defaultValue: 'Start a New Project' })}
|
||||
</div>
|
||||
<div className="grid grid-cols-2 gap-2 sm:grid-cols-3">
|
||||
{PROJECT_TEMPLATES.map((tmpl) => (
|
||||
<button
|
||||
key={tmpl.key}
|
||||
onClick={() => onNewFromTemplate(tmpl.overrides)}
|
||||
className="group flex flex-col rounded border border-surface-700 bg-surface-950/60 p-2.5 text-left hover:border-blue-600 hover:bg-surface-800/60"
|
||||
title={tmpl.description}
|
||||
>
|
||||
<span className="mb-0.5 flex items-center gap-1 text-[12px] font-medium text-surface-100">
|
||||
<Plus size={11} className="text-blue-400" />
|
||||
{t(`projects.template.${tmpl.key}`, { defaultValue: tmpl.label })}
|
||||
</span>
|
||||
<span className="line-clamp-2 text-[10px] leading-snug text-surface-500">
|
||||
{t(`projects.templateDesc.${tmpl.key}`, {
|
||||
defaultValue: tmpl.description,
|
||||
})}
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Recent projects */}
|
||||
<div className="mb-2 flex items-center justify-between">
|
||||
<div className="text-[10px] font-semibold uppercase tracking-wider text-surface-400">
|
||||
{t('projects.recent', { defaultValue: 'Recent Projects' })}
|
||||
</div>
|
||||
<label className="flex items-center gap-1.5 text-[11px] text-surface-400">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={showArchived}
|
||||
onChange={(e) => setShowArchived(e.target.checked)}
|
||||
className="h-3 w-3 accent-blue-500"
|
||||
/>
|
||||
{t('projects.showArchived', { defaultValue: 'Show archived' })}
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="mb-2 flex flex-wrap items-center gap-2">
|
||||
<input
|
||||
type="text"
|
||||
value={search}
|
||||
onChange={(e) => setSearch(e.target.value)}
|
||||
placeholder={t('projects.filterPlaceholder', {
|
||||
defaultValue: 'Filter by path…',
|
||||
})}
|
||||
className="h-6 flex-1 rounded border border-surface-700 bg-surface-950 px-2 text-[11px] text-surface-100 placeholder-surface-500 focus:border-blue-500 focus:outline-none"
|
||||
/>
|
||||
{allTags.length > 0 && (
|
||||
<div className="flex flex-wrap gap-1">
|
||||
{allTags.map((tag) => (
|
||||
<button
|
||||
key={tag}
|
||||
onClick={() => setTagFilter(tagFilter === tag ? null : tag)}
|
||||
className={`rounded-full px-2 py-0.5 text-[10px] ${
|
||||
tagFilter === tag
|
||||
? 'bg-blue-600 text-white'
|
||||
: 'bg-surface-800 text-surface-300 hover:bg-surface-700'
|
||||
}`}
|
||||
>
|
||||
{tag}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{filtered.length === 0 ? (
|
||||
<div className="rounded border border-surface-800 bg-surface-950/40 px-3 py-6 text-center text-[11px] text-surface-500">
|
||||
{recents.length === 0
|
||||
? t('projects.emptyNone', {
|
||||
defaultValue: 'No recent projects yet — Save As creates one.',
|
||||
})
|
||||
: t('projects.emptyFilter', {
|
||||
defaultValue: 'No projects match the current filter.',
|
||||
})}
|
||||
</div>
|
||||
) : (
|
||||
<ul className="flex flex-col gap-1">
|
||||
{filtered.map((r) => (
|
||||
<li
|
||||
key={r.path}
|
||||
className={`rounded border border-surface-800 bg-surface-950/40 px-2.5 py-1.5 ${
|
||||
r.archived ? 'opacity-60' : ''
|
||||
}`}
|
||||
>
|
||||
<div className="flex items-center gap-2">
|
||||
<div
|
||||
className="min-w-0 flex-1 cursor-pointer"
|
||||
onDoubleClick={() => openProject(r.path)}
|
||||
title={r.path}
|
||||
>
|
||||
<div className="truncate text-[12px] text-surface-100">
|
||||
{basename(r.path)}
|
||||
{r.archived && (
|
||||
<span className="ml-1.5 text-[9px] uppercase text-surface-500">
|
||||
{t('projects.archivedTag', { defaultValue: 'archived' })}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="truncate text-[10px] text-surface-500">{r.path}</div>
|
||||
</div>
|
||||
<button
|
||||
onClick={() => openProject(r.path)}
|
||||
title={t('projects.open', { defaultValue: 'Open' })}
|
||||
className="rounded p-1 text-surface-400 hover:bg-surface-800 hover:text-blue-300"
|
||||
>
|
||||
<FolderOpen size={14} />
|
||||
</button>
|
||||
<button
|
||||
onClick={() => void toggleArchive(r)}
|
||||
title={
|
||||
r.archived
|
||||
? t('projects.unarchive', { defaultValue: 'Unarchive' })
|
||||
: t('projects.archive', { defaultValue: 'Archive' })
|
||||
}
|
||||
className="rounded p-1 text-surface-400 hover:bg-surface-800 hover:text-amber-300"
|
||||
>
|
||||
{r.archived ? <ArchiveRestore size={14} /> : <Archive size={14} />}
|
||||
</button>
|
||||
<button
|
||||
onClick={() => void remove(r.path)}
|
||||
title={t('projects.remove', { defaultValue: 'Remove from list' })}
|
||||
className="rounded p-1 text-surface-400 hover:bg-surface-800 hover:text-red-300"
|
||||
>
|
||||
<Trash2 size={14} />
|
||||
</button>
|
||||
</div>
|
||||
{/* Tags */}
|
||||
<div className="mt-1 flex flex-wrap items-center gap-1">
|
||||
{(r.tags ?? []).map((tag) => (
|
||||
<span
|
||||
key={tag}
|
||||
className="rounded-full bg-surface-800 px-1.5 py-px text-[9px] text-surface-300"
|
||||
>
|
||||
{tag}
|
||||
</span>
|
||||
))}
|
||||
{editingTags === r.path ? (
|
||||
<input
|
||||
autoFocus
|
||||
value={tagDraft}
|
||||
onChange={(e) => setTagDraft(e.target.value)}
|
||||
onBlur={() => void saveTags(r.path)}
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === 'Enter') void saveTags(r.path);
|
||||
if (e.key === 'Escape') {
|
||||
setEditingTags(null);
|
||||
setTagDraft('');
|
||||
}
|
||||
}}
|
||||
placeholder={t('projects.tagsPlaceholder', {
|
||||
defaultValue: 'tag1, tag2',
|
||||
})}
|
||||
className="h-5 w-40 rounded border border-surface-700 bg-surface-950 px-1.5 text-[10px] text-surface-100 focus:border-blue-500 focus:outline-none"
|
||||
/>
|
||||
) : (
|
||||
<button
|
||||
onClick={() => {
|
||||
setEditingTags(r.path);
|
||||
setTagDraft((r.tags ?? []).join(', '));
|
||||
}}
|
||||
className="rounded-full border border-dashed border-surface-700 px-1.5 py-px text-[9px] text-surface-500 hover:border-blue-600 hover:text-blue-300"
|
||||
>
|
||||
{t('projects.editTags', { defaultValue: '+ tags' })}
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1086,6 +1086,38 @@
|
||||
"emptyNoData": "No internal HTML pages crawled yet — run a crawl, then select pages to check.",
|
||||
"emptyNoMatch": "No pages match the current filter."
|
||||
},
|
||||
"projects": {
|
||||
"title": "Projects",
|
||||
"newFromTemplate": "Start a New Project",
|
||||
"recent": "Recent Projects",
|
||||
"showArchived": "Show archived",
|
||||
"filterPlaceholder": "Filter by path…",
|
||||
"emptyNone": "No recent projects yet — Save As creates one.",
|
||||
"emptyFilter": "No projects match the current filter.",
|
||||
"open": "Open",
|
||||
"archive": "Archive",
|
||||
"unarchive": "Unarchive",
|
||||
"remove": "Remove from list",
|
||||
"archivedTag": "archived",
|
||||
"editTags": "+ tags",
|
||||
"tagsPlaceholder": "tag1, tag2",
|
||||
"template": {
|
||||
"blank": "Blank",
|
||||
"ecommerce": "E-commerce",
|
||||
"blog": "Blog / Content",
|
||||
"corporate": "Corporate site",
|
||||
"news": "News / Publisher",
|
||||
"saas": "SaaS / Web app"
|
||||
},
|
||||
"templateDesc": {
|
||||
"blank": "Your current default settings — a clean crawl with nothing pre-filled.",
|
||||
"ecommerce": "Skips cart, checkout, account and faceted-filter URLs; larger URL budget for big catalogs.",
|
||||
"blog": "Skips author, tag and date-archive noise so the crawl focuses on articles.",
|
||||
"corporate": "Balanced defaults for a marketing / brochure site.",
|
||||
"news": "Large URL budget; skips print variants and comment-page pagination.",
|
||||
"saas": "Skips authenticated app areas (login, signup, dashboard, settings)."
|
||||
}
|
||||
},
|
||||
"spelling": {
|
||||
"loading": "Loading…",
|
||||
"notChecked": "This page has not been checked yet. Select it in the Spelling tab and run a check.",
|
||||
|
||||
@@ -1086,6 +1086,38 @@
|
||||
"emptyNoData": "Henüz taranmış iç HTML sayfası yok — önce bir crawl çalıştır, sonra denetlenecek sayfaları seç.",
|
||||
"emptyNoMatch": "Mevcut filtreyle eşleşen sayfa yok."
|
||||
},
|
||||
"projects": {
|
||||
"title": "Projeler",
|
||||
"newFromTemplate": "Yeni Proje Başlat",
|
||||
"recent": "Son Projeler",
|
||||
"showArchived": "Arşivlenenleri göster",
|
||||
"filterPlaceholder": "Yola göre filtrele…",
|
||||
"emptyNone": "Henüz son proje yok — Farklı Kaydet bir tane oluşturur.",
|
||||
"emptyFilter": "Mevcut filtreyle eşleşen proje yok.",
|
||||
"open": "Aç",
|
||||
"archive": "Arşivle",
|
||||
"unarchive": "Arşivden çıkar",
|
||||
"remove": "Listeden kaldır",
|
||||
"archivedTag": "arşivli",
|
||||
"editTags": "+ etiket",
|
||||
"tagsPlaceholder": "etiket1, etiket2",
|
||||
"template": {
|
||||
"blank": "Boş",
|
||||
"ecommerce": "E-ticaret",
|
||||
"blog": "Blog / İçerik",
|
||||
"corporate": "Kurumsal site",
|
||||
"news": "Haber / Yayıncı",
|
||||
"saas": "SaaS / Web uygulaması"
|
||||
},
|
||||
"templateDesc": {
|
||||
"blank": "Mevcut varsayılan ayarların — hiçbir şey doldurulmamış temiz bir crawl.",
|
||||
"ecommerce": "Sepet, ödeme, hesap ve filtre URL'lerini atlar; büyük kataloglar için daha yüksek URL bütçesi.",
|
||||
"blog": "Yazar, etiket ve tarih-arşiv gürültüsünü atlar; crawl makalelere odaklanır.",
|
||||
"corporate": "Pazarlama / tanıtım sitesi için dengeli varsayılanlar.",
|
||||
"news": "Büyük URL bütçesi; yazdırma varyantlarını ve yorum-sayfası sayfalamasını atlar.",
|
||||
"saas": "Kimlik doğrulamalı uygulama alanlarını atlar (giriş, kayıt, panel, ayarlar)."
|
||||
}
|
||||
},
|
||||
"spelling": {
|
||||
"loading": "Yükleniyor…",
|
||||
"notChecked": "Bu sayfa henüz denetlenmedi. Yazım sekmesinden seçip denetim çalıştırın.",
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
import type { CrawlConfig } from '@freecrawl/shared-types';
|
||||
|
||||
/**
|
||||
* A project template seeds a fresh crawl with sensible defaults for a site
|
||||
* type — mainly a set of starter exclude patterns (regex) plus URL/depth
|
||||
* budgets. Applying one clears the current scratch project and merges the
|
||||
* overrides over the user's current config.
|
||||
*/
|
||||
export interface ProjectTemplate {
|
||||
key: string;
|
||||
label: string;
|
||||
description: string;
|
||||
/** Merged over the current config via `setConfig` when applied. */
|
||||
overrides: Partial<CrawlConfig>;
|
||||
}
|
||||
|
||||
export const PROJECT_TEMPLATES: ProjectTemplate[] = [
|
||||
{
|
||||
key: 'blank',
|
||||
label: 'Blank',
|
||||
description: 'Your current default settings — a clean crawl with nothing pre-filled.',
|
||||
overrides: {},
|
||||
},
|
||||
{
|
||||
key: 'ecommerce',
|
||||
label: 'E-commerce',
|
||||
description:
|
||||
'Skips cart, checkout, account and faceted-filter URLs; larger URL budget for big catalogs.',
|
||||
overrides: {
|
||||
maxUrls: 200_000,
|
||||
excludePatterns: [
|
||||
'/cart',
|
||||
'/checkout',
|
||||
'/account',
|
||||
'/wishlist',
|
||||
'/compare',
|
||||
'[?&](add-to-cart|orderby|filter|price|color|size)=',
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
key: 'blog',
|
||||
label: 'Blog / Content',
|
||||
description: 'Skips author, tag and date-archive noise so the crawl focuses on articles.',
|
||||
overrides: {
|
||||
excludePatterns: ['/tag/', '/author/', '/category/page/', '/\\d{4}/\\d{2}/'],
|
||||
},
|
||||
},
|
||||
{
|
||||
key: 'corporate',
|
||||
label: 'Corporate site',
|
||||
description: 'Balanced defaults for a marketing / brochure site.',
|
||||
overrides: {
|
||||
maxDepth: 10,
|
||||
},
|
||||
},
|
||||
{
|
||||
key: 'news',
|
||||
label: 'News / Publisher',
|
||||
description: 'Large URL budget; skips print variants and comment-page pagination.',
|
||||
overrides: {
|
||||
maxUrls: 500_000,
|
||||
excludePatterns: ['/print/', '\\?replytocom=', '/comment-page-', '/amp/'],
|
||||
},
|
||||
},
|
||||
{
|
||||
key: 'saas',
|
||||
label: 'SaaS / Web app',
|
||||
description: 'Skips authenticated app areas (login, signup, dashboard, settings).',
|
||||
overrides: {
|
||||
excludePatterns: [
|
||||
'/login',
|
||||
'/signup',
|
||||
'/sign-in',
|
||||
'/dashboard',
|
||||
'/app/',
|
||||
'/settings',
|
||||
'/account',
|
||||
],
|
||||
},
|
||||
},
|
||||
];
|
||||
@@ -344,6 +344,9 @@ interface AppState {
|
||||
recentUrls: string[];
|
||||
dataVersion: number;
|
||||
setConfig: (patch: Partial<CrawlConfig>) => void;
|
||||
/** Replace the whole config from a project's saved settings WITHOUT
|
||||
* writing it back to the global default prefs (per-project settings). */
|
||||
applyProjectConfig: (config: CrawlConfig) => void;
|
||||
setProgress: (p: CrawlProgress) => void;
|
||||
setSummary: (s: CrawlSummary) => void;
|
||||
setOverview: (o: OverviewCounts) => void;
|
||||
@@ -388,6 +391,47 @@ function loadInitialConfig(): CrawlConfig {
|
||||
return { ...DEFAULT_CRAWL_CONFIG, ...(saved as Partial<CrawlConfig>), startUrl: '' };
|
||||
}
|
||||
|
||||
/**
|
||||
* Debounced persistence for config edits. `setConfig` updates the in-memory
|
||||
* Zustand state synchronously so the UI stays instant, but the two persistence
|
||||
* sinks — the global prefs JSON and the per-project SQLite `setMeta` write —
|
||||
* are coalesced here. Without this, typing in the URL/scope fields (TopBar
|
||||
* fires `setConfig` per keystroke) triggered one IPC round-trip + one
|
||||
* synchronous main-thread SQLite write per character. A `beforeunload` flush
|
||||
* guarantees the final value is persisted even if the app closes mid-debounce.
|
||||
*/
|
||||
const CONFIG_PERSIST_DEBOUNCE_MS = 350;
|
||||
let configPersistTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let pendingConfig: CrawlConfig | null = null;
|
||||
|
||||
function flushConfigPersist(): void {
|
||||
if (configPersistTimer !== null) {
|
||||
clearTimeout(configPersistTimer);
|
||||
configPersistTimer = null;
|
||||
}
|
||||
const cfg = pendingConfig;
|
||||
pendingConfig = null;
|
||||
if (!cfg) return;
|
||||
try {
|
||||
window.freecrawl?.prefsSet('crawl-config', cfg);
|
||||
// No-ops in main when on the default scratch DB, so global prefs stay
|
||||
// the source of truth there.
|
||||
void window.freecrawl?.projectConfigSet(cfg);
|
||||
} catch {
|
||||
// best-effort persistence
|
||||
}
|
||||
}
|
||||
|
||||
function persistConfigDebounced(config: CrawlConfig): void {
|
||||
pendingConfig = config;
|
||||
if (configPersistTimer !== null) clearTimeout(configPersistTimer);
|
||||
configPersistTimer = setTimeout(flushConfigPersist, CONFIG_PERSIST_DEBOUNCE_MS);
|
||||
}
|
||||
|
||||
if (typeof window !== 'undefined') {
|
||||
window.addEventListener('beforeunload', flushConfigPersist);
|
||||
}
|
||||
|
||||
export const useAppStore = create<AppState>((set) => ({
|
||||
config: loadInitialConfig(),
|
||||
progress: null,
|
||||
@@ -406,16 +450,18 @@ export const useAppStore = create<AppState>((set) => ({
|
||||
setConfig: (patch) =>
|
||||
set((state) => {
|
||||
const next = { ...state.config, ...patch };
|
||||
// Persist every config edit so the next launch starts from the
|
||||
// user's last settings — even fields the Settings dialog doesn't
|
||||
// expose (e.g. live URL / scope edits in the top bar).
|
||||
try {
|
||||
window.freecrawl?.prefsSet('crawl-config', next);
|
||||
} catch {
|
||||
// best-effort persistence
|
||||
}
|
||||
// Persist every config edit so the next launch starts from the user's
|
||||
// last settings — even fields the Settings dialog doesn't expose (e.g.
|
||||
// live URL / scope edits in the top bar). Debounced so per-keystroke
|
||||
// edits don't each fire an IPC round-trip + synchronous SQLite write.
|
||||
persistConfigDebounced(next);
|
||||
return { config: next };
|
||||
}),
|
||||
applyProjectConfig: (config) =>
|
||||
// Swap the UI to a project's saved config on open. Deliberately does
|
||||
// NOT touch global prefs — opening a project must not overwrite the
|
||||
// user's default settings.
|
||||
set({ config: { ...config, startUrl: '' } }),
|
||||
setProgress: (p) => set({ progress: p }),
|
||||
setSummary: (s) => set({ summary: s }),
|
||||
setOverview: (o) => set({ overview: o }),
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/mcp-server",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "freecrawl-seo",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"description": "Open-source desktop SEO crawler — a free alternative to Screaming Frog",
|
||||
"license": "MIT",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/core",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/db",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/shared-types",
|
||||
"version": "0.8.5",
|
||||
"version": "0.8.7",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -25,6 +25,7 @@ import type {
|
||||
} from './pagespeed.js';
|
||||
import type { CruxRow, CruxRunFormFactor } from './crux.js';
|
||||
import type { SpellingMatch, SpellingRow } from './spelling.js';
|
||||
import type { RecentProject } from './project.js';
|
||||
import type {
|
||||
GoogleAuthState,
|
||||
GoogleAuthResult,
|
||||
@@ -65,6 +66,21 @@ export const IPC = {
|
||||
projectSaveAs: 'project:save-as',
|
||||
projectOpen: 'project:open',
|
||||
projectCurrentPath: 'project:current-path',
|
||||
/** Per-project crawl config. `projectConfigGet` reads the active
|
||||
* project's saved `CrawlConfig` (null on the default scratch DB, where
|
||||
* global prefs apply); `projectConfigSet` persists edits to the active
|
||||
* project's `project_meta.lastCrawlConfig`; `projectConfigChanged` is
|
||||
* pushed main→renderer when a project opens so the UI rehydrates. */
|
||||
projectConfigGet: 'project:config-get',
|
||||
projectConfigSet: 'project:config-set',
|
||||
projectConfigChanged: 'project:config-changed',
|
||||
/** Recent-projects management (archiving / tagging). `recentProjectsList`
|
||||
* returns the full list incl. archived; the mutators toggle the archived
|
||||
* flag, replace tags, or drop an entry. */
|
||||
recentProjectsList: 'project:recent-list',
|
||||
recentProjectSetArchived: 'project:recent-set-archived',
|
||||
recentProjectSetTags: 'project:recent-set-tags',
|
||||
recentProjectRemove: 'project:recent-remove',
|
||||
/** V1 #4 — Save an AES-256-GCM-encrypted snapshot of the active
|
||||
* `.seoproject` file. Renderer provides the password; main shows a
|
||||
* save dialog for the destination `.seoproject.enc` path. */
|
||||
@@ -429,6 +445,7 @@ export interface ExportCsvInput {
|
||||
|
||||
export type MenuEvent =
|
||||
| 'new-project'
|
||||
| 'manage-projects'
|
||||
| 'clear-crawl'
|
||||
| 'toggle-sidebar'
|
||||
| 'toggle-detail-panel'
|
||||
@@ -1513,6 +1530,11 @@ export interface LogEntry {
|
||||
/** Originating subsystem: 'main', 'crawler', 'ipc', 'console', 'uncaught', 'renderer', 'fetch'. */
|
||||
source: string;
|
||||
message: string;
|
||||
/** Multi-window: `webContents.id` of the project window this entry belongs
|
||||
* to (crawler activity is tagged per-window). Undefined for app-global
|
||||
* infrastructure logs (main / mcp-bridge / console) that are relevant to
|
||||
* every window's Logs view. */
|
||||
windowId?: number;
|
||||
}
|
||||
|
||||
export interface FreeCrawlApi {
|
||||
@@ -1525,6 +1547,22 @@ export interface FreeCrawlApi {
|
||||
projectSaveAs(): Promise<{ filePath: string; bytesWritten: number } | null>;
|
||||
projectOpen(filePath?: string): Promise<{ filePath: string } | null>;
|
||||
projectCurrentPath(): Promise<string | null>;
|
||||
/** Active project's saved crawl config, or null on the default scratch
|
||||
* DB (renderer keeps its global-prefs config in that case). */
|
||||
projectConfigGet(): Promise<CrawlConfig | null>;
|
||||
/** Persist a crawl config to the active project (no-op on scratch DB). */
|
||||
projectConfigSet(config: CrawlConfig): Promise<void>;
|
||||
/** Fires when the active project changes; payload is the project's saved
|
||||
* config, or null when it has none yet / is the scratch DB. */
|
||||
onProjectConfigChanged(cb: (config: CrawlConfig | null) => void): () => void;
|
||||
/** Full recent-projects list including archived entries. */
|
||||
recentProjectsList(): Promise<RecentProject[]>;
|
||||
/** Archive / unarchive a recent project (hides it from Open Recent). */
|
||||
recentProjectSetArchived(path: string, archived: boolean): Promise<void>;
|
||||
/** Replace the tag set on a recent project. */
|
||||
recentProjectSetTags(path: string, tags: string[]): Promise<void>;
|
||||
/** Drop a project from the recent list. */
|
||||
recentProjectRemove(path: string): Promise<void>;
|
||||
projectSaveEncrypted(
|
||||
password: string,
|
||||
): Promise<{ filePath: string; bytesWritten: number } | { error: string } | null>;
|
||||
@@ -1568,7 +1606,10 @@ export interface FreeCrawlApi {
|
||||
prefsSet(key: string, value: unknown): void;
|
||||
prefsDelete(key: string): void;
|
||||
confirmClear(): Promise<ConfirmClearResult>;
|
||||
logsGetAll(): Promise<LogEntry[]>;
|
||||
/** Snapshot of the live log ring. `ownerId` (a project window's
|
||||
* webContents.id) scopes the result to that window's crawler entries plus
|
||||
* app-global entries; omitted returns everything (primary/legacy view). */
|
||||
logsGetAll(ownerId?: number): Promise<LogEntry[]>;
|
||||
logsClear(): Promise<void>;
|
||||
logsOpenWindow(): Promise<void>;
|
||||
openVisualizationWindow(): Promise<void>;
|
||||
|
||||
@@ -4,3 +4,19 @@ export interface ProjectMeta {
|
||||
startUrl: string;
|
||||
notes: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* One entry in the recent-projects list. Persisted in `preferences.json`
|
||||
* under `recentProjects`. Legacy entries were bare path strings and are
|
||||
* migrated to this shape on read (see `getRecentProjectEntries`).
|
||||
*/
|
||||
export interface RecentProject {
|
||||
/** Absolute `.seoproject` file path. */
|
||||
path: string;
|
||||
/** Hidden from the default recent list + File → Open Recent menu. */
|
||||
archived?: boolean;
|
||||
/** Free-form user labels for filtering (e.g. "client-x", "monthly"). */
|
||||
tags?: string[];
|
||||
/** ISO timestamp this project was last opened. */
|
||||
lastOpened?: string;
|
||||
}
|
||||
Reference in new issue
Block a user