A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, 150+ SEO issue checks, and zero native dependencies.

✨ Highlights

🚀 Crawler

undici HTTP client with keep-alive Agent
80–150 URL/s typical throughput
1M+ URL crawls on a single machine
robots.txt, sitemaps, redirect chains
HTTP Basic / Bearer auth, proxy override
Pause / Resume / manual URL injection

🔍 Analysis

150+ SEO issue checks across 24 top-level tabs
Near-duplicate clustering (SimHash + LSH) + exact-duplicate content hash
Full hreflang validation (reciprocity, self-ref, inconsistent lang)
OpenGraph / Twitter Card / JSON-LD / AMP / Web App Manifest parsing
Structured-data validation — duplicate @id, malformed @type, missing required props
Accessibility — <main> landmark, skip-link, ARIA roles, heading order
SSL/TLS cert chain audit, security headers, CORS audit
Active vs passive mixed content split
Readability (Flesch, Flesch–Kincaid, Gunning Fog)
Custom CSS + Regex extraction (10 rules) + /regex/ literal Custom Search

🖥 Desktop UI

Dense dark theme, virtualized 1M+ row tables
Multi-language UI (English + Turkish) — switch instantly from Settings → Language
Live-streaming rows during crawl
Advanced AND/OR search (24 fields × 12 ops)
Detail panel with 13 sub-tabs — HTTP Headers has Request/Response toggle + status-code diagnosis banner (decodes Cloudflare 52x/530 and 4xx/5xx with header signals)
Per-URL Duplicates view (cluster siblings)
Live memory monitor in status bar (RSS / Sys Free / per-URL cost / capacity projection)
Robots.txt syntax validator in the Robots Tester (typo suggestions, orphan rules, sitemap URL check)
In-app scheduled crawl (hourly / daily / weekly / custom) per project
.seoproject file association — double-click a project file from the OS to open it
URL rewriting (regex + whitelist + live preview)
Cytoscape graph + anchor-text word cloud + Top Words report
Custom CSS theme override (custom-theme.css in user-data folder)
Diagnostic popups for DNS/TLS/proxy issues

📤 Export & Reports

21 reports — histograms, top/bottom URLs, link positions, top words
Export Dialog with column picker — CSV or XLSX (multi-sheet)
Bulk export (22 categorised CSVs)
Standalone HTML audit report
Sitemap generator (image / hreflang / sharded / gz)
Project-vs-project compare diff
MCP server for AI agents — drive crawls live from Claude Code, poll progress, plus full read-only DB access
Webhook on completion + OS notifications

🛠 Tech Stack

Layer	Choice
🟢 Runtime	Node.js 22 LTS+ (ESM-first)
📘 Language	TypeScript 5.7+ strict
🪟 Desktop shell	Electron 41
⚡ Build	electron-vite 5 / Vite 7
🎨 UI	React 19 + Tailwind 3.4 + Zustand 5
📊 Tables	`@tanstack/react-table` + `@tanstack/react-virtual`
🌐 HTTP	undici 8
🔎 HTML parse	cheerio (htmlparser2 fast path)
📥 Queue	p-queue 8
🤖 robots	robots-parser 3
💾 Storage	`node:sqlite` + WAL — zero native deps
📦 Distribution	electron-builder 26

🚀 Quick Start

Tip

End users: download the prebuilt installer from the Releases page — no setup required.

🪟 Windows — easiest path is the .bat launcher

Double-click FreeCrawl-SEO-Tool-Start.bat at the repo root. It verifies Node.js, runs npm install on first launch, then starts the app with npm run dev.

Don't want to install? Grab the portable .exe from the Releases page — runs without installation.

Or manually:

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

🍎 macOS — Apple Silicon + Intel

Easiest path is the FreeCrawl-SEO-Tool-Start.sh launcher at the repo root — same one-click flow as the Windows .bat (verifies Node, prompts to install on first run, then starts the app).

chmod +x FreeCrawl-SEO-Tool-Start.sh
./FreeCrawl-SEO-Tool-Start.sh

Or manually:

# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install      # Command Line Tools — required once

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

If macOS Gatekeeper blocks an unsigned local build ("App is damaged"):

xattr -cr "/Applications/FreeCrawl SEO.app"

🐧 Linux — Debian / Ubuntu / Fedora / Arch

Easiest path is the FreeCrawl-SEO-Tool-Start.sh launcher at the repo root (same as macOS).

Prebuilt installers are available for all three families: .AppImage (universal), .deb (Debian / Ubuntu), and .rpm (Fedora / RHEL).

# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git

# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch          : sudo pacman -S nodejs npm git

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

Some headless / minimal distros also need GTK/X11 runtime libs for Electron: sudo apt install -y libgtk-3-0 libnss3 libasound2t64

⌨ CLI (headless crawl)

npm run build:cli
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON

📦 Production build (per-platform installers)

npm run build                                  # all packages + desktop + CLI
npm --workspace apps/desktop run build:win     # Windows installer (NSIS) + portable .exe
npm --workspace apps/desktop run build:mac     # macOS DMG (arm64 + x64)
npm --workspace apps/desktop run build:linux   # AppImage / .deb / .rpm

🤖 MCP server — query AND drive crawls from Claude / any MCP client

FreeCrawl ships an MCP (Model Context Protocol) server that exposes the active .seoproject to AI agents over stdio. Two capabilities in one server:

Read-only data access to the SQLite project — runs alongside the desktop app without contention (WAL allows concurrent readers).
Live crawl control — when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to <userData>/mcp-bridge.json on app launch).

14 tools:

Group	Tools
📊 Data queries (always available)	`get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail`
📁 Project management	`list_projects`, `set_project`, `current_project`
🕷 Crawl control (desktop must be open)	`start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `get_crawl_progress`, `get_desktop_project`

start_crawl accepts a startUrl plus optional whitelisted overrides (scope, maxDepth, maxUrls, maxConcurrency, maxRps, crawlDelayMs, requestTimeoutMs, respectRobotsTxt, followRedirects, crawlExternal, userAgent, include/excludePatterns) — anything you don't override keeps the desktop user's saved value. Crawls launched via MCP go through the same code path as the UI's Start button, so progress shows up in the desktop app live as the agent drives it.

1. Build it once:

npm run build:mcp

This produces apps/mcp-server/dist/index.js.

2. Register it with your MCP client.

Claude Desktop

Edit your Claude Desktop config:

Platform	Path
Windows	`%APPDATA%\Claude\claude_desktop_config.json`
macOS	`~/Library/Application Support/Claude/claude_desktop_config.json`
Linux	`~/.config/Claude/claude_desktop_config.json`

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js"]
    }
  }
}

Restart Claude Desktop. The freecrawl server appears under the tool 🔌 icon.

Claude Code (CLI)

claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

Other MCP clients

Run the binary directly with stdio transport:

node apps/mcp-server/dist/index.js

The server speaks newline-delimited JSON-RPC 2.0 — point any MCP-compatible client at it.

3. Try it. Ask your agent things like:

"Crawl https://example.com with maxDepth 3 and watch the progress." "Show the 10 URLs with the longest response time in my last crawl." "What are the top 5 issue categories with the most affected pages?" "List every URL with a missing meta description." "Pause the running crawl, then resume it once I've checked the first 1000 URLs."

Pointing at a non-default project:

By default the server reads <userData>/projects/default.seoproject (the same file the desktop app uses). Override with the FREECRAWL_PROJECT env var, or call the set_project tool mid-session:

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/path/to/apps/mcp-server/dist/index.js"],
      "env": { "FREECRAWL_PROJECT": "/path/to/audit.seoproject" }
    }
  }
}

📋 Prerequisites

For developers / source builds

Component	Minimum	Where
Node.js	22 LTS (24 also OK)	nodejs.org
npm	10+ (ships with Node)	bundled
Git	any recent	git-scm.com

Why no Python / MSBuild / node-gyp? FreeCrawl uses Node 22's built-in node:sqlite instead of better-sqlite3. There are zero native dependencies — npm install never invokes a C++ compiler.

Verify your setup:

node --version    # v22.x.x or v24.x.x
npm --version     # 10+

Runtime requirements (any platform)

Outbound HTTPS access to the sites you crawl. Behind a corporate proxy? Set HTTPS_PROXY=http://your-proxy:port before launch — undici's ProxyAgent routes through it automatically.
TLS root certificates. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt — otherwise crawls fail with UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

Disk + memory budget

Resource	Size
`node_modules` after `npm install`	~600 MB
Production Electron build	~150 MB
Peak RAM, 100K-URL crawl	~100 MB
1M-URL crawl	comfortably under 1 GB

📁 Project Structure

FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat   # Windows one-click launcher
├── 📄 FreeCrawl-SEO-Tool-Start.sh    # macOS / Linux one-click launcher
├── 📄 CHANGELOG.md                   # versioned release notes
├── 📂 apps/
│   ├── 🪟 desktop/                   # Electron app (main + preload + renderer)
│   ├── ⌨  cli/                       # headless Node CLI
│   └── 🤖 mcp-server/                # MCP server for AI agents
└── 📂 packages/
    ├── 🔗 shared-types/              # IPC + domain types
    ├── 💾 db/                        # ProjectDb (node:sqlite) + migrations
    └── 🕷 core/                      # crawler engine (UI-agnostic)

Dependency graph

graph LR
  A[shared-types] --> B[db]
  B --> C[core]
  C --> D[desktop]
  C --> E[cli]
  B --> F[mcp-server]

📈 Status

Note

Active development. All 24 analysis tabs, advanced search, 150+ issue categories, sitemap export variants, JSON / CSV / XLSX / HTML reports, list mode, custom extraction, near-duplicate + exact-duplicate detection, hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, MCP server with crawl control + live progress, OS notifications, robots.txt syntax validator, URL rewriting + preview, status-code diagnosis banner, live memory monitor, in-app scheduled crawl, multi-language UI (EN + TR), .seoproject file association, in-app logs, and diagnostic popups are working. Cross-platform installers (Windows .exe + portable, macOS .dmg, Linux .AppImage / .deb / .rpm). Live-streaming UX with first row in ~1 s.

Upcoming: plugin system, JavaScript rendering, log analyzer, PageSpeed API integration.

🤝 Contributing & Support


🐛 Found a bug?	Open an issue
💡 Have a feature idea?	Start a discussion
📦 Want the prebuilt app?	Download a release
🌐 Project website	freecrawl.net

📜 License

MIT — see LICENSE

_{Built with ❤ for SEO professionals who want a fast, free, open alternative to Screaming Frog.}

README.md Unescape Escape

FreeCrawl SEO Tool

Open-source desktop SEO crawler — a free, cross-platform alternative to Screaming Frog