FreeCrawl-SEO-Tool/README.md

FreeCrawl SEO Tool

Open-source desktop SEO crawler — a free, cross-platform alternative to Screaming Frog

🌐 Website · 📦 Download · 🐛 Report Bug · 📝 Changelog

A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, ~70 SEO issue checks, and zero native dependencies.

---

✨ Highlights

🚀 Crawler undici HTTP client with keep-alive Agent 80–150 URL/s typical throughput 1M+ URL crawls on a single machine robots.txt, sitemaps, redirect chains HTTP Basic / Bearer auth, proxy override Pause / Resume / manual URL injection	🔍 Analysis ~70 SEO issue checks across 14 tabs Near-duplicate clustering (SimHash + LSH) Full hreflang validation, sitemap diff OpenGraph / Twitter Card / JSON-LD / AMP SSL/TLS cert audit, security headers Custom CSS + Regex extraction (10 rules)
🖥 Desktop UI Dense dark theme, virtualized 1M+ row tables Live-streaming rows during crawl Advanced AND/OR search (24 fields × 12 ops) Detail panel with 9 sub-tabs per URL Cytoscape graph + anchor-text word cloud Diagnostic popups for DNS/TLS/proxy issues	📤 Export & Reports 16 reports (histograms, top URLs, link positions) Bulk export (22 categorised CSVs) Standalone HTML audit report Sitemap generator (image / hreflang / sharded / gz) Project-vs-project compare diff Webhook on completion + OS notifications

---

🛠 Tech Stack

Layer	Choice
🟢 Runtime	Node.js 22 LTS+ (ESM-first)
📘 Language	TypeScript 5.7+ strict
🪟 Desktop shell	Electron 41
⚡ Build	electron-vite 5 / Vite 7
🎨 UI	React 19 + Tailwind 3.4 + Zustand 5
📊 Tables	@tanstack/react-table + @tanstack/react-virtual
🌐 HTTP	undici 8
🔎 HTML parse	cheerio (htmlparser2 fast path)
📥 Queue	p-queue 8
🤖 robots	robots-parser 3
💾 Storage	node:sqlite + WAL — zero native deps
📦 Distribution	electron-builder 26

---

🚀 Quick Start

Tip

End users: download the prebuilt installer from the Releases page — no setup required.

🪟 Windows — easiest path is the .bat launcher

Double-click FreeCrawl-SEO-Tool-Start.bat at the repo root. It verifies Node.js, runs npm install on first launch, then starts the app with npm run dev.

Or manually:

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

🍎 macOS — Apple Silicon + Intel

# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install      # Command Line Tools — required once

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

If macOS Gatekeeper blocks an unsigned local build ("App is damaged"):

xattr -cr "/Applications/FreeCrawl SEO.app"

🐧 Linux — Debian / Ubuntu / Fedora / Arch

# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git

# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch          : sudo pacman -S nodejs npm git

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

Some headless / minimal distros also need GTK/X11 runtime libs for Electron: sudo apt install -y libgtk-3-0 libnss3 libasound2t64

⌨ CLI (headless crawl)

npm run build:cli
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON

📦 Production build (per-platform installers)

npm run build                                  # all packages + desktop + CLI
npm --workspace apps/desktop run build:win     # Windows installer (NSIS)
npm --workspace apps/desktop run build:mac     # macOS DMG
npm --workspace apps/desktop run build:linux   # AppImage / .deb

🤖 MCP server — query your crawl from Claude / any MCP client

FreeCrawl ships an MCP (Model Context Protocol) server that exposes the active .seoproject to AI agents over stdio. Read-only — runs alongside the desktop app without contention (SQLite WAL).

Tools: get_summary, get_overview_counts, top_issues, query_urls, get_url_detail, list_projects, set_project, current_project.

1. Build it once:

npm run build:mcp

This produces apps/mcp-server/dist/index.js.

2. Register it with your MCP client.

Claude Desktop

Edit your Claude Desktop config:

Platform	Path
Windows	%APPDATA%\Claude\claude_desktop_config.json
macOS	~/Library/Application Support/Claude/claude_desktop_config.json
Linux	~/.config/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js"]
    }
  }
}

Restart Claude Desktop. The freecrawl server appears under the tool 🔌 icon.

Claude Code (CLI)

claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

Other MCP clients

Run the binary directly with stdio transport:

node apps/mcp-server/dist/index.js

The server speaks newline-delimited JSON-RPC 2.0 — point any MCP-compatible client at it.

3. Try it. Ask your agent things like:

"Show the 10 URLs with the longest response time in my last crawl." "What are the top 5 issue categories with the most affected pages?" "List every URL with a missing meta description."

Pointing at a non-default project:

By default the server reads <userData>/projects/default.seoproject (the same file the desktop app uses). Override with the FREECRAWL_PROJECT env var, or call the set_project tool mid-session:

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/path/to/apps/mcp-server/dist/index.js"],
      "env": { "FREECRAWL_PROJECT": "/path/to/audit.seoproject" }
    }
  }
}

---

📋 Prerequisites

For developers / source builds

Component	Minimum	Where
Node.js	22 LTS (24 also OK)	nodejs.org
npm	10+ (ships with Node)	bundled
Git	any recent	git-scm.com

Why no Python / MSBuild / node-gyp? FreeCrawl uses Node 22's built-in node:sqlite instead of better-sqlite3. There are zero native dependencies — npm install never invokes a C++ compiler.

Verify your setup:

node --version    # v22.x.x or v24.x.x
npm --version     # 10+

Runtime requirements (any platform)

• Outbound HTTPS access to the sites you crawl. Behind a corporate proxy? Set HTTPS_PROXY=http://your-proxy:port before launch — undici's ProxyAgent routes through it automatically.
• TLS root certificates. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt — otherwise crawls fail with UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

Disk + memory budget

Resource	Size
node_modules after npm install	~600 MB
Production Electron build	~150 MB
Peak RAM, 100K-URL crawl	~100 MB
1M-URL crawl	comfortably under 1 GB

---

📁 Project Structure

FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat   # Windows one-click launcher
├── 📄 CHANGELOG.md                   # versioned release notes
├── 📂 apps/
│   ├── 🪟 desktop/                   # Electron app (main + preload + renderer)
│   └── ⌨  cli/                       # headless Node CLI
└── 📂 packages/
    ├── 🔗 shared-types/              # IPC + domain types
    ├── 💾 db/                        # ProjectDb (node:sqlite) + migrations
    └── 🕷 core/                      # crawler engine (UI-agnostic)

Dependency graph

graph LR
  A[shared-types] --> B[db]
  B --> C[core]
  C --> D[desktop]
  C --> E[cli]

---

📈 Status

Note

Active development. All 14 analysis tabs, advanced search, ~70 issue categories, sitemap export variants, JSON / CSV / HTML reports, list mode, custom extraction, near-duplicate detection, hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, OS notifications, robots.txt tester, in-app logs, and diagnostic popups are working. Live-streaming UX with first row in ~1 s.

Upcoming: plugin system, JavaScript rendering, log analyzer, PageSpeed API integration.

---

🤝 Contributing & Support

🐛 Found a bug?	Open an issue
💡 Have a feature idea?	Start a discussion
📦 Want the prebuilt app?	Download a release
🌐 Project website	freecrawl.net

---

📜 License

MIT — see LICENSE

_{Built with ❤ for SEO professionals who want a fast, free, open alternative to Screaming Frog.}