FreeCrawl SEO Tool
Open-source desktop SEO crawler — a free, cross-platform alternative to Screaming Frog
🌐 Website · 📦 Download · 🐛 Report Bug · 📝 Changelog
A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, ~80 SEO issue checks, and zero native dependencies.
✨ Highlights
🚀 Crawler
|
🔍 Analysis
|
🖥 Desktop UI
|
📤 Export & Reports
|
🛠 Tech Stack
| Layer | Choice |
|---|---|
| 🟢 Runtime | Node.js 22 LTS+ (ESM-first) |
| 📘 Language | TypeScript 5.7+ strict |
| 🪟 Desktop shell | Electron 41 |
| ⚡ Build | electron-vite 5 / Vite 7 |
| 🎨 UI | React 19 + Tailwind 3.4 + Zustand 5 |
| 📊 Tables | @tanstack/react-table + @tanstack/react-virtual |
| 🌐 HTTP | undici 8 |
| 🔎 HTML parse | cheerio (htmlparser2 fast path) |
| 📥 Queue | p-queue 8 |
| 🤖 robots | robots-parser 3 |
| 💾 Storage | node:sqlite + WAL — zero native deps |
| 📦 Distribution | electron-builder 26 |
🚀 Quick Start
Tip
End users: download the prebuilt installer from the Releases page — no setup required.
🪟 Windows — easiest path is the .bat launcher
Double-click FreeCrawl-SEO-Tool-Start.bat at the repo root. It verifies Node.js, runs npm install on first launch, then starts the app with npm run dev.
Or manually:
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev
🍎 macOS — Apple Silicon + Intel
# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install # Command Line Tools — required once
# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev
If macOS Gatekeeper blocks an unsigned local build ("App is damaged"):
xattr -cr "/Applications/FreeCrawl SEO.app"
🐧 Linux — Debian / Ubuntu / Fedora / Arch
# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git
# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch : sudo pacman -S nodejs npm git
# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev
Some headless / minimal distros also need GTK/X11 runtime libs for Electron:
sudo apt install -y libgtk-3-0 libnss3 libasound2t64
⌨ CLI (headless crawl)
npm run build:cli
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json # list mode + JSON
📦 Production build (per-platform installers)
npm run build # all packages + desktop + CLI
npm --workspace apps/desktop run build:win # Windows installer (NSIS)
npm --workspace apps/desktop run build:mac # macOS DMG
npm --workspace apps/desktop run build:linux # AppImage / .deb
🤖 MCP server — query your crawl from Claude / any MCP client
FreeCrawl ships an MCP (Model Context Protocol) server that exposes the active .seoproject to AI agents over stdio. Read-only — runs alongside the desktop app without contention (SQLite WAL).
Tools: get_summary, get_overview_counts, top_issues, query_urls, get_url_detail, list_projects, set_project, current_project.
1. Build it once:
npm run build:mcp
This produces apps/mcp-server/dist/index.js.
2. Register it with your MCP client.
Claude Desktop
Edit your Claude Desktop config:
| Platform | Path |
|---|---|
| Windows | %APPDATA%\Claude\claude_desktop_config.json |
| macOS | ~/Library/Application Support/Claude/claude_desktop_config.json |
| Linux | ~/.config/Claude/claude_desktop_config.json |
{
"mcpServers": {
"freecrawl": {
"command": "node",
"args": ["/absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js"]
}
}
}
Restart Claude Desktop. The freecrawl server appears under the tool 🔌 icon.
Claude Code (CLI)
claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js
Other MCP clients
Run the binary directly with stdio transport:
node apps/mcp-server/dist/index.js
The server speaks newline-delimited JSON-RPC 2.0 — point any MCP-compatible client at it.
3. Try it. Ask your agent things like:
"Show the 10 URLs with the longest response time in my last crawl." "What are the top 5 issue categories with the most affected pages?" "List every URL with a missing meta description."
Pointing at a non-default project:
By default the server reads <userData>/projects/default.seoproject (the same file the desktop app uses). Override with the FREECRAWL_PROJECT env var, or call the set_project tool mid-session:
{
"mcpServers": {
"freecrawl": {
"command": "node",
"args": ["/path/to/apps/mcp-server/dist/index.js"],
"env": { "FREECRAWL_PROJECT": "/path/to/audit.seoproject" }
}
}
}
📋 Prerequisites
For developers / source builds
| Component | Minimum | Where |
|---|---|---|
| Node.js | 22 LTS (24 also OK) | nodejs.org |
| npm | 10+ (ships with Node) | bundled |
| Git | any recent | git-scm.com |
Why no Python / MSBuild / node-gyp? FreeCrawl uses Node 22's built-in
node:sqliteinstead ofbetter-sqlite3. There are zero native dependencies —npm installnever invokes a C++ compiler.
Verify your setup:
node --version # v22.x.x or v24.x.x
npm --version # 10+
Runtime requirements (any platform)
- Outbound HTTPS access to the sites you crawl. Behind a corporate proxy? Set
HTTPS_PROXY=http://your-proxy:portbefore launch — undici'sProxyAgentroutes through it automatically. - TLS root certificates. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set
NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt— otherwise crawls fail withUNABLE_TO_GET_ISSUER_CERT_LOCALLY.
Disk + memory budget
| Resource | Size |
|---|---|
node_modules after npm install |
~600 MB |
| Production Electron build | ~150 MB |
| Peak RAM, 100K-URL crawl | ~100 MB |
| 1M-URL crawl | comfortably under 1 GB |
📁 Project Structure
FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat # Windows one-click launcher
├── 📄 CHANGELOG.md # versioned release notes
├── 📂 apps/
│ ├── 🪟 desktop/ # Electron app (main + preload + renderer)
│ └── ⌨ cli/ # headless Node CLI
└── 📂 packages/
├── 🔗 shared-types/ # IPC + domain types
├── 💾 db/ # ProjectDb (node:sqlite) + migrations
└── 🕷 core/ # crawler engine (UI-agnostic)
Dependency graph
graph LR
A[shared-types] --> B[db]
B --> C[core]
C --> D[desktop]
C --> E[cli]
📈 Status
Note
Active development. All 14 analysis tabs, advanced search, ~80 issue categories, sitemap export variants, JSON / CSV / XLSX / HTML reports, list mode, custom extraction, near-duplicate detection (per-URL Duplicates view), hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, MCP server, OS notifications, robots.txt tester, URL rewriting + preview, in-app logs, and diagnostic popups are working. Live-streaming UX with first row in ~1 s.
Upcoming: plugin system, JavaScript rendering, log analyzer, PageSpeed API integration.
🤝 Contributing & Support
| 🐛 Found a bug? | Open an issue |
| 💡 Have a feature idea? | Start a discussion |
| 📦 Want the prebuilt app? | Download a release |
| 🌐 Project website | freecrawl.net |
📜 License
MIT — see LICENSE
Built with ❤ for SEO professionals who want a fast, free, open alternative to Screaming Frog.