A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog–style UI, ~70 SEO issue checks, and zero native dependencies.

✨ Highlights

🚀 Crawler

undici HTTP client with keep-alive Agent
80–150 URL/s typical throughput
1M+ URL crawls on a single machine
robots.txt, sitemaps, redirect chains
HTTP Basic / Bearer auth, proxy override
Pause / Resume / manual URL injection

🔍 Analysis

~70 SEO issue checks across 14 tabs
Near-duplicate clustering (SimHash + LSH)
Full hreflang validation, sitemap diff
OpenGraph / Twitter Card / JSON-LD / AMP
SSL/TLS cert audit, security headers
Custom CSS + Regex extraction (10 rules)

🖥 Desktop UI

Dense dark theme, virtualized 1M+ row tables
Live-streaming rows during crawl
Advanced AND/OR search (24 fields × 12 ops)
Detail panel with 9 sub-tabs per URL
Cytoscape graph + anchor-text word cloud
Diagnostic popups for DNS/TLS/proxy issues

📤 Export & Reports

16 reports (histograms, top URLs, link positions)
Bulk export (22 categorised CSVs)
Standalone HTML audit report
Sitemap generator (image / hreflang / sharded / gz)
Project-vs-project compare diff
Webhook on completion + OS notifications

🛠 Tech Stack

Layer	Choice
🟢 Runtime	Node.js 22 LTS+ (ESM-first)
📘 Language	TypeScript 5.7+ strict
🪟 Desktop shell	Electron 41
⚡ Build	electron-vite 5 / Vite 7
🎨 UI	React 19 + Tailwind 3.4 + Zustand 5
📊 Tables	`@tanstack/react-table` + `@tanstack/react-virtual`
🌐 HTTP	undici 8
🔎 HTML parse	cheerio (htmlparser2 fast path)
📥 Queue	p-queue 8
🤖 robots	robots-parser 3
💾 Storage	`node:sqlite` + WAL — zero native deps
📦 Distribution	electron-builder 26

🚀 Quick Start

Tip

End users: download the prebuilt installer from the Releases page — no setup required.

🪟 Windows — easiest path is the .bat launcher

Double-click FreeCrawl-SEO-Tool-Start.bat at the repo root. It verifies Node.js, runs npm install on first launch, then starts the app with npm run dev.

Or manually:

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

🍎 macOS — Apple Silicon + Intel

# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install      # Command Line Tools — required once

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

If macOS Gatekeeper blocks an unsigned local build ("App is damaged"):

xattr -cr "/Applications/FreeCrawl SEO.app"

🐧 Linux — Debian / Ubuntu / Fedora / Arch

# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git

# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch          : sudo pacman -S nodejs npm git

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

Some headless / minimal distros also need GTK/X11 runtime libs for Electron: sudo apt install -y libgtk-3-0 libnss3 libasound2t64

⌨ CLI (headless crawl)

npm run build:cli
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON

📦 Production build (per-platform installers)

npm run build                                  # all packages + desktop + CLI
npm --workspace apps/desktop run build:win     # Windows installer (NSIS)
npm --workspace apps/desktop run build:mac     # macOS DMG
npm --workspace apps/desktop run build:linux   # AppImage / .deb

📋 Prerequisites

For developers / source builds

Component	Minimum	Where
Node.js	22 LTS (24 also OK)	nodejs.org
npm	10+ (ships with Node)	bundled
Git	any recent	git-scm.com

Why no Python / MSBuild / node-gyp? FreeCrawl uses Node 22's built-in node:sqlite instead of better-sqlite3. There are zero native dependencies — npm install never invokes a C++ compiler.

Verify your setup:

node --version    # v22.x.x or v24.x.x
npm --version     # 10+

Runtime requirements (any platform)

Outbound HTTPS access to the sites you crawl. Behind a corporate proxy? Set HTTPS_PROXY=http://your-proxy:port before launch — undici's ProxyAgent routes through it automatically.
TLS root certificates. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt — otherwise crawls fail with UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

Disk + memory budget

Resource	Size
`node_modules` after `npm install`	~600 MB
Production Electron build	~150 MB
Peak RAM, 100K-URL crawl	~100 MB
1M-URL crawl	comfortably under 1 GB

📁 Project Structure

FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat   # Windows one-click launcher
├── 📄 CHANGELOG.md                   # versioned release notes
├── 📂 apps/
│   ├── 🪟 desktop/                   # Electron app (main + preload + renderer)
│   └── ⌨  cli/                       # headless Node CLI
└── 📂 packages/
    ├── 🔗 shared-types/              # IPC + domain types
    ├── 💾 db/                        # ProjectDb (node:sqlite) + migrations
    └── 🕷 core/                      # crawler engine (UI-agnostic)

Dependency graph

graph LR
  A[shared-types] --> B[db]
  B --> C[core]
  C --> D[desktop]
  C --> E[cli]

📈 Status

Note

Active development. All 14 analysis tabs, advanced search, ~70 issue categories, sitemap export variants, JSON / CSV / HTML reports, list mode, custom extraction, near-duplicate detection, hreflang validation, project compare, Cytoscape visualization, auth, proxy, webhook, OS notifications, robots.txt tester, in-app logs, and diagnostic popups are working. Live-streaming UX with first row in ~1 s.

Upcoming: plugin system, JavaScript rendering, log analyzer, PageSpeed API integration.

🤝 Contributing & Support


🐛 Found a bug?	Open an issue
💡 Have a feature idea?	Start a discussion
📦 Want the prebuilt app?	Download a release
🌐 Project website	freecrawl.net

📜 License

MIT — see LICENSE

_{Built with ❤ for SEO professionals who want a fast, free, open alternative to Screaming Frog.}

README.md Unescape Escape

FreeCrawl SEO Tool

Open-source desktop SEO crawler — a free, cross-platform alternative to Screaming Frog