A high-performance website crawler built for serious SEO audits. Targets 1M+ URLs on a single machine, with a dense Screaming Frog-style UI, 181 SEO issue checks, an interface in 11 languages with dark and light themes, and zero native dependencies.

Important

v1.0.0 is the full release. Everything planned for 1.0 ships in this build, and it is ready for daily audit work. Development does not stop here: further updates will follow based on user feedback and bug reports, so if something gets in your way, open an issue.

✨ What FreeCrawl Does

FreeCrawl crawls your website and checks it for SEO problems. It is a free desktop alternative to Screaming Frog that handles very large sites (1M+ URLs on one machine) and streams the results into a fast, dense table while the crawl is still running.

It runs 181 SEO checks across 33 tabs: page titles, meta descriptions, headings, duplicate content, broken links, redirects, canonicals, hreflang, structured data, security headers, readability, mobile usability, soft 404s, and more. It can also render JavaScript pages with a real headless browser to capture the final DOM, screenshots, and Core Web Vitals. Any check you do not care about can be switched off per project.

Every link type (images, CSS, JavaScript, media, canonicals, pagination, hreflang, AMP, iframes) has its own Crawl and Store switch, so you decide exactly what gets fetched and what gets reported. Right-click any table selection to copy cells, rows, or columns straight to the clipboard.

The crawler is built for real sites, not only clean ones. It recognises bot-protection challenges (Cloudflare, AWS WAF, Vercel, Sucuri, DataDome) and marks those URLs as blocked instead of reporting hundreds of false broken links, re-checks external links with a GET and a browser User-Agent before calling them broken, probes the site for soft-404 behaviour after every spider crawl, and can slow itself down on 429 / Retry-After responses with the opt-in Adaptive speed switch. srcset image variants are listed but not fetched unless you turn them on, the status bar shows every post-crawl phase (measuring images, running checks) as it happens, and stored page bodies are zstd-compressed with a per-project dictionary so large projects stay small on disk.

Pull in real data from Google Search Console, Analytics 4, PageSpeed Insights, Chrome UX Report, and Yandex Metrica, check spelling and grammar with LanguageTool (with a bundled offline dictionary for languages LanguageTool has no rules for, such as Turkish), and run your own AI prompts on each page (OpenAI, Claude, or Ollama). Open several projects side by side, each in its own window running its own crawl, and manage them from a Projects dialog with search, tags, and starter templates.

Export to Excel, CSV, JSON, XML, an HTML or brandable PDF report, sitemaps, a Screaming Frog-compatible SEO Audit folder, Google Sheets, or BigQuery. Save each project as one compressed file (or a password-protected encrypted file), and export any URL Details sub-table on its own. A built-in MCP server (100+ tools) lets AI agents like Claude drive crawls live and read every result. Every action a human can take in the desktop is callable from an agent, and several agents can run in parallel, each in its own isolated crawl session.

The interface speaks 11 languages (English, Turkish, Spanish, French, Portuguese, Italian, Russian, Chinese, Korean, Hindi, Azerbaijani), follows your system language on first launch, and comes in dark and light themes. Everything runs fully local: no telemetry, no cloud, MIT-licensed.

📥 Download

Grab the prebuilt app from the Releases page. Every file name says which platform and CPU it is for, so pick the row that matches your machine (<version> stands for the release number, for example 1.0.0).

Your machine	File to download	Notes
🪟 Windows 10 / 11 (64-bit)	`FreeCrawl-SEO-Tool-<version>-Windows-Setup.exe`	Installer with Start Menu and desktop shortcuts
🪟 Windows, no install	`FreeCrawl-SEO-Tool-<version>-Windows-Portable.exe`	Single file, runs from anywhere, including a USB stick
🍎 Mac with Apple Silicon (M1, M2, M3, M4 and newer)	`FreeCrawl-SEO-Tool-<version>-macOS-Apple-Silicon.dmg`	Native build, fastest on these Macs
🍎 Mac with an Intel processor	`FreeCrawl-SEO-Tool-<version>-macOS-Intel.dmg`	For Intel-based Macs (2020 and earlier models)
🐧 Linux, any distribution	`FreeCrawl-SEO-Tool-<version>-Linux-x86_64.AppImage`	Make it executable and run, no install (needs FUSE 2, see below)
🐧 Linux, Debian / Ubuntu	`FreeCrawl-SEO-Tool-<version>-Linux-amd64.deb`	`sudo apt install ./FreeCrawl-SEO-Tool-<version>-Linux-amd64.deb`
🐧 Linux, Fedora / RHEL	`FreeCrawl-SEO-Tool-<version>-Linux-x86_64.rpm`	`sudo dnf install ./FreeCrawl-SEO-Tool-<version>-Linux-x86_64.rpm`

Not sure which Mac you have? Open the Apple menu and choose About This Mac. A line that says Chip: Apple M... means Apple Silicon; a line that says Processor: Intel... means Intel. If you already run the Intel build on an Apple Silicon Mac, Help → Check for Updates offers the native build.

macOS Gatekeeper: the builds are not yet notarized. On macOS 15 Sequoia and newer the first launch is blocked: open System Settings → Privacy & Security and click Open Anyway (the old right-click → Open override no longer works there). On macOS 12 to 14, right-click the app and choose Open. If macOS reports the app as damaged, run xattr -cr "/Applications/FreeCrawl SEO Tool.app" once in Terminal.

System requirements (set by Electron 41): Windows 10 or newer (64-bit); macOS 12 Monterey or newer; Linux on Ubuntu 18.04 / Debian 10 / Fedora 32 or newer. The AppImage needs FUSE 2 (sudo apt install libfuse2, on Ubuntu 24.04 libfuse2t64). On Ubuntu 24.04+ the kernel denies unprivileged user namespaces to apps without an AppArmor profile, which an AppImage cannot install, so there the AppImage starts with Chromium's sandbox disabled (it tells you so on stderr); the .deb keeps the sandbox on, because its installer registers an AppArmor profile, so prefer it on Ubuntu.

The Playwright Chromium browser is bundled in every installer, so JavaScript rendering works on first launch without an extra download. All files are built by the same GitHub Actions workflow on every release, and the .yml and .blockmap files listed next to them are build metadata you can ignore.

🛠 Tech Stack

Layer	Choice
🟢 Runtime	Node.js 22.13+ (ESM-first; `node:sqlite` is unflagged from 22.13, zstd body compression from 22.15)
📘 Language	TypeScript 5.7+ strict
🪟 Desktop shell	Electron 41
⚡ Build	electron-vite 5 / Vite 7
🎨 UI	React 19 + Tailwind 3.4 + Zustand 5
📊 Tables	Canvas-rendered URL grid + `@tanstack/react-table` + `@tanstack/react-virtual`
🌐 HTTP	undici 8
🔎 HTML parse	cheerio (htmlparser2 fast path)
📥 Queue	p-queue 8
🤖 robots	robots-parser 3
💾 Storage	`node:sqlite` + WAL, zero native deps
🌍 i18n	i18next, 11 UI languages
📦 Distribution	electron-builder 26

🚀 Quick Start

Tip

End users: you do not need any of this. Download the prebuilt app from the table above and run it. The steps below are for running or building from source.

🪟 Windows: easiest path is the .bat launcher

Double-click FreeCrawl-SEO-Tool-Start.bat at the repo root. It verifies Node.js, runs npm install on first launch, then starts the app with npm run dev.

Or manually:

git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

🍎 macOS: Apple Silicon + Intel

Easiest path is the FreeCrawl-SEO-Tool-Start.sh launcher at the repo root. It is the same one-click flow as the Windows .bat: it verifies Node, prompts to install on first run, then starts the app.

chmod +x FreeCrawl-SEO-Tool-Start.sh
./FreeCrawl-SEO-Tool-Start.sh

Or manually:

# 1. Install prerequisites (skip any you already have)
brew install node@22 git
xcode-select --install      # Command Line Tools, required once

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

If macOS Gatekeeper blocks an unsigned local build ("App is damaged"):

xattr -cr "/Applications/FreeCrawl SEO Tool.app"

🐧 Linux: Debian / Ubuntu / Fedora / Arch

Easiest path is the FreeCrawl-SEO-Tool-Start.sh launcher at the repo root (same as macOS).

Prebuilt packages are available for all three families: .AppImage (universal), .deb (Debian / Ubuntu), and .rpm (Fedora / RHEL). See the download table.

# 1. Install Node.js 22 LTS (Debian / Ubuntu via NodeSource)
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs git

# Fedora / RHEL : sudo dnf install nodejs:22 git
# Arch          : sudo pacman -S nodejs npm git

# 2. Clone and run
git clone https://github.com/kemalai/FreeCrawl-SEO-Tool.git
cd FreeCrawl-SEO-Tool
npm install
npm run dev

Some headless / minimal distros also need GTK/X11 runtime libs for Electron: sudo apt install -y libgtk-3-0 libnss3 libasound2t64

⌨ CLI (headless crawl)

npx tsc -b apps/cli     # builds the CLI + its workspace packages
node apps/cli/dist/index.js https://example.com --depth 2 --max 500 --out out.csv
node apps/cli/dist/index.js --list urls.txt --out out.json     # list mode + JSON
node apps/cli/dist/index.js https://example.com --adaptive-speed --image-variants   # back off on 429 / Retry-After, crawl srcset variants
node apps/cli/dist/index.js compare before.seoproject after.seoproject              # diff two crawls

# Yandex Metrica: pull per-page pageviews / visits / users into a project (YANDEX_METRICA_TOKEN in the env)
node apps/cli/dist/index.js metrica-counters
node apps/cli/dist/index.js metrica-fetch --counter 12345678 --days 28 --project crawl.seoproject

# Log file analysis: parse an access log into the project for crawl x log reporting
node apps/cli/dist/index.js analyze-logs samples/apache-access.txt --project crawl.seoproject
node apps/cli/dist/index.js analyze-logs access.log --format iis-w3c --verify-bots --json
node apps/cli/dist/index.js analyze-logs access_log-Jul.gz              # gzipped logs work as-is

Sample logs ship in samples/ (apache-access.txt, iis-access.txt) so you can try the Log File Analyzer immediately. In the desktop app, open Log Analyzer → Open Log Analyzer Window… and import one.

CI / CD recipes: ready-to-use GitHub Actions and GitLab CI examples that crawl your site on a schedule, fail the build when the broken-URL count exceeds a threshold, and upload the crawl as an artifact. See the CI/CD guide for the exit-code contract and a regression-gate recipe.

📦 Production build (per-platform installers)

npm run build                                  # all packages + desktop + CLI
npm --workspace apps/desktop run build:win     # Windows installer (NSIS) + portable .exe
npm --workspace apps/desktop run build:mac     # macOS DMG for this Mac's CPU (add -- --arm64 --x64 for both)
npm --workspace apps/desktop run build:linux   # AppImage / .deb / .rpm

A local build:mac names the image -macOS-arm64.dmg or -macOS-x64.dmg; the release workflow builds each architecture on its own runner and names them -macOS-Apple-Silicon.dmg / -macOS-Intel.dmg.

🤖 MCP server: query AND drive crawls from Claude / any MCP client

FreeCrawl ships an MCP (Model Context Protocol) server that exposes the active .seoproject to AI agents over stdio. Two capabilities in one server:

Read-only data access to the SQLite project. It runs alongside the desktop app without contention (WAL allows concurrent readers).
Live crawl control: when the desktop app is open, an agent can start / pause / resume / stop crawls and poll progress in real time. This goes through a localhost-only HTTP bridge (127.0.0.1, ephemeral random port, 32-byte Bearer token auth, discovery file written to <userData>/mcp-bridge.json on app launch).

100+ tools: every action a human can take in the desktop is callable from an agent, plus multi-agent session management so several agents can work in parallel. Representative groups:

Group	Tools
📊 Top-level data	`get_summary`, `get_overview_counts`, `top_issues`, `query_urls`, `get_url_detail`
🔬 Per-URL detail sub-tabs	`get_url_source` (raw + rendered + screenshot paths), `get_url_inlinks`, `get_url_outlinks`, `get_url_images`, `get_url_headers`, `get_url_duplicates`, `get_url_analytics`, `get_url_cert`
🧩 Integration rows	`query_gsc`, `query_ga4`, `query_metrica`, `query_pagespeed`, `query_crux`, `query_spelling`, `get_url_spelling`, `query_ai`, `query_seo`
🔎 Specialised queries	`query_images`, `query_broken_links`, `list_duplicate_clusters`
📈 Reports	`report_status_code_histogram`, `report_indexability_distribution`, `report_content_kind_distribution`, `report_depth_histogram`, `report_response_time_histogram`, `report_inlinks_histogram`, `report_word_count_histogram`, `report_url_length_histogram`, `report_top_urls_by`, `report_top_anchor_texts`, `report_external_domain_health`, `report_image_weight_per_page`, `report_link_position_breakdown`, `report_pages_per_directory`, `report_word_count_per_directory`, `report_sitemap_orphans`, `report_orphan_cross_source`, `report_analytics_coverage`, `report_server_header_breakdown`, `report_query_string_variants`, `report_sitemap_priority_mismatch`, `report_top_words`
📤 Exports & mutations	`export_csv`, `export_json`, `export_xml`, `export_html_report`, `export_tabular`, `export_bulk`, `export_broken_links`, `export_images`, `sitemap_generate`, `sitemap_validate`, `robots_test`, `robots_validate`, `compare_load`, `crawl_add_url`, `respider_urls`, `remove_urls`, `data_delete_by_domain`, `graph_snapshot`, `get_crawl_path`, `url_rewrite_preview`, `extraction_preview`
🔐 Project files	`project_save_as`, `project_save_encrypted`, `project_open_encrypted`
🔌 Integration fetch	`google_auth_status`, `gsc_list_sites`, `gsc_fetch`, `ga4_list_properties`, `ga4_fetch`, `metrica_list_counters`, `metrica_fetch`, `pagespeed_run`, `crux_run`, `spelling_run`, `ai_run`, `seo_run`, `get_fetch_progress`, `cancel_fetch`
⚙ Config & schedule	`get_crawl_config`, `set_crawl_config`, `schedule_get`, `schedule_set`
📁 Project management	`list_projects`, `set_project`, `current_project`
🕷 Crawl control (desktop must be open)	`start_crawl`, `stop_crawl`, `pause_crawl`, `resume_crawl`, `clear_crawl`, `get_crawl_progress`, `get_desktop_project`
🧑‍🤝‍🧑 Agent sessions (parallel agents)	`session_create`, `session_close`, `session_save`, `session_list`, `session_status`

start_crawl accepts a startUrl plus optional whitelisted overrides (scope, maxDepth, maxUrls, maxConcurrency, maxRps, crawlDelayMs, requestTimeoutMs, respectRobotsTxt, followRedirects, crawlExternal, userAgent, include/excludePatterns). Anything you don't override keeps the desktop user's saved value. Crawls launched via MCP go through the same code path as the UI's Start button, so progress shows up in the desktop app live as the agent drives it. Call clear_crawl first when you want a fresh BFS after a completed crawl with the same seed URL (otherwise the crawler treats the re-start as a resume and exits because every URL is already in the DB).

Several agents at once. Multiple MCP clients can drive one running desktop app safely. Crawl control is a lease: start_crawl returns a crawlId you own, a second start against a busy session fails with crawl-in-progress (or, with onBusy: "queue", waits its turn, or "takeover" to replace it), and stop/pause/resume refuse to touch a crawl owned by another client unless you pass force: true. clear_crawl won't wipe the desktop window's project when it has unsaved work. For full isolation, call session_create to get your own headless session inside the same app, with its own database, crawler and worker pool, so each agent crawls its own project without seeing the others; the MCP read tools automatically repoint to it. session_create({ mode: "open", projectPath }) reopens a saved .seoproject to keep working on it (PageSpeed, spelling, exports, re-crawl), and session_save / session_close({ save: true }) persist it, so the data no longer disappears when the session that produced it ends. Sessions are capped and idle ones are auto-closed (saving first if they have a document).

1. Build it once:

npm run build:mcp

This produces apps/mcp-server/dist/index.js.

2. Register it with your MCP client.

Claude Desktop

Edit your Claude Desktop config:

Platform	Path
Windows	`%APPDATA%\Claude\claude_desktop_config.json`
macOS	`~/Library/Application Support/Claude/claude_desktop_config.json`
Linux	`~/.config/Claude/claude_desktop_config.json`

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js"]
    }
  }
}

Restart Claude Desktop. The freecrawl server appears under the tool 🔌 icon.

Claude Code (CLI)

claude mcp add freecrawl -- node /absolute/path/to/FreeCrawl-SEO-Tool/apps/mcp-server/dist/index.js

Other MCP clients

Run the binary directly with stdio transport:

node apps/mcp-server/dist/index.js

The server speaks newline-delimited JSON-RPC 2.0, so point any MCP-compatible client at it.

3. Try it. Ask your agent things like:

"Crawl https://example.com with maxDepth 3 and watch the progress." "Show the 10 URLs with the longest response time in my last crawl." "What are the top 5 issue categories with the most affected pages?" "List every URL with a missing meta description." "Pause the running crawl, then resume it once I've checked the first 1000 URLs."

Pointing at a non-default project:

By default the server reads <userData>/projects/default.seoproject (the same file the desktop app uses). Override with the FREECRAWL_PROJECT env var, or call the set_project tool mid-session:

{
  "mcpServers": {
    "freecrawl": {
      "command": "node",
      "args": ["/path/to/apps/mcp-server/dist/index.js"],
      "env": { "FREECRAWL_PROJECT": "/path/to/audit.seoproject" }
    }
  }
}

📋 Prerequisites

For developers / source builds

Component	Minimum	Where
Node.js	22.13 or newer (24 also OK)	nodejs.org
npm	10+ (ships with Node)	bundled
Git	any recent	git-scm.com

Why no Python / MSBuild / node-gyp? FreeCrawl uses Node 22's built-in node:sqlite instead of better-sqlite3. There are zero native dependencies, so npm install never invokes a C++ compiler.

Verify your setup:

node --version    # v22.x.x or v24.x.x
npm --version     # 10+

Runtime requirements (any platform)

Outbound HTTPS access to the sites you crawl. Behind a corporate proxy? Set HTTPS_PROXY=http://your-proxy:port before launch, or enter a proxy URL in Settings → Network. HTTP/HTTPS proxies route through undici's ProxyAgent, and SOCKS5 / SOCKS4 proxies (socks5://…, including the h / 4a remote-DNS variants, handy for routing crawls through Tor or an SSH tunnel) are tunnelled automatically.
TLS root certificates. Node ships with the Mozilla CA bundle. If your antivirus or company proxy performs HTTPS inspection (Kaspersky, ESET, Zscaler, BlueCoat, …), set NODE_EXTRA_CA_CERTS=C:\path\to\corp-ca-bundle.crt; otherwise crawls fail with UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

Disk + memory budget

Resource	Size
`node_modules` after `npm install`	~600 MB
Production Electron build	~150 MB (installers are larger because Chromium for JS rendering is bundled)
Peak RAM, 100K-URL crawl	~100 MB
1M-URL crawl	comfortably under 1 GB

📁 Project Structure

FreeCrawl-SEO-Tool/
├── 📄 FreeCrawl-SEO-Tool-Start.bat   # Windows one-click launcher
├── 📄 FreeCrawl-SEO-Tool-Start.sh    # macOS / Linux one-click launcher
├── 📄 CHANGELOG.md                   # versioned release notes
├── 📂 apps/
│   ├── 🪟 desktop/                   # Electron app (main + preload + renderer)
│   ├── ⌨  cli/                       # headless Node CLI
│   └── 🤖 mcp-server/                # MCP server for AI agents
└── 📂 packages/
    ├── 🔗 shared-types/              # IPC + domain types
    ├── 💾 db/                        # ProjectDb (node:sqlite) + migrations
    └── 🕷 core/                      # crawler engine (UI-agnostic)

Dependency graph

graph LR
  A[shared-types] --> B[db]
  B --> C[core]
  C --> D[desktop]
  C --> E[cli]
  B --> F[mcp-server]

📈 Status

Note

v1.0.0, the full release. All 33 analysis tabs, 181 issue checks (each one can be disabled per project), JavaScript rendering (screenshots, LCP candidate, mobile usability), multi-window projects, the link visualiser, every Google integration (Search Console / GA4 / PageSpeed / CrUX) plus Yandex Metrica, bot-protection and soft-404 detection, LanguageTool spelling, AI per-URL prompts, SEO Authority providers, Google Sheets + BigQuery export, HTML and PDF reports, the Screaming Frog-compatible SEO Audit export, scheduled crawls with automatic exports and webhooks, encrypted projects, the MCP server with live crawl control, the standalone Log File Analyzer, dark and light themes, and an interface in 11 languages all ship today.

Cross-platform installers ship for Windows (setup + portable), macOS (Apple Silicon + Intel), and Linux (.AppImage / .deb / .rpm), with Playwright Chromium bundled so JS rendering works on first launch. The canvas-rendered URL table stays smooth on 25K+ row crawls, streams the first row in about 1 s, and long crawls are tuned for Apple Silicon (see docs).

After 1.0: updates continue based on user feedback and bug reports. Still on the list: a plugin system, code-signing and notarization, and in-app auto-update.

🤝 Contributing & Support


🐛 Found a bug?	Open an issue
💡 Have a feature idea?	Start a discussion
📦 Want the prebuilt app?	Download a release
🌐 Project website	freecrawl.net
❤️ Like the tool?	Support development on Patreon

📜 License

MIT, see LICENSE

_{Built with ❤ for SEO professionals who want a fast, free, open alternative to Screaming Frog.}