# Changelog
## [1.1.0] — 2026-10-18
### Added
- Adaptive speed (off by default) slows the crawl automatically on 429 / 503 / Retry-After responses and recovers once the server answers cleanly again.
- Bot-protection detection marks pages blocked by Cloudflare, AWS WAF, Vercel, Sucuri or DataDome as "Blocked" instead of reporting them as broken links or 4xx errors.
- A site-level soft-404 probe warns when missing pages return 200, and a Soft 404 issue flags "not found" pages served with a 2xx status.
- New benchmark-driven checks (hreflang outside head, HTTP/2, page size, dead external domains, CORS and more) bring the catalogue to 181 checks (thanks @alvinycheung).
- The status bar shows every post-crawl phase, such as measuring images or running checks, with live counts.
- srcset and picture image variants are listed without being fetched unless the new Spider setting or the CLI --image-variants flag turns them on.
- Images and other resources now show their Inlinks in URL Details.
- CLI: --adaptive-speed and --image-variants flags, Yandex Metrica commands (metrica-counters, metrica-fetch) and support for desktop-saved projects.
### Changed
- Stored page bodies are zstd-compressed with a per-project dictionary, shrinking working project files several times over on template-heavy sites.
- The Linux AppImage now starts on Ubuntu 24.04+ by disabling Chromium's sandbox only where the kernel denies unprivileged user namespaces; the .deb and .rpm keep the sandbox.
- README and the update dialog explain the macOS 15 Sequoia Gatekeeper flow (Open Anyway); the CLI now requires Node 22.13 or newer.
- README refreshed with the current feature set, counts and a new live-crawl demo GIF.
### Fixed
- Lazy-loaded images, tracking pixels, aria-labelled links, duplicate alt text and title/meta lengths measured in code points no longer produce false alarms.
- External links are re-checked with GET and a browser User-Agent before being reported broken.
- Sentence counting no longer stalls on long unpunctuated pages (thanks @alvinycheung).
- Cookie checks show "Off" instead of 0 when all cookies are rejected, crawls mark the project as unsaved, and keyboard hints show the Command key on macOS.
## [1.0.1] — 2026-10-13
### Added
- Yandex Metrica integration with unsampled pageviews, visits, users and search visits per crawled URL (thanks @danzerzine).
- Metrica data has its own tab with filters and exports, appears in URL Details and the orphan report, and is available over MCP.
### Fixed
- Clicking a row in the Broken Links, GA4, Search Console, PageSpeed, CrUX, AI and SEO Authority tabs now opens it in URL Details.
- Anchor text and target=_blank checks now catch links to pages the menu already links to, instead of reporting 0.
- MCP query and report tools now return Search Console and GA4 data instead of empty results.
- An expired Google sign-in now explains the one-time fix for the 7-day token limit instead of asking you to reconnect weekly.
- Smaller translation fixes in the setup guides.
## [1.0.0] — 2026-09-21
### Added
- FreeCrawl 1.0.0 is the full release; updates will keep coming based on user feedback.
- The interface is available in 11 languages, from Turkish and Spanish to Chinese, Korean and Hindi, and follows your system language on first launch.
- Light theme, switchable from View → Theme or Settings → Theme, with every table, badge and banner tuned for contrast.
- Canvas-rendered URL table that keeps scrolling smooth and live updates flicker-free on crawls of 25K+ URLs.
- PDF report export with a brandable header (name, logo, accent colour) and a recommendations section.
- Screaming Frog-compatible SEO Audit export so existing audit templates and Looker sources work without remapping.
- Scheduled crawls can export bulk, audit, HTML and PDF reports automatically and notify a webhook when they finish.
- Custom Extraction rules get their own table columns, plus ready-made templates for price, stock, author, dates, ratings and SKU.
- Optional crawl setting to follow JavaScript redirects (meta refresh and inline location changes).
- Canonical chains are followed from redirect targets, with a new "Redirect → Canonical Chain" issue.
- New checks: Content Wider Than Screen, Mobile / Desktop Mismatch, and structured-data findings that jump to the offending JSON-LD block.
- Settings → Issues lets you disable any of the 172 checks per project, and Settings → System shows live memory use and environment details.
- Auto-save the open project every N crawled URLs (Settings → Storage).
- New E-commerce, News and Accessibility-focus crawl presets.
- "Pages Over Max Links per Page" report, and Top Words can now include body text.
### Changed
- Release files are named by platform and CPU (Windows-Setup, Windows-Portable, macOS-Apple-Silicon, macOS-Intel, Linux) so the right download is obvious.
- Check for Updates now recognises an Apple Silicon Mac running the Intel build under Rosetta and offers the native build.
- Bulk Export gained the remaining datasets: redirect and canonical chains, hreflang, orphans, AMP, structured data, issue severities and link-level CSVs.
### Fixed
- Stop now responds immediately on large crawls instead of waiting for the post-crawl analysis to finish.
- Saving a large project no longer freezes the window; the title shows "Saving…" while it runs in the background.
- Sidebar issue counts and table queries no longer stall the app on 60K+ URL crawls.
- The table no longer goes black while scrolling during a live crawl.
- The duplicate URL check now respects the "normalise before comparing" setting.
## [0.9.9] — 2026-09-09
### Added
- In-app AI Assistant tab — chat over your crawl data with OpenAI, Claude, or Ollama, and connect external MCP servers to add their tools.
- Crawl setting to skip URLs with query strings, plus a parameter whitelist, so faceted navigation no longer eats the whole URL budget.
- SPA route discovery for JavaScript rendering — hash routes and History-API navigations are now crawled as their own pages.
### Changed
- The Settings dialog and every table column tooltip are now fully translated into Turkish.
- The Log Analyzer URL Hits tab gained a response-code filter, with hover explanations for each threat category and the threat score.
### Fixed
- X-Robots-Tag directives are now scoped to the correct bot, so a rule meant for one crawler no longer leaks onto the others.
- robots.txt parsing is capped at 512 KiB to stay fast and within spec on oversized files.
## [0.9.8] — 2026-08-27
### Added
- New "Suspicious Requests" tab in the Log Analyzer flags SQL-injection, XSS, and scanner probes in your access logs with the sender IP and the decoded payload.
- Export (CSV/Excel) buttons on every integration tab, and all integration tables added to the "Export Crawl Data" dialog.
- Optional adaptive crawl speed that automatically slows down when a site returns 429 / Retry-After and speeds back up once it recovers.
- Richer `debug.txt` diagnostics so a remote user's issue can be pinpointed from a single file.
### Changed
- Removed the Privacy settings page — FreeCrawl keeps no telemetry, so the opt-in served no purpose.
### Fixed
- Fixed the detail panel showing no Inlinks/Outlinks for a selected URL.
- Fixed crawl freezes and sidebar filters hanging on large sites, caused by slow issue queries.
- Fixed the crawl appearing stuck during the post-crawl image probe by pacing it and reporting progress.
- Fixed the Copy button in the Logs window.
- Fixed the Log Analyzer showing old sample data after a fresh crawl.
- Fixed the red "—" status badge shown for URLs blocked by robots.txt.
- Various performance, stability, and UI refinements.
## [0.9.7] — 2026-08-23
### Added
- Images tab now lists each image per page it appears on, with a Page column, so Missing/Empty Alt is accurate for every page.
- Opt-in "Allow invalid HTTPS certificate" for browser form-login — off by default so credentials aren't typed into an unverified page.
### Changed
- Crawl credentials (auth, proxy, custom headers) are now encrypted at rest and stripped from shared settings exports.
- Exports now honour the active search and advanced filter, so the file matches exactly what the table shows.
- robots.txt is now evaluated per origin, so cross-protocol and subdomain URLs are checked correctly.
- URLs blocked by robots.txt are now recorded, so the "Blocked by robots.txt" count is no longer always zero.
### Fixed
- Fixed the Overview panel staying blank in RAM-only storage mode.
- Fixed several crashes, freezes, and out-of-memory cases on large crawls, exports, and log imports.
- Fixed quitting with unsaved changes (Ctrl/Cmd+Q) wiping the crawl.
- Fixed tabs falsely reporting "no broken links / no images" and detail panels stuck on "Loading…" when a query failed.
- Fixed relative links and canonicals resolving wrongly when a page declares a `` tag.
- Fixed robots.txt user-agent matching, X-Robots-Tag per-bot rules, and `noindex, follow` pages not being crawled through.
- Fixed sitemap comparisons, favicon/thin-content flags on redirects, and the non-ASCII and uppercase URL checks.
- Fixed Missing/Empty Alt being judged site-wide instead of per page, and external links reporting a redirect's final status.
- Fixed crawl-control buttons and destructive actions failing silently instead of reporting the error.
- Hardened against credential leaks in logs and exports, CSV formula injection, path traversal, and untrusted-content navigation.
- Various stability, correctness, and UI refinements.
## [0.9.6] — 2026-08-20
### Added
- Four new canonical checks — Conflicting Canonicals, Canonical → Cross-Domain, Noindex + Canonical Conflict, and Redirect → Noindex.
- A Crawl Trap filter that flags faceted-navigation, session-ID, calendar, and repeating-path URLs which quietly drain crawl budget.
- Optional URL normalisation settings to sort query parameters and collapse duplicate slashes, removing a common source of false duplicates.
### Fixed
- The Log Analyzer now reads gzipped `.gz` access logs, which previously imported as zero lines.
- Relative canonicals such as `href="/page"` are now resolved before comparison, so self-referencing pages are no longer reported as canonicalised away.
- Gzipped sitemaps are now read instead of skipped, and oversized or mislabelled sitemaps are reported as warnings.
- URLs that differ only by percent-escape casing (`%2f` versus `%2F`) no longer appear as two separate rows.
- Crawls no longer run away on link loops like `/shop/shop/shop/`, and any skipped URLs are reported when the crawl finishes.
- Long post-crawl passes on large projects no longer hit a locked database or abort crawl teardown.
## [0.9.5] — 2026-08-15
### Added
- Ready-to-use GitHub Actions and GitLab CI templates that run a crawl as a build gate, with a setup guide.
### Changed
- "Delete Domain Data" now completes much faster on large single-domain projects.
### Fixed
- Opening a project crawled from the command line, or saved by an older build, now restores the start URL in the toolbar instead of coming up blank.
- The table's search, sort, and advanced filter now stay set when you leave a tab and return, instead of resetting to empty.
- The Overview sidebar no longer reports Response Codes and Security percentages above 100%.
## [0.9.4] — 2026-08-13
### Added
- Several automated agents can now share one crawler safely, each working in its own isolated session with a crawl queue.
- The Advanced table filter can now match on the values captured by your custom extraction rules.
### Changed
- Custom extraction values now read as plain text in the results table instead of raw JSON, matching the extraction preview.
- Sitemap and List crawls now stay within the supplied URL set instead of also following on-page links and fetching every asset.
### Fixed
- Redirect and error pages are no longer reported as missing title, meta description, H1, and other on-page elements they can never have.
- Search Console data now attaches to a crawled page even when the two URLs differ only by a trailing slash or letter case.
- Opening a saved project by double-clicking it now restores the start URL and mode in the toolbar instead of coming up blank.
- The results table no longer briefly draws two rows over each other while a crawl is running.
- Consistent toolbar strip across every detail-panel sub-tab, plus various interface refinements.
## [0.9.3] — 2026-08-08
### Added
- Connect several Google Search Console and Analytics accounts, and pick which one each project reports on from the tab toolbar.
- Every integration now has its own settings page, so one long shared page no longer buries the option you are looking for.
- Search Console gains a date range, device / search type / country / query filters, and twelve ready-made URL filter presets.
- A "Crawl New URLs Discovered In Google Search Console" option that pulls in pages Search Console knows about but your crawl never reached.
- Import several server log files at once, both in the app and from the command line.
### Changed
- Integrations is now a collapsible heading in the settings sidebar rather than a page of its own.
### Fixed
- Resuming an interrupted Sitemap or List crawl now genuinely carries on instead of re-fetching every page it had already done.
- Resuming a crawl no longer risks clearing the project when the sitemap changed while the app was closed.
- The toolbar now shows the mode and start URL of a resumed crawl, and of a project you open, instead of resetting to an empty Spider.
- Screenshots are now captured in unsaved projects too — the Screenshot tab used to stay empty unless you had saved the project first.
- Saving a project now takes its screenshots along, so they still appear after Save As and when the project is reopened.
- Robots directives are now read from every robots meta tag on a page, including ones written in uppercase.
- Google sign-in now explains a blocked consent screen and how to fix it, instead of failing with a generic timeout after five minutes.
- A timed-out database read no longer piles more work onto the busy worker, which could leave every table in the window waiting.
- Various improvements and refinements.
## [0.9.2] — 2026-07-29
### Added
- A Crawl and Store switch for every link type — images, CSS, JavaScript, media, canonicals, pagination, hreflang, AMP, iframes — so you control exactly what is fetched and what is reported.
- Right-click any table selection to copy cells, rows, or columns to the clipboard.
- A Mobile Alternate column that surfaces separate-URL (m-dot) mobile pages, including the classic setup with no canonical pointing back.
- A change marker on rows that differ from the previous crawl, so a re-crawl shows at a glance what moved.
- An optional "Respect robots.txt Crawl-delay" setting for teams whose policy requires it.
### Changed
- robots.txt `Crawl-delay` is now ignored by default, matching Google and Screaming Frog — sites that publish it no longer turn a 40-second crawl into hours.
- Pages are now crawled ahead of images, CSS, and JavaScript, so the table fills with real pages from the first seconds instead of assets.
- Assets can no longer consume a whole Max URLs budget, so a capped crawl returns pages rather than one page and its images.
- A robots.txt `Crawl-delay` is now reported in the log whether it is honoured or ignored, so a slow crawl is never unexplained.
### Fixed
- Fixed Clear taking around 17 seconds on a large project; it now finishes in under a second.
## [0.9.1] — 2026-07-23
### Added
- Offline spell checking for languages LanguageTool can't cover — Turkish ships with a bundled dictionary, so its pages get real findings instead of a warning.
- Each project now saves as a single compressed file, so moving or emailing one means one file instead of three.
- Any sub-table in the URL Details panel (Inlinks, Outlinks, Images, Resources) can now be exported on its own to CSV or Excel.
- JavaScript rendering now provisions its own Chromium automatically — bundled in the installer, or downloaded in the background when the bundle can't run on your machine.
- Turkish tooltips for the on-screen ⓘ help text across table columns and settings.
### Changed
- Spelling & grammar now detects each page's language locally before checking, so Turkish and other pages are no longer graded against the wrong language and flagged word-by-word.
- Pages in a language LanguageTool has no dictionary for are now marked "unsupported" instead of being scored against a substitute language.
- The "Add API Key" and Spelling "Options" buttons now open Settings directly on the right panel and highlight the field to fill in.
- Applying a crawl preset now shows a clear confirmation so it's obvious the settings changed.
## [0.9.0] — 2026-07-21
### Added
- The link visualiser now opens in WebGL 3D — a force-directed crawl tree and link graph you can orbit and fly through, matching how large sites are usually explored.
- Crawl Tree and Directory Tree stay 2D and gained click-to-collapse branches, so a 100K-URL site can be folded down to the part you care about.
- Hovering any node now shows a summary card with title, response code, indexability, H1, depth, inlinks, outlinks, and word count instead of just the URL.
- Long crawls on macOS now hold a power assertion so App Nap can't throttle them while the window is in the background.
- Crawl concurrency now scales down automatically as macOS reports rising thermal pressure, so fanless Macs degrade gracefully instead of stalling.
### Changed
- macOS keyboard shortcuts now match Windows — Cmd+C and the rest of the edit shortcuts work everywhere in the app.
- Encrypted project save and open now stream, so protecting a multi-gigabyte project no longer depends on it fitting in memory.
- Exports that must build the whole file in memory now check available headroom first and suggest a streaming format instead of failing late.
- The Top Anchor Texts panel is now a ranked list with usage bars, replacing the hard-to-read word cloud.
### Fixed
- Fixed a crash that could end a long crawl on macOS during the post-crawl phase by running out of memory.
- Fixed sidebar filter counts that didn't match the rows you got when clicking them, across Response Codes, Security, and Indexability.
- Fixed word count including JavaScript and CSS from the page, which inflated it many times over and broke the thin-content, empty-page, and readability checks.
- Fixed boilerplate detection reporting every sampled page as fully templated regardless of the site.
- Various visualisation and performance refinements.
## [0.8.9] — 2026-07-20
### Added
- New Sitemap crawl mode plus an optional sitemap seed for spider crawls — crawl exactly what a sitemap lists, or feed one in for faster discovery and orphan detection.
- A Desktop/Mobile toggle in the toolbar that crawls a site's mobile version (mobile User-Agent + viewport), useful when the mobile page renders differently.
- New "Duplicate Alt Text" image check that flags distinct images sharing the same alt text.
### Changed
- The crawl mode selector (Spider / List / Sitemap) now sits in the toolbar beside the URL bar, with an inline editor for the List URL set.
- The app now always opens in Spider mode with an empty URL bar.
- Various small UI improvements across the Search Console and Images tabs.
### Fixed
- The "Empty Alt" image issue now counts images (not pages) and its click-through filter works, so the sidebar number matches what you see.
## [0.8.8] — 2026-07-18
### Added
- New Video and News XML sitemap variants — build a Google News sitemap or a video sitemap for the crawled site.
- The Generate Sitemap menu is now a submenu exposing every variant (Standard, Images, Hreflang, News, Video).
- New mobile-usability checks that flag pages with text or tap targets too small to use comfortably on phones (JavaScript rendering only).
- A system tray icon with quick show/hide, stop-crawl, and quit actions.
## [0.8.7] — 2026-07-14
### Added
- Multi-window support — open several projects at once, each in its own window with an independent crawl, logs, settings, and enrichment runs.
- Projects manager dialog to browse, search, tag, and archive recent projects, and start new ones from a template.
- Project templates (E-commerce, Blog, News, Corporate, SaaS) that pre-fill sensible exclude rules and crawl budgets.
- Each project file now remembers its own crawl configuration.
### Changed
- Various improvements and enhancements.
## [0.8.5] — 2026-07-06
### Added
- Chrome UX Report tab — real-user Core Web Vitals (p75 LCP, INP, CLS, FCP, TTFB) per page for phone and desktop.
- Spelling and grammar checking via LanguageTool, with a custom dictionary, picky rule level, and in-context highlighting of every finding.
- A "Spelling/Grammar Issues" sidebar filter that lists pages with at least one finding.
- Five new MCP tools so agents can run and read CrUX and spelling checks.
### Changed
- Various improvements and enhancements.
## [0.8.2] — 2026-07-06
### Added
- Internal PageRank "Link Score" (0–100) per page — a sortable column, a detail-panel row, and a "By Link Score" graph colour mode showing where internal link equity concentrates.
- RAM-only storage mode — run a crawl entirely in memory for maximum speed, writing to disk only when you Save As (Settings → Storage).
### Changed
- Cross-source orphan pages (sitemap + Search Console + Analytics) now rank by traffic and are available as an MCP report tool.
- Various performance and reliability improvements.
## [0.8.1] — 2026-07-02
### Added
- Full Turkish and English localization across right-click menus, native dialogs, and settings panels.
### Changed
- The requests-per-second limit and robots.txt crawl-delay are now honoured during crawls.
- Various performance, reliability, and UI polish improvements.
### Fixed
- Crawls now keep your configured proxy instead of silently dropping it after the first page.
- Links with encoded characters are now fetched as the correct URL instead of a broken duplicate.
- Legacy-encoded pages (e.g. older Turkish sites) no longer render as garbled text.
- Country-code domains like .com.tr no longer let broad-scope crawls escape onto unrelated sites.
- Image usage counts no longer inflate when the same pages are re-crawled.
- External integrations no longer time out after running a crawl in the same session.
- Large in-page link and report tables no longer freeze the interface.
## [0.8.0] — 2026-06-29
### Added
- Three new site-graph layouts — Force-Directed Tree, Radial Tree, and Directory Tree — for clearer structural views of the crawl.
- Crawl Path Report — click any page in the graph to trace its shortest discovery path from the homepage.
- A "By LCP" graph colour overlay that flags pages whose largest above-the-fold element is an image, a prime optimisation target.
- Browser-driven login mode that drives a real Chromium through JS-heavy SPA login forms, then reuses the session for the crawl.
- XPath parent, ancestor, and sibling axes in custom extraction, for pulling values plain CSS selectors can't reach.
- A new MCP `get_crawl_path` tool so AI agents can fetch any page's crawl path.
### Changed
- HTTP Digest auth now caches each host's challenge and authenticates proactively, avoiding a 401 round-trip on every URL.
### Fixed
- Importing custom-extraction rules no longer silently drops XPath and JSONPath rules.
## [0.7.3] — 2026-06-26
### Added
- Log File Analyzer — a standalone window that parses Apache / Nginx / IIS / custom access logs into bot hits per URL, crawl budget, response-code distribution, and a daily trend.
- Crawl × log join that surfaces URLs search-engine bots hit but the crawl never reached, with one click to seed them into a running crawl.
- Bot detection across 40+ crawlers (Googlebot, Bingbot, YandexBot, GPTBot, …) with optional reverse-DNS verification to catch spoofed user-agents.
- A per-bot filter on the URL Hits table to see exactly which pages any one bot crawled.
- One-click export of every Log Analyzer table to a single CSV or a multi-sheet Excel workbook.
- A headless `analyze-logs` CLI command to ingest access logs without opening the app.
- New MCP tools to run a bulk export and to save / open (encrypted) project snapshots from an AI agent.
## [0.7.2] — 2026-06-23
### Added
- The Internal tab now lists images, CSS, JavaScript, and fonts — each with its own status code, type, and size — not just HTML pages.
- New "Check Images / CSS / JavaScript" toggles let you include or exclude page resources from the crawl, mirroring Screaming Frog.
- Broken external resources (third-party scripts, stylesheets, images) and external HTML pages are now probed and classified, so dead third-party assets surface in Broken Links.
## [0.7.1] — 2026-06-21
### Added
- SOCKS5 / SOCKS4 proxy support (including the DNS-at-proxy `socks5h` / `socks4a` variants), so crawls can route through Tor or an SSH tunnel — not just HTTP proxies.
- HTTP Digest authentication alongside Basic and Bearer, for crawling staging sites and APIs protected by Digest auth.
- Form-based multi-step login that establishes an authenticated session (with CSRF capture) before the crawl so member-only pages are reachable.
- XPath custom-extraction rules for pulling values off the page with XPath 1.0 location paths and predicates.
- JSONPath custom-extraction rules for pulling values straight out of JSON API responses, with live preview.
- PDF metadata extraction that reads each crawled PDF's title, author, page count, creation date, and producer into the detail panel.
- Issue severity tiers (Critical / Warning / Info) with a "split bulk export by severity" option that writes one rollup file or sheet per tier.
- Start, watch, and cancel batch PageSpeed / AI / SEO-authority runs from connected AI agents over the MCP server.
- List and fetch Google Search Console and Analytics 4 data directly from connected AI agents over the MCP server.
### Fixed
- Batch PageSpeed audits no longer fail under concurrency — fixed the timeout that killed every audit but the first when running many URLs.
- PageSpeed audits now honor corporate HTTP/HTTPS proxy settings, so they work behind a proxy just like the crawler.
- A proxy change between crawls in the same session now takes effect instead of the first crawl's proxy sticking for the whole session.
## [0.7.0] — 2026-06-18
### Added
- Performance budget — set per-page response-time / page-size / LCP / CLS ceilings and flag every page that exceeds them under a new issue filter.
- Structured-data validation now warns on missing Google-recommended properties (not just required ones) and covers 17 schema types including LocalBusiness, JobPosting, Review, and Course.
## [0.6.9] — 2026-06-14
### Added
- Social image aspect-ratio validation — flags OG / Twitter images too small or wrongly shaped to render as a proper share card.
- Core Web Vitals INP, TTI, and Max Potential FID in PageSpeed audits, with INP shown alongside LCP and CLS as a Core Web Vital.
- Accessibility audit (opt-in, JS render) — flags low colour-contrast text (WCAG AA) and stylesheets that suppress the keyboard focus outline.
### Fixed
- macOS: added Edit and Window menus so copy / paste and ⌘M work, and cleared a ⌘R shortcut conflict between Reload and Reports.
## [0.6.8] — 2026-06-04
### Added
- Canonical Conflict (Near-Duplicate) issue filter — flags pages clustered as near-duplicates whose declared canonical URLs disagree, a classic CMS localisation / pagination drift bug.
- Query String Variant Grouping report — surfaces base URLs that share many tracker / session-id / faceted-nav variants so you can spot crawl bloat from un-stripped query parameters.
- Sitemap Priority Mismatch report — high-priority sitemap URLs whose actual crawl outcome contradicts the declared importance (4xx, noindex, canonicalised, redirect, blocked, never reached).
- Content area selector — Settings → Duplicates accepts a CSS selector that pins the duplicate fingerprint to a specific page region, overriding the heuristic.
- Template / Boilerplate Detection — post-crawl sampling pass identifies repeated chrome (nav, footer, sidebar) across pages and flags pages whose unique 5-word shingles are over 50% boilerplate.
## [0.6.7] — 2026-06-01
### Added
- MCP server action tools — 29 new actions covering every export format, sitemap generate / validate, robots tester, project compare, schedule, respider, remove, settings, URL rewrite + extraction previews, and graph snapshot.
- MCP integration triggers — `gsc_fetch`, `ga4_fetch`, `gsc_list_sites`, `ga4_list_properties`, and `google_auth_status` let agents pull Search Console and Analytics data on demand.
- In-app Integration Setup Guide modal — every integration card now opens a localised step-by-step walkthrough with deep links to the relevant Google Cloud Console pages and troubleshooting for the most common OAuth and API errors.
### Changed
- README Highlights rewritten as flowing paragraphs that reflect what the current release can actually do — shorter, less wall-of-bullets.
## [0.6.6] — 2026-06-01
### Added
- Cluster view on the Duplicates tab — surfaces every near-duplicate cluster with grouped headers and inline members.
- Column pin (sticky left) — pin frequently-referenced columns so they stay visible during horizontal scroll.
- Column reorder via drag-drop — grab the grip handle on a header to rearrange columns within their pin region.
- AMP smoke validation — pages declaring themselves AMP now flag missing boilerplate, runtime, canonical, and forbidden tags.
- Extraction Preview button in Settings → Custom Extraction — test selectors / regex against a real URL before committing to a crawl.
- Custom Extraction rule import / export — versioned JSON envelope, shareable as a file or via source control.
- MCP server read parity — 33 new tools covering every detail sub-tab, integration row, report, and specialised query.
- MCP `clear_crawl` tool — same primitive as the desktop "Clear" button so an agent can force a fresh BFS.
### Fixed
- Re-running a same-seed crawl via MCP no longer no-ops once the prior crawl already completed.
- Advanced filter clauses now reset on tab change so a filter set on one tab doesn't silently apply to every other tab.
- Column picker popover no longer hides behind the bottom detail panel — rendered through a portal at document.body.
## [0.6.5] — 2026-05-28
### Added
- JavaScript rendering via headless Chromium (Playwright) — post-JS DOM capture for SPAs and hydration-only sites, used for link extraction + extraction rules.
- View Rendered HTML detail sub-tab — compare raw HTTP response against post-JS DOM with side-by-side search and per-render timing badge.
- Screenshot capture during JS render — full-page, above-the-fold, and 375×667 mobile-viewport PNGs stored next to the project file with a dedicated detail panel sub-tab.
- LCP candidate detection — identifies the largest above-the-fold element after JS render and stores its CSS selector, dimensions, and resource URL.
- Mobile Usability audit — re-renders each page on a mobile viewport and checks viewport meta, horizontal overflow, font size, and tap-target spacing.
- Settings → Rendering panel — viewport, wait conditions, resource blocking, Chromium channel, pre-render JS snippet, screenshot mode, and advanced audit toggles.
- "Export Crawl Data…" dialog — single menu entry replacing the three "Export Current View as …" items, with Excel `.xlsx` / CSV-UTF-8 / JSON / XML formats and a hierarchical tree picker that lays out top tables and their structural sub-categories under nested folders.
- Per-tab quick-filter dropdown — Overview sub-categories accessible directly from each tab's toolbar without leaving the view.
- List ↔ Tree view toggle — switch between flat virtualized table and URL-path hierarchy with collapsible folders.
- Visualization moved to its own native window — accessible from the new "Visualization" top-level menu (Ctrl+G), free to park on a second monitor.
- Full Settings dialog Turkish translation coverage — sidebar sections, dialog chrome, and every field label across the 24 panels.
- Optional offline-ready installers — release builds now ship Playwright Chromium inside the installer so JavaScript rendering works on first launch with no extra download.
### Changed
- Overview sidebar clicks now auto-switch to the matching top-level tab (e.g. Response Codes → Client Error 4xx jumps to the Response Codes tab) for every structural group and 30+ issue filters.
- File menu "About FreeCrawl SEO" now opens https://freecrawl.net/ in the default browser instead of doing nothing.
- Browser pool startup checks the Playwright binary up-front and, if missing, surfaces a single download prompt instead of spamming a warning per URL.
### Fixed
- Mobile screenshot now actually renders on a 375×667 mobile viewport instead of producing a desktop above-the-fold duplicate.
- Pre-render JS snippet no longer accumulates across pooled pages — it is registered once at the browser context level so long crawls run the snippet exactly once per navigation.
- Stop button now cancels in-flight Playwright navigations immediately instead of waiting for the request timeout to elapse.
- Browser pool no longer crashes the crawler when closed while page acquires are queued — waiters are rejected with a clean error.
- Screenshot file IPC reader hardened against path-traversal attempts; only PNGs inside the active project's sidecar folder are returned.
- Body snapshot truncation respects UTF-8 boundaries so the View Source / View Rendered tail no longer ends in a replacement character.
- Render artefacts (rendered HTML, screenshots, LCP, mobile-usability) are now persisted inline before the URL is considered complete, eliminating a race window with the Clear button.
- CSV / XLSX exports now neutralise cells beginning with `=`, `+`, `-`, `@`, TAB, or CR to prevent formula-injection attacks via crawled page content.
- Tree view distinguishes root URLs that differ only by query string or fragment.
- "Export Crawl Data" single-file mode now rewrites the chosen output path's extension to match the selected format.
- Quick-filter dropdown shows a placeholder when the active category isn't in the predefined list instead of silently mislabelling itself as the first option.
## [0.6.0] — 2026-05-25
### Added
- Google Search Console integration — pull clicks / impressions / CTR / position per crawled page and overlay them on the URL list.
- Google Search Console URL Inspection — fetch coverage verdict, last-crawl time, and canonical decisions for batches of pages.
- Google Analytics 4 integration — join GA4 sessions / users / engagement / avg session duration onto each crawled URL.
- AI tab — run OpenAI, Anthropic, or local Ollama prompts per URL with shared variables ({url}/{title}/{description}/{h1}/{body}).
- SEO Authority tab — unified Ahrefs / Majestic / Moz / Semrush per-URL metrics behind a single provider dropdown.
- Google Sheets export — File menu writes the active category to a fresh Sheet and copies the link to clipboard.
- BigQuery export — service-account streaming insert into a timestamped table inside the `freecrawl_seo` dataset.
- Analytics sub-tab in the detail panel — combined GSC + GA4 + URL Inspection card view for the selected URL.
- Reports → "Orphan Pages (Cross-Source)" — surfaces URLs that PSI / GSC / GA4 / sitemap know about but the crawl never reached.
- Encrypted project snapshots — File menu now saves and opens AES-256-GCM-encrypted `.seoproject.enc` snapshots protected by a password.
- Images tab — Export button writes every image row to CSV honouring the active missing-alt filter and search box.
- CLI: ready-made GitHub Actions and GitLab CI templates that crawl a target on schedule and fail the pipeline on broken-URL thresholds.
### Changed
- PageSpeed Insights now requires a free API key — Google's keyless shared quota is 0/day so the tab gates the Run button and shows an "Add API Key…" call-to-action when no key is configured.
- PageSpeed failure handling surfaces the exact Google error inline (QUOTA / RATE / TIMEOUT / NET tag plus a hover-tooltip), and the batch log lists the top distinct errors instead of just a failed count.
- Memory limit setting now actively pauses and auto-resumes the crawler when RSS crosses the configured threshold instead of being a display-only value.
### Fixed
- Ahrefs integration — the v3 combined `/site-explorer/overview` endpoint was removed by Ahrefs, replaced with a parallel fan-out across the four per-metric endpoints with partial-success tolerance.
- Semrush integration — `type=url_overview` was renamed to `type=url_rank` and the now-mandatory `export_columns` parameter is supplied.
- Various crawler, UI, and integration stability fixes.
## [0.5.6] — 2026-05-22
### Added
- Google PageSpeed Insights integration — a new PageSpeed tab runs on-demand Lighthouse audits (mobile + desktop) for the crawled pages you select.
- Broken Links tab now also shows a "unique broken URLs" count — a stable figure that doesn't swing with how many pages a crawl reached.
### Fixed
- Re-crawling a URL no longer duplicates its link and image rows, which had been inflating broken-link and image counts.
- macOS — closing the main window no longer prevents reopening it from the dock.
- The Logs window no longer drags down the main window's frame rate while it is open.
- Bulk actions (delete / re-spider / copy) on very large URL selections no longer fail.
- Scheduled crawls no longer fire twice in a row when a crawl outlasts its interval.
- "Check for Updates" now downloads the installer instead of the portable build.
- Various crawler and UI stability fixes.
## [0.5.5] — 2026-05-19
### Added
- Integrations panel (Settings → Integrations) — securely store API keys and OAuth credentials for OpenAI, Anthropic, Ollama, PageSpeed, Ahrefs, Moz, Semrush, Majestic, and Google services.
- Rendering Mode setting — Text Only or Old AJAX Crawling Scheme (`_escaped_fragment_`); JavaScript rendering is reserved for V2.
- Quick User-Agent preset switcher (Settings → Requests) — one click between Googlebot, Chrome, and Bingbot profiles.
- Broken Links tab now has an Export button — writes every broken link to CSV, honouring the active internal/external filter.
- Broken Links tab now supports cell selection, drag-select, Ctrl+C copy, and a right-click menu (copy cells, copy/open source & target URLs).
### Changed
- Near-duplicate detection now fingerprints the main content area only — site-wide nav/header/footer no longer makes distinct pages look like duplicates.
## [0.5.2] — 2026-05-19
### Fixed
- Linux `.deb` / `.rpm` installer build — added `author.email` maintainer metadata, pinned the linux artifact filename (so the scoped package name no longer breaks the deb output path), and forced a space-free package name for rpm.
## [0.5.1] — 2026-05-19
### Fixed
- Linux installer build — added the `homepage` / `repository` metadata that electron-builder's fpm target needs to package `.deb` and `.rpm` artifacts.
### Changed
- README updated with multi-language UI, scheduled crawl, file association, portable `.exe`, Linux `.rpm`, and the macOS / Linux `.sh` launcher.
## [0.5.0] — 2026-05-19
### Added
- Multi-language UI (English + Turkish) — instant swap from Settings → Language, covers menus, sidebar, every tab, every dialog, and the detail panel.
- Settings → Storage panel — pick a default folder for new `.seoproject` files; falls back to the OS Documents folder when unset.
- Settings → Privacy panel — opt-in toggle for anonymous telemetry (off by default; no endpoint wired up yet, consent is persisted for future builds).
- Linux .rpm installer target alongside AppImage / deb.
- Windows portable .exe target alongside the NSIS installer — runs without installation.
- `.seoproject` file association — double-clicking a project file from the OS opens it in FreeCrawl (Windows / macOS / Linux).
- Custom CSS override — drop a `custom-theme.css` in the user-data folder to tweak colours, spacing, or typography without rebuilding the app.
## [0.4.1] — 2026-05-15
### Added
- In-app scheduled crawl — File → Scheduled Crawl… arms an hourly/daily/weekly/custom recurring crawl per project, fires while FreeCrawl stays open.
- Two new reports — Indexability Distribution and Content-Type Distribution histograms (Reports total 21 → 23).
- Linux/macOS launcher script (.sh) mirrors the existing Windows .bat launcher with Node/npm checks, dependency install prompt, and production start.
## [0.4.0] — 2026-05-15
### Added
- Visualization and SERP top-level tabs — the link graph and Google-style snippet preview moved out of a modal/sub-panel into permanent tabs.
- Column show/hide menu in every URL table — toolbar button, header right-click to hide, and per-tab persistence.
- Bulk Export added ~37 named CSVs across redirect, canonical, pagination, hreflang, duplicate, security, image, structured-data, AMP, and SERP families.
### Fixed
- MCP bridge silently failed to start because the address was read before the socket bound — Claude Code can now reliably drive desktop crawls again.
## [0.3.7] — 2026-05-13
### Added
- MCP crawl control — Claude Code and any MCP client can now start, pause, resume, and stop crawls in the running desktop app and poll live progress, in addition to the existing read-only data queries.
## [0.3.6] — 2026-05-12
### Added
- Top Words report — frequency table over the title + meta description + H1 corpus, with English/Turkish stopword filtering.
- Status-code diagnosis banner in the HTTP Headers panel — decodes Cloudflare 52x/530 and standard 4xx/5xx codes with header-derived signals (cf-ray, Retry-After, CDN/server detection) and crawl-context fixes.
- Request headers view in the HTTP Headers panel — toggle between captured response headers and the request headers reconstructed from the active crawl config.
### Changed
- Crawler settings relabelled to their Screaming Frog equivalents and the duplicated throughput inputs consolidated into the Speed section for clarity.
- Settings dialog gained a "Reset to Defaults" button.
## [0.3.5] — 2026-05-10
### Added
- Skipped Heading Level and Too Many Requests issue filters — surface accessibility outline gaps and pages with >100 subresources.
- Security and Duplicates top-level tabs — security-header audit and near-duplicate cluster review in dedicated views.
- Specialised columns on Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction/Search, and Links tabs — each surfaces its domain data instead of generic placeholders.
- Live memory monitor in the status bar — RSS, system free, per-URL cost, and capacity projection update every 2 seconds.
- Robots.txt syntax validation — the Robots Tester custom-policy editor lints typos, orphan rules, sitemap URLs, crawl-delay, and deprecated noindex inline.
### Changed
- Boolean flag cells render as "Y" / "—" instead of "true" / "false" for a more compact table.
### Fixed
- Duplicate "Check for Updates" popup — concurrent silent and manual checks now share a re-entrancy guard so only one dialog appears.
## [0.3.4] — 2026-05-09
### Added
- Accessibility issue filters — Main Landmark Missing, Skip Link Missing, and Invalid ARIA Role surface WCAG gaps.
- Critical performance tier — Render-Blocking Head >20 and Page Size >3MB highlight LCP regressions beyond the warning thresholds.
- Pagination Canonical Conflict filter — flags pages declaring both rel=next/prev and a non-self canonical.
- Structured-data validation — Duplicate `@id`, Malformed `@type`, and Missing Required Property catch JSON-LD bugs that hurt rich-result eligibility.
- Seven new top-level filter tabs — Pagination, Hreflang, AMP, Structured Data, Meta Refresh, Custom Extraction, Custom Search.
- View Source match navigation — Enter / Shift+Enter cycle through hits, ↑/↓ buttons, and the active match auto-scrolls into view.
### Changed
- Clear button is ~4-5× faster (~2 s → ~400 ms) by wrapping the reset in a single SQLite transaction and truncating the WAL afterwards.
- Various improvements and refinements.
### Fixed
- Bot-blocked pages with no URL extension now stay visible in the Internal tab instead of disappearing into the Other content kind.
## [0.3.3] — 2026-05-04
### Added
- Web App Manifest parsing — pages’ PWA theme_color, short_name, display, scope, and number of icons can now be inspected.
- TLS certificate chain view — CA hops from leaf to root are listed in the URL Details panel.
- “Duplicate URL normalize” toggle — under Settings → Duplicates, you can choose between byte-exact vs. normalized comparison.
- A Security Policy (SECURITY.md) has been added to the repository.
### Changed
- Various improvements and enhancements.
## [0.3.2] — 2026-05-04
### Added
- **In-app installer download** in the "Update Available" dialog — when GitHub Releases lists a matching artifact for the current platform/arch (Windows `.exe`, macOS Intel `.dmg`, macOS ARM64 `-arm64.dmg`), the dialog now offers a "Download Installer" button alongside "Open Release Page". Saves to the user's Downloads folder via `session.defaultSession.downloadURL` with taskbar/dock progress, then reveals the file with `shell.showItemInFolder`. Linux falls back to the release page (no single canonical artifact). Code-signing-free path — the user installs manually after Gatekeeper / SmartScreen prompt.
- **Markdown release notes** rendered as plain text — `stripMarkdownToPlain` collapses code fences, bold / italic, headings, links (`[text](url)` → `text (url)`), images, bullets, and strikethrough. Preview window expanded from 600 to 1500 chars.
- **"Don't show this version again"** checkbox in the Update Available dialog — persists in the existing `skipDiag:update-${tag}` prefs namespace, so Help → "Reset Diagnostic Warnings" already clears it. Silent startup auto-check (8 s after launch) respects dismissals; manual menu invocation always shows.
- **`srcset` + `` support for images** — html-parser now captures every candidate URL from `` and ` > ` (WHATWG-compatible split), feeding them into the same broken-image / large-image probe pipeline so retina + breakpoint variants no longer slip through. Per-page `images_responsive` (count of `` slots that carry `srcset` or sit inside a `` parent) and `picture_count` columns (migration v49). New issue filter **Low Responsive Image Adoption** under Sidebar → Issues → Images, mirroring the lazy-loading rule's ≥5-image / <50%-adoption threshold. Detail Panel surfaces "Responsive Images" ratio + "`` Elements" count rows.
- **Malformed URL detection** — `isUrlMalformed` helper in `url-utils.ts` flags URLs that browsers tolerate but search engines / log analysers diverge on: multiple `?` or `#`, ASCII control chars (<0x20 / 0x7F), unescaped reserved chars (`<>"`{}|\^`), and double-encoding sequences (`%2520` etc.). The crawler sets a per-URL `url_malformed` 0/1 flag at every upsert path (HTML, redirect, non-HTML, size-cap, network-fail, seed-stub). New **Malformed URL** issue filter under Sidebar → Issues → URL.
- **OpenGraph extras** — `og:type` (lowercased), `og:url` (verbatim raw, useful for canonical comparison), `og:site_name` (entity-decoded), and `og:locale` are now stored as separate columns and shown in the URL Details panel.
- **Android / PWA icon detection** — first `` whose declared dimension is ≥192px (or `sizes="any"` for vectors) is captured as `android_icon` — the right adoption signal for home-screen / PWA launchers, distinct from the smaller favicon. Surfaced in the URL Details panel as "Android / PWA Icon".
- **Database migrations v49 + v50** — bundled column additions: `images_responsive`, `picture_count` (v49); `url_malformed`, `og_type`, `og_url`, `og_site_name`, `og_locale`, `android_icon` (v50).
### Changed
- **Update Available dialog** is now a 3-button dialog (`Download Installer / Open Release Page / Later`) on platforms with a matching installer asset, falling back to the 2-button form (`Open Release Page / Later`) elsewhere. Silent startup check skips network errors, up-to-date results, and dismissed versions so it isn't intrusive.
## [0.3.1] — 2026-05-04
### Fixed
- **`Queue error: database is locked` / `External probe error: database is locked`** during crawl — root cause was two SQLite writer connections (main-thread `ProjectDb` + writer-worker `ProjectDb`) contending for the same file's writer lock. With `busy_timeout=10s` set on both, the periodic `recomputeUrlsIssuesYielding` pass (every 30 s, 70+ short transactions with `setImmediate` yields) could starve the main thread out of the lock window and surface as a visible error. Fixed by routing the crawler's high-frequency main-thread sync writes (`upsertUrl` for redirect / non-HTML / size-cap / network-fail rows, `updateExternalProbe`, `setUrlHeaders`, `setSitemapUrls`, `checkpointQueue`) through the same writer-worker pool that already serves `writeFetchedUrl`. Now the DB sees a single writer connection during a crawl — SQLite's lock contention is gone and serialisation happens at the worker's JS-level FIFO instead. New `dbCall(method, args)` opt callback on `Crawler` (default = sync `this.db[method](...args)`, so CLI / tests are unchanged); desktop main injects a `dbCallViaPool` dispatcher with main-thread fallback.
## [0.3.0] — 2026-05-04
### Added
- **Manual update check** — Help → "Check for Updates…" menu item. Fetches the latest GitHub Releases tag (`api.github.com/repos/kemalai/FreeCrawl-SEO-Tool/releases/latest`, 10 s timeout) and compares it against `app.getVersion()` with a hand-rolled `compareSemver` (no `semver` dependency). Three native dialogs: up-to-date / update available with first-600-char release-notes preview + "Open Release Page" button / network error fallback with "Open Releases Page" link. Pure on-demand — no background polling, no telemetry. `electron-updater`-based auto-install deferred to V2 (requires macOS notarisation + Windows code-signing first).
## [0.2.9] — 2026-05-04
### Added
- **Readability scoring** (Flesch Reading Ease, Flesch–Kincaid Grade, Gunning Fog Index) — computed inline during HTML parse from the same body-text token stream that drives `wordCount`. Dictionary-free vowel-group syllable heuristic (Hemingway / textstat / Yoast convention). Returns `null` scores when the page has <50 words or 0 sentences so navigation-only landing pages don't show a meaningless 0.0 grade. New issue filters: **Hard to Read (Flesch <30)** and **Gunning Fog >17** under Sidebar → Issues → Content. Detail panel surfaces Sentence Count, Flesch with band ("very difficult" → "very easy"), F-K Grade, Fog Index, and Complex Words.
- **CORS header audit** — captures `Access-Control-Allow-Origin / -Credentials / -Methods / -Headers` per URL (1 KB cap on `-Headers`). Two new issue filters: **CORS Wildcard + Credentials** (CRITICAL — `*` paired with credentials true; XSS-leveraged credential exfiltration vector) and **CORS Wildcard Origin** (informational — `*` without credentials). Detail panel renders all four headers; credentials value as human-readable true/false/missing.
- **HTTP (not HTTPS) issue filter** — flags internal pages still served over plain HTTP. Excludes `localhost`, `127.0.0.1`, and `*.local` hosts so local-dev crawls don't trip the filter.
- **Active vs passive mixed content split** — separate counts and issue filters for **Mixed Content Active (Blocked)** (script/iframe/object/embed/stylesheet over HTTP — browsers silently drop these) and **Mixed Content Passive (Warning)** (img/video/audio/source over HTTP — rendered with "Not Secure" UI). Legacy `mixed_content_count` kept as the total for back-compat. `