Their pricing page now lists $279 / £199 / €245 per user per year, with
bulk bands below that from five seats up. The old figure was also across
librecrawl.com, which has been corrected separately.
The authorize request sent prompt=select_account, which Zoho doesn't
support (its only valid value is "consent"). Zoho showed a generic error
page instead of the sign-in screen. Drop the parameter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Setting ZOHO_OAUTH_ENABLED with a client ID and secret adds a "Login
with Zoho" button to the login page. Zoho users are matched by a new
users.zoho_id column. An existing account with the same email is linked
to Zoho. Otherwise a new verified account is created, unless
ZOHO_ALLOW_SIGNUP is off, which is the default when registration is
disabled.
Also configurable: redirect URI, data-center accounts server, scopes,
allowed email domains and the tier for new accounts. All are documented
in .env.example and passed through in docker-compose.yml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
External links that were only HEAD-checked (Crawl External off) now get
a link_check_only flag, saved with the crawl. The Overview leaves those
rows out, since those sites were not crawled; the External tab still
lists them. Off-site images and crawled pages are not flagged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Crawl External off, external <a> targets are now HEAD-checked like
images and get a synthesized row (status, type, size), so the External
tab and Links-tab statuses fill in without fetching the other site.
HEAD refusals (403/405/501) are confirmed with a streamed GET. Image
and external checks share one helper across both crawl paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Off-domain images were skipped unless Crawl External was on, so sites
serving every image from a CDN (Wix, Shopify, Squarespace) got no image
statuses or Images-tab rows. Embedded images are page assets: check them
wherever they're hosted. Crawl External still gates off-domain pages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
extract_images used img.get('alt', ''), so an image with no alt attribute
and one correctly marked decorative with alt="" both became the empty
string. The detector then tested truthiness, and correct markup counted
as a defect.
Images now carry has_alt alongside alt, and only a missing attribute is
flagged. The warning also says how many decorative images were found, so
the count is explainable rather than just smaller.
The Links tab had the same conflation, showing "(no alt text)" for both;
it now distinguishes "(decorative)" from "(no alt attribute)".
Crawls saved before has_alt existed carry no flag. Those are left
unflagged rather than guessed at, since the old data cannot tell the two
cases apart. Re-crawl for accurate numbers.
Thanks to musclemintco for the diagnosis, which was exactly right.
Duplicate-content detection compared every page with every other page using
SequenceMatcher and emitted two issues per matching pair. On a 1,127-URL site
with ~900 templated suburb pages that meant 4+ minutes of CPU, 703,902 issues,
and a 90-second SQLite insert that locked the database for every other
request (status polls, settings) — the UI sat in "finishing up" until the
crawl was force-stopped.
- issue_detector: one issue per page (count + closest five), identical
title/description/H1 grouped first, candidate pairs from a prefix filter on
title tokens, word-level Dice similarity, only 200 HTML pages considered,
and a 120s time budget so the pass can never hang a crawl.
900 templated pages: 1.4s / 900 issues (was ~4min / 703k).
- auth_db/crawl_db: 30s busy timeout and WAL mode so reads no longer fail
with "database is locked" while the crawler saves a batch.
Exports: "Excel (XLSX)" was offered in Settings but never implemented (it
silently produced CSV), XML only applied to the URLs file (links/issues fell
back to CSV), and the export button read the format from the server copy of
the settings, which silently lags the browser copy when the background sync
fails — e.g. while the database was locked by the bug above.
- main: XLSX (openpyxl) and XML for URLs, links and issues on both the
streaming and legacy export endpoints; shared row helpers.
- app.js/settings.js: export with the settings the browser is showing;
surface a failed server sync instead of swallowing it; decode base64
binary exports.
- tests: fixture tests for one-issue-per-page duplicate detection (crawl +
1,500-page scale check) and for every format × every data type on both
export endpoints.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#93: with JS rendering on, response_time was measured around the whole
render, which includes the js_wait_time sleep. At the 3s default every
page cleared the 3000ms threshold and got flagged "Slow Response Time".
Reported on a site scoring 97-99 on PSI with all 54 pages flagged.
render_page now reports the browser's own request timing (responseEnd,
falling back to the measured navigation) as response_time, and the full
render separately as render_time. On librecrawl.com with a 3s wait that
is 957ms vs 4287ms, and nothing is flagged. render_time is stored, shown
in the URL details modal, and added to crawled_urls via migration.
#94: "URLs Discovered" ignored the rows synthesized from image HEAD
checks, since those are never queued. The counter sat at the HTML page
count while crawled climbed past it. Image rows are now marked
discovered when their row is created.
Fixture tests cover both.
- Workshop button in the app header
- README and the plugins README point at it for finding/sharing plugins
The workshop is back up at workshop.librecrawl.com. On the marketing site
the /workshop redirect now goes there instead of to GitHub, and the nav
links point straight at the subdomain.
requests defaults to 10 connections per host, but crawl workers and the
image HEAD checks (5 threads of their own) draw from the same pool, so
urllib3 kept discarding live connections and reopening them:
WARNING:urllib3.connectionpool: Connection pool is full, discarding
connection: developer.mozilla.org
Mount an adapter sized to concurrency + IMAGE_CHECK_WORKERS + 5, and
resize it when concurrency changes in settings. An MDN crawl went from a
steady stream of these warnings to none.
Incremental polling sliced server arrays by the client's row count, so
updates to already-sent rows (linked_from, link statuses, image HEAD
results) never reached the UI, and the issue cursor desynced whenever an
exclusion pattern dropped one. Replaced with an event log (seq + epoch);
clients apply appends/updates by key.
- Relative links on redirected pages resolved against the requested URL,
minting internal 404s from the target site's links; chains now recorded
- Image HEAD checks ignored crawl_external
- Image rows counted against max_urls (25-URL react.dev crawl = 1 page)
- _save_batch_to_db cleared its queues after writing, dropping rows
appended mid-write (400-page Wikipedia crawl lost 3,693 links)
- Fetches in flight at max_urls hit the DB uncounted
- Sitemap discovery blocked start_crawl; now backgrounded, completion
waits on it
- start_crawl read session_id before creation, disabling persistence
- Link/issue cells used innerHTML on crawled page content
- Virtual scroller measures real row heights, rAF-throttled, spacers
track data length
Adds tests/: fixture tests (no network) + real-site harness.
Follow-up to #48:
- Issues export used nonexistent keys (issue_type/severity/description);
now matches the real issue dicts and legacy format (type/category/issue/details)
- Crawls loaded from a saved file exist only in browser state, which the
streaming endpoint can't see — exportData() now falls back to the legacy
POST /api/export_data path for those instead of downloading a 404 body
- target_status None renders as 'Not crawled' instead of blank (0 stays 0)
- Export field names from the query string are restricted to identifier-like
strings since they become XML tag names and CSV headers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to #87: full image downloads are now behind a 'Crawl Images'
checkbox (default off) instead of being in the default include_extensions.
When off, the Images tab is populated from the broken-image HEAD checks
that already run (status, content-type, size from Content-Length) at no
extra request cost. Also guards the issue detector against flagging
non-HTML results for missing titles/meta descriptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phiality