Their pricing page now lists $279 / £199 / €245 per user per year, with
bulk bands below that from five seats up. The old figure was also across
librecrawl.com, which has been corrected separately.
The authorize request sent prompt=select_account, which Zoho doesn't
support (its only valid value is "consent"). Zoho showed a generic error
page instead of the sign-in screen. Drop the parameter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Setting ZOHO_OAUTH_ENABLED with a client ID and secret adds a "Login
with Zoho" button to the login page. Zoho users are matched by a new
users.zoho_id column. An existing account with the same email is linked
to Zoho. Otherwise a new verified account is created, unless
ZOHO_ALLOW_SIGNUP is off, which is the default when registration is
disabled.
Also configurable: redirect URI, data-center accounts server, scopes,
allowed email domains and the tier for new accounts. All are documented
in .env.example and passed through in docker-compose.yml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
External links that were only HEAD-checked (Crawl External off) now get
a link_check_only flag, saved with the crawl. The Overview leaves those
rows out, since those sites were not crawled; the External tab still
lists them. Off-site images and crawled pages are not flagged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Crawl External off, external <a> targets are now HEAD-checked like
images and get a synthesized row (status, type, size), so the External
tab and Links-tab statuses fill in without fetching the other site.
HEAD refusals (403/405/501) are confirmed with a streamed GET. Image
and external checks share one helper across both crawl paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Off-domain images were skipped unless Crawl External was on, so sites
serving every image from a CDN (Wix, Shopify, Squarespace) got no image
statuses or Images-tab rows. Embedded images are page assets: check them
wherever they're hosted. Crawl External still gates off-domain pages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
extract_images used img.get('alt', ''), so an image with no alt attribute
and one correctly marked decorative with alt="" both became the empty
string. The detector then tested truthiness, and correct markup counted
as a defect.
Images now carry has_alt alongside alt, and only a missing attribute is
flagged. The warning also says how many decorative images were found, so
the count is explainable rather than just smaller.
The Links tab had the same conflation, showing "(no alt text)" for both;
it now distinguishes "(decorative)" from "(no alt attribute)".
Crawls saved before has_alt existed carry no flag. Those are left
unflagged rather than guessed at, since the old data cannot tell the two
cases apart. Re-crawl for accurate numbers.
Thanks to musclemintco for the diagnosis, which was exactly right.
Duplicate-content detection compared every page with every other page using
SequenceMatcher and emitted two issues per matching pair. On a 1,127-URL site
with ~900 templated suburb pages that meant 4+ minutes of CPU, 703,902 issues,
and a 90-second SQLite insert that locked the database for every other
request (status polls, settings) — the UI sat in "finishing up" until the
crawl was force-stopped.
- issue_detector: one issue per page (count + closest five), identical
title/description/H1 grouped first, candidate pairs from a prefix filter on
title tokens, word-level Dice similarity, only 200 HTML pages considered,
and a 120s time budget so the pass can never hang a crawl.
900 templated pages: 1.4s / 900 issues (was ~4min / 703k).
- auth_db/crawl_db: 30s busy timeout and WAL mode so reads no longer fail
with "database is locked" while the crawler saves a batch.
Exports: "Excel (XLSX)" was offered in Settings but never implemented (it
silently produced CSV), XML only applied to the URLs file (links/issues fell
back to CSV), and the export button read the format from the server copy of
the settings, which silently lags the browser copy when the background sync
fails — e.g. while the database was locked by the bug above.
- main: XLSX (openpyxl) and XML for URLs, links and issues on both the
streaming and legacy export endpoints; shared row helpers.
- app.js/settings.js: export with the settings the browser is showing;
surface a failed server sync instead of swallowing it; decode base64
binary exports.
- tests: fixture tests for one-issue-per-page duplicate detection (crawl +
1,500-page scale check) and for every format × every data type on both
export endpoints.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#93: with JS rendering on, response_time was measured around the whole
render, which includes the js_wait_time sleep. At the 3s default every
page cleared the 3000ms threshold and got flagged "Slow Response Time".
Reported on a site scoring 97-99 on PSI with all 54 pages flagged.
render_page now reports the browser's own request timing (responseEnd,
falling back to the measured navigation) as response_time, and the full
render separately as render_time. On librecrawl.com with a 3s wait that
is 957ms vs 4287ms, and nothing is flagged. render_time is stored, shown
in the URL details modal, and added to crawled_urls via migration.
#94: "URLs Discovered" ignored the rows synthesized from image HEAD
checks, since those are never queued. The counter sat at the HTML page
count while crawled climbed past it. Image rows are now marked
discovered when their row is created.
Fixture tests cover both.
- Workshop button in the app header
- README and the plugins README point at it for finding/sharing plugins
The workshop is back up at workshop.librecrawl.com. On the marketing site
the /workshop redirect now goes there instead of to GitHub, and the nav
links point straight at the subdomain.
requests defaults to 10 connections per host, but crawl workers and the
image HEAD checks (5 threads of their own) draw from the same pool, so
urllib3 kept discarding live connections and reopening them:
WARNING:urllib3.connectionpool: Connection pool is full, discarding
connection: developer.mozilla.org
Mount an adapter sized to concurrency + IMAGE_CHECK_WORKERS + 5, and
resize it when concurrency changes in settings. An MDN crawl went from a
steady stream of these warnings to none.
Incremental polling sliced server arrays by the client's row count, so
updates to already-sent rows (linked_from, link statuses, image HEAD
results) never reached the UI, and the issue cursor desynced whenever an
exclusion pattern dropped one. Replaced with an event log (seq + epoch);
clients apply appends/updates by key.
- Relative links on redirected pages resolved against the requested URL,
minting internal 404s from the target site's links; chains now recorded
- Image HEAD checks ignored crawl_external
- Image rows counted against max_urls (25-URL react.dev crawl = 1 page)
- _save_batch_to_db cleared its queues after writing, dropping rows
appended mid-write (400-page Wikipedia crawl lost 3,693 links)
- Fetches in flight at max_urls hit the DB uncounted
- Sitemap discovery blocked start_crawl; now backgrounded, completion
waits on it
- start_crawl read session_id before creation, disabling persistence
- Link/issue cells used innerHTML on crawled page content
- Virtual scroller measures real row heights, rAF-throttled, spacers
track data length
Adds tests/: fixture tests (no network) + real-site harness.
Follow-up to #48:
- Issues export used nonexistent keys (issue_type/severity/description);
now matches the real issue dicts and legacy format (type/category/issue/details)
- Crawls loaded from a saved file exist only in browser state, which the
streaming endpoint can't see — exportData() now falls back to the legacy
POST /api/export_data path for those instead of downloading a 404 body
- target_status None renders as 'Not crawled' instead of blank (0 stays 0)
- Export field names from the query string are restricted to identifier-like
strings since they become XML tag names and CSV headers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Race condition: the cleanup thread calls stop_crawl() after a crawl has already been marked "completed", changing its status back to "stopped".
Added a status check before overwriting — if the crawl is already
"completed", leave it alone.
Follow-up to #87: full image downloads are now behind a 'Crawl Images'
checkbox (default off) instead of being in the default include_extensions.
When off, the Images tab is populated from the broken-image HEAD checks
that already run (status, content-type, size from Content-Length) at no
extra request cost. Also guards the issue detector against flagging
non-HTML results for missing titles/meta descriptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sqlite3.connect() fails with 'unable to open database file' on fresh
installs where data/ directory does not exist. This happens with both
Docker (bind mount creates ./data as root, container user can't write)
and bare Python (no data/ directory at all).
Fix ensures data/ exists before any database operation:
- main.py: add os.makedirs before init_db() call
- start-librecrawl.sh: mkdir -p data at script start
- start-librecrawl.bat: mkdir data at script start
Fixes#65
app.secret_key was hardcoded to a value committed in the public repo, so
every stock deployment shared the same Flask session-signing key — anyone
could forge a signed session cookie (including tier=admin) for any instance.
Now read SECRET_KEY from the environment, falling back to a per-process
secrets.token_hex(32) random key (with a startup warning) when unset. The
fallback is intentionally ephemeral rather than a static default, so no
public key is ever used — worst case is sessions not surviving a restart,
not forgeable admin cookies. Documents SECRET_KEY in .env.example and README.
Combines the secure fallback approach from #71 with the documentation from
#41; applied directly since #71 carried whole-file line-ending churn and
#41 kept an insecure static fallback.
Co-authored-by: getuslocal <getuslocal@users.noreply.github.com>
Co-authored-by: Mobilpadde <Mobilpadde@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire LOCAL_MODE via os.getenv so it can be set through .env / docker-compose,
matching how DISABLE_GUEST/REGISTRATION_DISABLED already work. The
REGISTRATION_DISABLED half of #40 already landed via #53, leaving this single
line; applied directly since #40 was a fork branch that conflicted after #53.
Safe now that LOCAL_MODE defaults to false (e48141d).
Co-authored-by: Mobilpadde <Mobilpadde@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LOCAL_MODE disables all authentication and grants every visitor admin
tier. Both the documented setup path (cp .env.example .env) and the
docker-compose default shipped LOCAL_MODE=true, so a stock deployment
exposed an unauthenticated admin server. Flip both defaults to false and
add explicit warnings. This also makes it safe to wire LOCAL_MODE as an
env var (PR #40).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Resume button was shown for completed crawls, but resuming a completed crawl always fails. Only render the button for statuses
that can actually be resumed: paused, failed, running, and stopped.
Only "paused" and "running" statuses were allowed in the resume logic. Stopped and failed crawls returned a misleading "Cannot resume crawl with status: stopped" error despite being resumable.
Added "stopped" and "failed" to the list of resumable statuses.
Only the CLI arguments (--disable-register, --disable-guest) were checked. The environment variable path was missing, so setting
REGISTRATION_DISABLED=true or DISABLE_GUEST=true had no effect.
Added os.getenv() fallback for both flags, accepting "true", "1", or "yes".
* Change external port binding for librecrawl service
Adds support to specify the `EXTERNAL_PORT` to bind to. Falls back to :5000 if `EXTERNAL_PORT` is not set.
Port 5000 would conflict with Synology NAS if installed there.
* Add EXTERNAL_PORT to .env.example
Added EXTERNAL_PORT variable for application port binding.
- collect_all_links() now also collects <img src> URLs as links with
placement='image', so they appear in the Links tab alongside <a href>
links.
- After collecting images per page, HEAD-check their URLs (parallel,
cached, capped at 50/page) and store the HTTP status.
- Issue detector flags broken images (4xx/5xx/0) as Content errors.
- Removed the hard-coded 20-image cap in extract_images().
Closes#61
- Remove forced max_depth=0 override when the starting URL has a path
component. The crawler now respects the user's configured depth
regardless of whether the URL is a domain root or a subpath like
/blog. Closes#59
- Add No Response / DNS Not Found filter (#44): classify fetch errors
(dns_not_found, timeout, connection_refused, ssl_error) and surface
them via a new sidebar filter, per-error-type rows in the Status
Codes table, issue detection, and DB persistence.
- Demo mode (--demo / DEMO_MODE): per-user 1.5GB memory limit with
auto-stop and incremental memory accounting via UserMemoryTracker.
- Dangerously skip auth (--dangerously-skip-auth / DANGEROUSLY_SKIP_AUTH):
log in as any username with no password; username only separates
per-user sessions.
- README: minor update.
RobotFileParser.read() uses urllib with Python's default UA header.
Many servers return 403 for that, causing RobotFileParser to set
disallow_all=True and block every URL. Now fetches robots.txt with
the crawler's requests session which has the proper User-Agent.
- Hierarchical layout now uses crawl depth data instead of BFS graph
distance, which was flat because nav links make every page 1 hop from root
- Added depth to node data in both backend and frontend
- Loaded crawl data no longer overwritten when opening visualization tab
Adds a flag to disable guest login, same pattern as --disable-register.
Supports both CLI arg (--disable-guest / -dg) and .env var (DISABLE_GUEST=true).
DB path was 'users.db' (resolving to /app/users.db) but the Docker
volume maps ./data:/app/data. Changed all DB paths to data/users.db
so the database is created inside the mounted volume.
Phiality