mirror of
https://github.com/PhialsBasement/LibreCrawl.git
synced 2026-10-18 00:19:37 +02:00
PhialsBasementandClaude Fable 5.1
82f40a083a
Duplicate-content detection compared every page with every other page using SequenceMatcher and emitted two issues per matching pair. On a 1,127-URL site with ~900 templated suburb pages that meant 4+ minutes of CPU, 703,902 issues, and a 90-second SQLite insert that locked the database for every other request (status polls, settings) — the UI sat in "finishing up" until the crawl was force-stopped. - issue_detector: one issue per page (count + closest five), identical title/description/H1 grouped first, candidate pairs from a prefix filter on title tokens, word-level Dice similarity, only 200 HTML pages considered, and a 120s time budget so the pass can never hang a crawl. 900 templated pages: 1.4s / 900 issues (was ~4min / 703k). - auth_db/crawl_db: 30s busy timeout and WAL mode so reads no longer fail with "database is locked" while the crawler saves a batch. Exports: "Excel (XLSX)" was offered in Settings but never implemented (it silently produced CSV), XML only applied to the URLs file (links/issues fell back to CSV), and the export button read the format from the server copy of the settings, which silently lags the browser copy when the background sync fails — e.g. while the database was locked by the bug above. - main: XLSX (openpyxl) and XML for URLs, links and issues on both the streaming and legacy export endpoints; shared row helpers. - app.js/settings.js: export with the settings the browser is showing; surface a failed server sync instead of swallowing it; decode base64 binary exports. - tests: fixture tests for one-issue-per-page duplicate detection (crawl + 1,500-page scale check) and for every format × every data type on both export endpoints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
14 lines
189 B
Plaintext
14 lines
189 B
Plaintext
requests==2.31.0
|
|
beautifulsoup4==4.12.2
|
|
urllib3==2.0.7
|
|
flask==2.3.3
|
|
flask-compress
|
|
waitress
|
|
playwright
|
|
nest-asyncio==1.5.8
|
|
psutil
|
|
bcrypt==4.1.2
|
|
markdown
|
|
python-dotenv
|
|
openpyxl
|