Fix: crawl stuck in "finishing up" on big sites; exports ignoring the chosen format

Duplicate-content detection compared every page with every other page using
SequenceMatcher and emitted two issues per matching pair. On a 1,127-URL site
with ~900 templated suburb pages that meant 4+ minutes of CPU, 703,902 issues,
and a 90-second SQLite insert that locked the database for every other
request (status polls, settings) — the UI sat in "finishing up" until the
crawl was force-stopped.

- issue_detector: one issue per page (count + closest five), identical
  title/description/H1 grouped first, candidate pairs from a prefix filter on
  title tokens, word-level Dice similarity, only 200 HTML pages considered,
  and a 120s time budget so the pass can never hang a crawl.
  900 templated pages: 1.4s / 900 issues (was ~4min / 703k).
- auth_db/crawl_db: 30s busy timeout and WAL mode so reads no longer fail
  with "database is locked" while the crawler saves a batch.

Exports: "Excel (XLSX)" was offered in Settings but never implemented (it
silently produced CSV), XML only applied to the URLs file (links/issues fell
back to CSV), and the export button read the format from the server copy of
the settings, which silently lags the browser copy when the background sync
fails — e.g. while the database was locked by the bug above.

- main: XLSX (openpyxl) and XML for URLs, links and issues on both the
  streaming and legacy export endpoints; shared row helpers.
- app.js/settings.js: export with the settings the browser is showing;
  surface a failed server sync instead of swallowing it; decode base64
  binary exports.
- tests: fixture tests for one-issue-per-page duplicate detection (crawl +
  1,500-page scale check) and for every format × every data type on both
  export endpoints.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.gitignore

@@ -3,5 +3,7 @@
 .claude/settings.local.json
 users.db
 data/*.db
+data/*.db-wal
+data/*.db-shm
 
 .env

README.md

@@ -25,7 +25,7 @@ LibreCrawl crawls websites and gives you detailed information about pages, links
 - 📊 **SEO analysis** - Extract titles, meta descriptions, headings, etc.
 - 🔗 **Link analysis** - Track internal and external links with detailed relationship mapping
 - 📈 **PageSpeed Insights integration** - Analyze Core Web Vitals
-- 💾 **Multiple export formats** - CSV, JSON, or XML
+- 💾 **Multiple export formats** - CSV, Excel (XLSX), JSON, or XML
 - 🔍 **Issue detection** - Automated SEO issue identification
 - ⚡ **Real-time crawling progress** with live statistics
 
@@ -287,6 +287,7 @@ For PageSpeed analysis, add a Google API key in Settings > Requests for higher r
 ## Export formats
 
 - **CSV**: Spreadsheet-friendly format
+- **Excel (XLSX)**: Native spreadsheet workbook
 - **JSON**: Structured data with all details
 - **XML**: Markup format for other tools
 

main.py

@@ -9,7 +9,8 @@ import argparse
 import secrets
 import string
 import os
-from io import StringIO
+import base64
+from io import StringIO, BytesIO
 from datetime import datetime, timedelta
 from flask import Flask, render_template, request, jsonify, session, redirect, url_for
 from flask_compress import Compress
@@ -1301,6 +1302,18 @@ def export_stream():
                     mimetype='application/json',
                     headers={'Content-Disposition': f'attachment; filename=librecrawl_links_{timestamp}.json'}
                 )
+            elif export_format == 'xml':
+                return Response(
+                    _stream_links_xml(links),
+                    mimetype='application/xml',
+                    headers={'Content-Disposition': f'attachment; filename=librecrawl_links_{timestamp}.xml'}
+                )
+            elif export_format == 'xlsx':
+                return Response(
+                    _links_xlsx(links),
+                    mimetype=XLSX_MIMETYPE,
+                    headers={'Content-Disposition': f'attachment; filename=librecrawl_links_{timestamp}.xlsx'}
+                )
             else:
                 return Response(
                     _stream_links_csv(links),
@@ -1323,6 +1336,18 @@ def export_stream():
                     mimetype='application/json',
                     headers={'Content-Disposition': f'attachment; filename=librecrawl_issues_{timestamp}.json'}
                 )
+            elif export_format == 'xml':
+                return Response(
+                    _stream_issues_xml(issues),
+                    mimetype='application/xml',
+                    headers={'Content-Disposition': f'attachment; filename=librecrawl_issues_{timestamp}.xml'}
+                )
+            elif export_format == 'xlsx':
+                return Response(
+                    _issues_xlsx(issues),
+                    mimetype=XLSX_MIMETYPE,
+                    headers={'Content-Disposition': f'attachment; filename=librecrawl_issues_{timestamp}.xlsx'}
+                )
             else:
                 return Response(
                     _stream_issues_csv(issues),
@@ -1353,6 +1378,12 @@ def export_stream():
                     mimetype='application/xml',
                     headers={'Content-Disposition': f'attachment; filename=librecrawl_export_{timestamp}.xml'}
                 )
+            elif export_format == 'xlsx':
+                return Response(
+                    _urls_xlsx(urls, regular_fields),
+                    mimetype=XLSX_MIMETYPE,
+                    headers={'Content-Disposition': f'attachment; filename=librecrawl_export_{timestamp}.xlsx'}
+                )
             else:
                 return Response(
                     _stream_urls_csv(urls, regular_fields),
@@ -1394,6 +1425,77 @@ def _format_csv_value(value, field):
     return value
 
 
+XLSX_MIMETYPE = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
+_XLSX_CELL_LIMIT = 32767  # Excel's per-cell character limit
+
+
+def _xlsx_cell(value):
+    """Coerce an export value into something openpyxl will accept."""
+    if value is None or isinstance(value, bool):
+        return value
+    if isinstance(value, (int, float)):
+        return value
+    from openpyxl.cell.cell import ILLEGAL_CHARACTERS_RE
+    text = ILLEGAL_CHARACTERS_RE.sub('', str(value))
+    return text[:_XLSX_CELL_LIMIT]
+
+
+def _build_xlsx(sheet_title, headers, rows):
+    """Build an .xlsx workbook in memory: one sheet, a header row, then `rows`
+    (iterables of cell values). Write-only mode streams rows to a temp file,
+    so memory stays flat for large crawls."""
+    from openpyxl import Workbook
+    workbook = Workbook(write_only=True)
+    sheet = workbook.create_sheet(title=sheet_title)
+    sheet.append(list(headers))
+    for row in rows:
+        sheet.append([_xlsx_cell(v) for v in row])
+    buffer = BytesIO()
+    workbook.save(buffer)
+    return buffer.getvalue()
+
+
+LINK_EXPORT_FIELDS = ['source_url', 'target_url', 'anchor_text', 'is_internal', 'target_domain', 'target_status', 'placement']
+ISSUE_EXPORT_FIELDS = ['url', 'type', 'category', 'issue', 'details']
+
+
+def _link_export_row(link):
+    return {
+        'source_url': link.get('source_url', ''),
+        'target_url': link.get('target_url', ''),
+        'anchor_text': link.get('anchor_text', ''),
+        'is_internal': 'Yes' if link.get('is_internal') else 'No',
+        'target_domain': link.get('target_domain', ''),
+        'target_status': link['target_status'] if link.get('target_status') is not None else 'Not crawled',
+        'placement': link.get('placement', 'body')
+    }
+
+
+def _issue_export_row(issue):
+    return {
+        'url': issue.get('url', ''),
+        'type': issue.get('type', ''),
+        'category': issue.get('category', ''),
+        'issue': issue.get('issue', ''),
+        'details': str(issue.get('details', ''))
+    }
+
+
+def _urls_xlsx(urls, fields):
+    return _build_xlsx('URLs', fields,
+                       ([_format_csv_value(u.get(f, ''), f) for f in fields] for u in urls))
+
+
+def _links_xlsx(links):
+    return _build_xlsx('Links', LINK_EXPORT_FIELDS,
+                       ([_link_export_row(l)[f] for f in LINK_EXPORT_FIELDS] for l in links))
+
+
+def _issues_xlsx(issues):
+    return _build_xlsx('Issues', ISSUE_EXPORT_FIELDS,
+                       ([_issue_export_row(i)[f] for f in ISSUE_EXPORT_FIELDS] for i in issues))
+
+
 def _stream_urls_csv(urls, fields):
     """Generator that yields CSV rows one at a time"""
     output = StringIO()
@@ -1442,26 +1544,43 @@ def _stream_urls_xml(urls, fields):
     yield '</librecrawl_export>\n'
 
 
+def _stream_records_xml(root_tag, list_tag, item_tag, total_attr, total, records, fields):
+    """Generator yielding an XML document of flat records, one element per field
+    (same shape as _stream_urls_xml)."""
+    yield '<?xml version="1.0" encoding="UTF-8"?>\n'
+    yield f'<{root_tag} export_date="{time.strftime("%Y-%m-%d %H:%M:%S")}" {total_attr}="{total}">\n'
+    yield f'  <{list_tag}>\n'
+    for record in records:
+        yield f'    <{item_tag}>\n'
+        for field in fields:
+            escaped = str(record.get(field, '')).replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;')
+            yield f'      <{field}>{escaped}</{field}>\n'
+        yield f'    </{item_tag}>\n'
+    yield f'  </{list_tag}>\n'
+    yield f'</{root_tag}>\n'
+
+
+def _stream_links_xml(links):
+    return _stream_records_xml('librecrawl_links', 'links', 'link', 'total_links', len(links),
+                               (_link_export_row(link) for link in links), LINK_EXPORT_FIELDS)
+
+
+def _stream_issues_xml(issues):
+    return _stream_records_xml('librecrawl_issues', 'issues', 'issue', 'total_issues', len(issues),
+                               (_issue_export_row(issue) for issue in issues), ISSUE_EXPORT_FIELDS)
+
+
 def _stream_links_csv(links):
     """Generator that yields link CSV rows one at a time"""
-    fieldnames = ['source_url', 'target_url', 'anchor_text', 'is_internal', 'target_domain', 'target_status', 'placement']
     output = StringIO()
-    writer = csv.DictWriter(output, fieldnames=fieldnames)
+    writer = csv.DictWriter(output, fieldnames=LINK_EXPORT_FIELDS)
     writer.writeheader()
     yield output.getvalue()
     output.truncate(0)
     output.seek(0)
 
     for link in links:
-        writer.writerow({
-            'source_url': link.get('source_url', ''),
-            'target_url': link.get('target_url', ''),
-            'anchor_text': link.get('anchor_text', ''),
-            'is_internal': 'Yes' if link.get('is_internal') else 'No',
-            'target_domain': link.get('target_domain', ''),
-            'target_status': link['target_status'] if link.get('target_status') is not None else 'Not crawled',
-            'placement': link.get('placement', 'body')
-        })
+        writer.writerow(_link_export_row(link))
         yield output.getvalue()
         output.truncate(0)
         output.seek(0)
@@ -1489,22 +1608,15 @@ def _stream_links_json(links):
 
 def _stream_issues_csv(issues):
     """Generator that yields issue CSV rows one at a time"""
-    fieldnames = ['url', 'type', 'category', 'issue', 'details']
     output = StringIO()
-    writer = csv.DictWriter(output, fieldnames=fieldnames)
+    writer = csv.DictWriter(output, fieldnames=ISSUE_EXPORT_FIELDS)
     writer.writeheader()
     yield output.getvalue()
     output.truncate(0)
     output.seek(0)
 
     for issue in issues:
-        writer.writerow({
-            'url': issue.get('url', ''),
-            'type': issue.get('type', ''),
-            'category': issue.get('category', ''),
-            'issue': issue.get('issue', ''),
-            'details': str(issue.get('details', ''))
-        })
+        writer.writerow(_issue_export_row(issue))
         yield output.getvalue()
         output.truncate(0)
         output.seek(0)
@@ -1600,6 +1712,14 @@ def export_data():
                 issues_content = generate_issues_json_export(issues)
                 issues_mimetype = 'application/json'
                 issues_filename = f'librecrawl_issues_{int(time.time())}.json'
+            elif export_format == 'xml':
+                issues_content = ''.join(_stream_issues_xml(issues))
+                issues_mimetype = 'application/xml'
+                issues_filename = f'librecrawl_issues_{int(time.time())}.xml'
+            elif export_format == 'xlsx':
+                issues_content = base64.b64encode(_issues_xlsx(issues)).decode('ascii')
+                issues_mimetype = XLSX_MIMETYPE
+                issues_filename = f'librecrawl_issues_{int(time.time())}.xlsx'
             else:
                 issues_content = generate_issues_csv_export(issues)
                 issues_mimetype = 'text/csv'
@@ -1608,7 +1728,8 @@ def export_data():
             files_to_export.append({
                 'content': issues_content,
                 'mimetype': issues_mimetype,
-                'filename': issues_filename
+                'filename': issues_filename,
+                **({'encoding': 'base64'} if export_format == 'xlsx' else {})
             })
 
         # Generate links export if requested
@@ -1621,6 +1742,14 @@ def export_data():
                 links_content = generate_links_json_export(links)
                 links_mimetype = 'application/json'
                 links_filename = f'librecrawl_links_{int(time.time())}.json'
+            elif export_format == 'xml':
+                links_content = ''.join(_stream_links_xml(links))
+                links_mimetype = 'application/xml'
+                links_filename = f'librecrawl_links_{int(time.time())}.xml'
+            elif export_format == 'xlsx':
+                links_content = base64.b64encode(_links_xlsx(links)).decode('ascii')
+                links_mimetype = XLSX_MIMETYPE
+                links_filename = f'librecrawl_links_{int(time.time())}.xlsx'
             else:
                 links_content = generate_links_csv_export(links)
                 links_mimetype = 'text/csv'
@@ -1629,7 +1758,8 @@ def export_data():
             files_to_export.append({
                 'content': links_content,
                 'mimetype': links_mimetype,
-                'filename': links_filename
+                'filename': links_filename,
+                **({'encoding': 'base64'} if export_format == 'xlsx' else {})
             })
 
         # Generate regular export if there are regular fields
@@ -1646,13 +1776,18 @@ def export_data():
                 regular_content = generate_xml_export(urls, regular_fields)
                 regular_mimetype = 'application/xml'
                 regular_filename = f'librecrawl_export_{int(time.time())}.xml'
+            elif export_format == 'xlsx':
+                regular_content = base64.b64encode(_urls_xlsx(urls, regular_fields)).decode('ascii')
+                regular_mimetype = XLSX_MIMETYPE
+                regular_filename = f'librecrawl_export_{int(time.time())}.xlsx'
             else:
                 return jsonify({'success': False, 'error': 'Unsupported export format'})
 
             files_to_export.append({
                 'content': regular_content,
                 'mimetype': regular_mimetype,
-                'filename': regular_filename
+                'filename': regular_filename,
+                **({'encoding': 'base64'} if export_format == 'xlsx' else {})
             })
 
         # Handle special case where only special fields are selected but no data

requirements.txt

@@ -10,3 +10,4 @@ psutil
 bcrypt==4.1.2
 markdown
 python-dotenv
+openpyxl

src/auth_db.py

@@ -15,7 +15,8 @@ DB_FILE = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__)
 @contextmanager
 def get_db():
     """Context manager for database connections"""
-    conn = sqlite3.connect(DB_FILE)
+    # Wait for a busy database instead of failing immediately (see crawl_db).
+    conn = sqlite3.connect(DB_FILE, timeout=30)
     conn.row_factory = sqlite3.Row  # Return rows as dictionaries
     try:
         yield conn
@@ -30,6 +31,10 @@ def init_db():
     """Initialize the database with users and settings tables"""
     os.makedirs(os.path.dirname(DB_FILE), exist_ok=True)
     with get_db() as conn:
+        # Write-ahead logging lets status/settings reads proceed while the
+        # crawler thread is saving a batch, instead of "database is locked".
+        # Persistent: set once here, applies to every later connection.
+        conn.execute('PRAGMA journal_mode=WAL')
         cursor = conn.cursor()
         cursor.execute('''
             CREATE TABLE IF NOT EXISTS users (

src/core/issue_detector.py

@@ -1,17 +1,32 @@
 """SEO issue detection and reporting"""
+import math
+import re
 import threading
+import time
+from collections import defaultdict
 from fnmatch import fnmatch
 from urllib.parse import urlparse
-from difflib import SequenceMatcher
+
+# Near-duplicate detection is bounded so a large site can never hold a crawl in
+# "finishing up" indefinitely: when the budget is exhausted the pass stops with
+# a warning and partial results instead of running to completion.
+DUPLICATION_TIME_BUDGET_SECONDS = 120
+DUPLICATION_EXAMPLES_PER_PAGE = 5
+
+_TOKEN_SPLIT = re.compile(r'[^\w]+')
 
 
 class IssueDetector:
     """Detects SEO and technical issues in crawled pages"""
 
+    # Weights of the signals that make up the duplicate-content score.
+    _DUP_WEIGHTS = {'title': 0.35, 'desc': 0.35, 'h1': 0.20, 'word_count': 0.10}
+
     def __init__(self, exclusion_patterns=None):
         self.exclusion_patterns = exclusion_patterns or []
         self.detected_issues = []
         self.issues_lock = threading.Lock()
+        self.duplication_truncated = False
 
     def detect_issues(self, result):
         """Detect SEO issues for a crawled URL"""
@@ -370,121 +385,226 @@ class IssueDetector:
 
     def detect_duplication_issues(self, all_results, similarity_threshold=0.85):
         """
-        Detect content duplication across all crawled pages.
+        Flag pages whose title, meta description, H1 and length are near-identical
+        to other crawled pages.
+
+        Emits at most ONE issue per page — how many near-duplicates it has and
+        the closest few — so a site of N templated pages yields N issues, not N².
+        (The previous pairwise version produced two issues per matching pair:
+        700k rows for a 1,000-page site, which took minutes to compute and
+        locked SQLite for over a minute while saving.)
+
+        Only HTML pages that returned 200 and carry a title or description are
+        considered. Pages with identical normalised title/description/H1 are
+        grouped first; one representative per group is scored and the result
+        applies to every member. Candidate pairs between groups come from a
+        prefix filter on title tokens: with the default 0.85 threshold a pair
+        needs title similarity of at least 0.57, so any qualifying pair must
+        share one of each title's rarest few tokens. Sites with distinct titles
+        therefore score very few pairs; templated sites score many, but each
+        comparison is a handful of set operations, and the whole pass stops
+        after DUPLICATION_TIME_BUDGET_SECONDS with partial results rather than
+        holding the crawl in "finishing up".
 
         Args:
             all_results: List of all crawled result dictionaries
             similarity_threshold: Minimum similarity ratio to flag as duplicate (0.0-1.0)
         """
+        started = time.monotonic()
+        self.duplication_truncated = False
+
+        pages = self._duplication_candidates(all_results)
+        if len(pages) < 2:
+            return
+
+        # 1. Group pages with identical title / description / H1.
+        group_index = {}
+        groups = []
+        for page in pages:
+            signature = (page['title'], page['desc'], page['h1'])
+            idx = group_index.get(signature)
+            if idx is None:
+                idx = group_index[signature] = len(groups)
+                groups.append([])
+            groups[idx].append(page)
+        representatives = [group[0] for group in groups]
+
+        # 2. Score candidate pairs of groups, each pair once.
+        similar = defaultdict(list)  # group idx -> [(similarity, other group idx)]
+        compared = 0
+        for i, j in self._duplication_candidate_pairs(representatives, similarity_threshold):
+            compared += 1
+            if compared % 2000 == 0 and time.monotonic() - started > DUPLICATION_TIME_BUDGET_SECONDS:
+                self.duplication_truncated = True
+                print(f"Duplication detection stopped after {DUPLICATION_TIME_BUDGET_SECONDS}s "
+                      f"and {compared} comparisons; duplicate results are partial")
+                break
+            similarity = self._duplication_similarity(
+                representatives[i], representatives[j], similarity_threshold)
+            if similarity is not None:
+                similar[i].append((similarity, j))
+                similar[j].append((similarity, i))
+
+        # 3. One issue per page.
         issues = []
-        processed_pairs = set()
-
-        # Compare each result with all others
-        for i, result1 in enumerate(all_results):
-            url1 = result1.get('url', '')
-
-            # Skip if URL should be excluded
-            if self._should_exclude(url1):
+        for idx, group in enumerate(groups):
+            partners = [(1.0, member['url'], True) for member in group]
+            for similarity, j in similar.get(idx, ()):
+                partners.extend((similarity, other['url'], False) for other in groups[j])
+            if len(partners) < 2:
                 continue
-
-            for j, result2 in enumerate(all_results):
-                # Skip same URL or already processed pairs
-                if i >= j:
-                    continue
-
-                url2 = result2.get('url', '')
-
-                # Skip if URL should be excluded
-                if self._should_exclude(url2):
-                    continue
-
-                # Create unique pair identifier
-                pair_key = tuple(sorted([url1, url2]))
-                if pair_key in processed_pairs:
-                    continue
-
-                processed_pairs.add(pair_key)
-
-                # Calculate similarity
-                similarity = self._calculate_content_similarity(result1, result2)
-
-                # Flag as duplicate if above threshold
-                if similarity >= similarity_threshold:
-                    # Add issue for both URLs
-                    issues.append({
-                        'url': url1,
-                        'type': 'warning',
-                        'category': 'Duplication',
-                        'issue': 'Duplicate Content Detected',
-                        'details': f'Content is {similarity*100:.1f}% similar to {url2}'
-                    })
-                    issues.append({
-                        'url': url2,
-                        'type': 'warning',
-                        'category': 'Duplication',
-                        'issue': 'Duplicate Content Detected',
-                        'details': f'Content is {similarity*100:.1f}% similar to {url1}'
-                    })
+            partners.sort(key=lambda p: (-p[0], p[1]))
+            total = len(partners) - 1  # everything except the page itself
+            for page in group:
+                examples = [p for p in partners if p[1] != page['url']][:DUPLICATION_EXAMPLES_PER_PAGE]
+                described = '; '.join(
+                    f"{url} (identical title, description and H1)" if identical
+                    else f"{url} ({similarity * 100:.1f}% similar)"
+                    for similarity, url, identical in examples
+                )
+                more = total - len(examples)
+                issues.append({
+                    'url': page['url'],
+                    'type': 'warning',
+                    'category': 'Duplication',
+                    'issue': 'Duplicate Content Detected',
+                    'details': f"{total} similar page{'s' if total != 1 else ''}: {described}"
+                               + (f" (+{more} more)" if more > 0 else '')
+                })
 
         # Add all detected duplication issues
         with self.issues_lock:
             self.detected_issues.extend(issues)
 
-    def _calculate_content_similarity(self, result1, result2):
-        """
-        Calculate similarity between two page results.
+    def _duplication_candidates(self, all_results):
+        """Pages eligible for duplicate detection, with normalised fields and token sets."""
+        pages = []
+        for result in all_results:
+            url = result.get('url', '')
+            if not url or self._should_exclude(url):
+                continue
+            status = result.get('status_code')
+            if status is not None and status != 200:
+                continue
+            content_type = (result.get('content_type') or '').lower()
+            if content_type and 'html' not in content_type:
+                continue
+            page = self._duplication_page(result)
+            if not page['title'] and not page['desc']:
+                continue
+            pages.append(page)
+        return pages
 
-        Compares title, meta description, h1, and content length.
-        Returns a similarity ratio between 0.0 and 1.0.
-        """
-        # Extract content fields
-        title1 = result1.get('title', '').lower().strip()
-        title2 = result2.get('title', '').lower().strip()
-
-        desc1 = result1.get('meta_description', '').lower().strip()
-        desc2 = result2.get('meta_description', '').lower().strip()
-
-        h1_1 = result1.get('h1', '').lower().strip()
-        h1_2 = result2.get('h1', '').lower().strip()
-
-        word_count1 = result1.get('word_count', 0)
-        word_count2 = result2.get('word_count', 0)
-
-        # Calculate individual similarities
-        title_sim = self._text_similarity(title1, title2) if title1 and title2 else 0
-        desc_sim = self._text_similarity(desc1, desc2) if desc1 and desc2 else 0
-        h1_sim = self._text_similarity(h1_1, h1_2) if h1_1 and h1_2 else 0
-
-        # Word count similarity (1.0 if within 10% of each other)
-        if word_count1 and word_count2:
-            max_count = max(word_count1, word_count2)
-            min_count = min(word_count1, word_count2)
-            word_count_sim = min_count / max_count if max_count > 0 else 0
-        else:
-            word_count_sim = 0
-
-        # Weighted average (title and description are most important)
-        weights = {
-            'title': 0.35,
-            'desc': 0.35,
-            'h1': 0.20,
-            'word_count': 0.10
+    def _duplication_page(self, result):
+        title = self._normalise_text(result.get('title'))
+        desc = self._normalise_text(result.get('meta_description'))
+        h1 = self._normalise_text(result.get('h1'))
+        return {
+            'url': result.get('url', ''),
+            'title': title,
+            'desc': desc,
+            'h1': h1,
+            'title_tokens': self._tokens(title),
+            'desc_tokens': self._tokens(desc),
+            'h1_tokens': self._tokens(h1),
+            'word_count': result.get('word_count') or 0,
         }
 
-        overall_similarity = (
-            title_sim * weights['title'] +
-            desc_sim * weights['desc'] +
-            h1_sim * weights['h1'] +
-            word_count_sim * weights['word_count']
-        )
+    @staticmethod
+    def _normalise_text(value):
+        """Lower-cased, whitespace-collapsed text; H1 may arrive as a list."""
+        if isinstance(value, list):
+            value = next((v for v in value if isinstance(v, str) and v.strip()), '')
+        if not isinstance(value, str):
+            return ''
+        return ' '.join(value.lower().split())
 
-        return overall_similarity
+    @staticmethod
+    def _tokens(text):
+        return frozenset(t for t in _TOKEN_SPLIT.split(text) if t)
 
-    def _text_similarity(self, text1, text2):
-        """Calculate similarity ratio between two text strings using SequenceMatcher"""
-        if not text1 or not text2:
+    def _duplication_candidate_pairs(self, representatives, threshold):
+        """Yield (i, j) with i < j for the pairs worth scoring.
+
+        Uses a prefix filter on title tokens. The title carries 35% of the
+        score, so a pair can only reach `threshold` if the titles' Dice
+        similarity is at least (threshold - 0.65) / 0.35. Dice(A, B) >= t
+        implies |A ∩ B| >= t·|A| / (2 - t); ordering tokens by global rarity,
+        two such sets must share a token within the first
+        |A| - need + 1 tokens of each, so indexing those prefixes finds every
+        qualifying pair while skipping most pairs of distinct titles.
+        """
+        weights = self._DUP_WEIGHTS
+        min_title = (threshold - (1.0 - weights['title'])) / weights['title']
+        count = len(representatives)
+
+        if min_title <= 0:
+            # Threshold is too low for the title to prune anything.
+            for i in range(count):
+                for j in range(i + 1, count):
+                    yield i, j
+            return
+
+        frequency = defaultdict(int)
+        for rep in representatives:
+            for token in rep['title_tokens']:
+                frequency[token] += 1
+
+        index = defaultdict(list)
+        prefixes = []
+        for i, rep in enumerate(representatives):
+            ordered = sorted(rep['title_tokens'], key=lambda t: (frequency[t], t))
+            need = math.ceil(min_title * len(ordered) / (2 - min_title))
+            prefix = ordered[:max(1, len(ordered) - need + 1)] if ordered else []
+            prefixes.append(prefix)
+            for token in prefix:
+                index[token].append(i)
+
+        for i, prefix in enumerate(prefixes):
+            seen = set()
+            for token in prefix:
+                for j in index[token]:
+                    if j < i and j not in seen:
+                        seen.add(j)
+                        yield j, i
+
+    def _duplication_similarity(self, a, b, threshold):
+        """Weighted similarity of two candidate pages, or None once it is clear
+        the score cannot reach `threshold` (so cheap signals short-circuit)."""
+        weights = self._DUP_WEIGHTS
+        score = self._dice(a['title_tokens'], b['title_tokens']) * weights['title']
+        if score + weights['desc'] + weights['h1'] + weights['word_count'] < threshold:
+            return None
+        score += self._dice(a['desc_tokens'], b['desc_tokens']) * weights['desc']
+        if score + weights['h1'] + weights['word_count'] < threshold:
+            return None
+        score += self._dice(a['h1_tokens'], b['h1_tokens']) * weights['h1']
+        if score + weights['word_count'] < threshold:
+            return None
+        score += self._length_similarity(a['word_count'], b['word_count']) * weights['word_count']
+        return score if score >= threshold else None
+
+    @staticmethod
+    def _dice(tokens_a, tokens_b):
+        """Dice coefficient of two token sets (0.0 when either is empty)."""
+        if not tokens_a or not tokens_b:
             return 0.0
-        return SequenceMatcher(None, text1, text2).ratio()
+        return 2.0 * len(tokens_a & tokens_b) / (len(tokens_a) + len(tokens_b))
+
+    @staticmethod
+    def _length_similarity(count_a, count_b):
+        if not count_a or not count_b:
+            return 0.0
+        return min(count_a, count_b) / max(count_a, count_b)
+
+    def _calculate_content_similarity(self, result1, result2):
+        """
+        Similarity of two page results in [0.0, 1.0]: word-level Dice
+        similarity of title (35%), meta description (35%) and H1 (20%), plus
+        the ratio of their word counts (10%).
+        """
+        return self._duplication_similarity(
+            self._duplication_page(result1), self._duplication_page(result2), 0.0)
 
     def _should_exclude(self, url):
         """Check if URL should be excluded from issue detection"""

src/crawl_db.py

@@ -16,7 +16,10 @@ DB_FILE = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__)
 @contextmanager
 def get_db():
     """Context manager for database connections"""
-    conn = sqlite3.connect(DB_FILE)
+    # Wait for a busy database instead of failing immediately: the crawler
+    # thread writes batches while request handlers read status and settings,
+    # and sqlite's default 5s was not enough during large end-of-crawl saves.
+    conn = sqlite3.connect(DB_FILE, timeout=30)
     conn.row_factory = sqlite3.Row  # Return rows as dictionaries
     try:
         yield conn

tests/README.md

@@ -13,7 +13,7 @@ python tests/fixture_tests.py
 ```
 
 Prints a PASS/FAIL line per behaviour and exits non-zero if any fail. If ports
-8911-8917 are taken locally, set `FIXTURE_BASE_PORT` to move the range.
+8911-8920 are taken locally, set `FIXTURE_BASE_PORT` to move the range.
 
 Each test pins a bug that reached production once:
 
@@ -24,6 +24,8 @@ Each test pins a bug that reached production once:
 | `test_sitemap_discovery_is_async` | Sitemap probing blocks the start-crawl request, or the crawl finishes before discovery delivers its URLs |
 | `test_max_urls_counts_pages_not_images` | Image rows consume the max-URL budget, so an image-heavy site crawls only a page or two |
 | `test_event_ordering_under_polling` | The UI receives an update for a row it was never sent, or the same row twice |
+| `test_duplicate_detection_is_linear` | A site of templated pages holds the crawl in "finishing up" for minutes and produces two duplicate issues per matching pair (700k rows for 1,000 pages), locking the database while they save |
+| `test_export_formats_apply_to_every_data_type` | The format chosen in Settings is ignored for links/issues exports, or Excel silently produces CSV |
 
 ## crawl_harness.py
 

tests/fixture_tests.py

@@ -376,6 +376,134 @@ def test_js_response_time_excludes_render_wait():
         a.shutdown()
 
 
+def test_duplicate_detection_is_linear():
+    """A site of N near-identical pages (suburb "doorway" pages) used to hold
+    the crawl in "finishing up" for minutes and produce N² duplicate issues
+    (700k rows for 1,000 pages), locking SQLite while they were saved. It must
+    finish promptly with exactly one issue per page."""
+    from collections import Counter
+    from src.core.issue_detector import IssueDetector
+
+    site = BASE_PORT + 9
+    count = 60
+    routes = {'/': html(''.join(f'<a href="/area-{i}/">{i}</a>' for i in range(count)))}
+    for i in range(count):
+        routes[f'/area-{i}/'] = (
+            f'<html><head><title>Asbestos Removal Area{i} | Get a Free Quote Today!</title>'
+            f'<meta name="description" content="Proudly known as the best Asbestos Removal '
+            f'experts in Area{i}! Enjoy peace of mind."></head>'
+            f'<body><h1>Asbestos Removal Area{i}</h1><p>{"word " * 300}</p></body></html>')
+    a = serve(make_handler(routes), site)
+    try:
+        started = time.time()
+        crawler = crawl(f'http://127.0.0.1:{site}/', max_urls=count + 1,
+                        enable_duplication_check=True, duplication_threshold=0.85)
+        elapsed = time.time() - started
+        dupes = [i for i in crawler.issue_detector.get_issues()
+                 if i.get('issue') == 'Duplicate Content Detected']
+        per_url = Counter(i['url'] for i in dupes)
+        result('one duplicate issue per templated page',
+               len(dupes) == count and per_url and max(per_url.values()) == 1,
+               f'{len(dupes)} issues for {count} pages')
+        result('each issue counts the other templated pages',
+               bool(dupes) and all(f'{count - 1} similar pages' in i['details'] for i in dupes),
+               dupes[0]['details'][:100] if dupes else 'no issues')
+        result('the unrelated index page is not flagged',
+               f'http://127.0.0.1:{site}/' not in per_url)
+        result('crawl finished promptly', elapsed < 60, f'{elapsed:.1f}s')
+    finally:
+        a.shutdown()
+
+    # Scale check on the detector alone: 1,500 templated pages used to take
+    # ~10 minutes of pairwise SequenceMatcher calls.
+    pages = [{'url': f'https://x.au/p{i}/', 'status_code': 200, 'content_type': 'text/html',
+              'title': f'Asbestos Removal Area{i} | Get a Free Quote Today!',
+              'meta_description': f'Proudly known as the best Asbestos Removal experts in Area{i}!',
+              'h1': f'Asbestos Removal Area{i}', 'word_count': 1000 + i % 9} for i in range(1500)]
+    detector = IssueDetector()
+    started = time.time()
+    detector.detect_duplication_issues(pages, 0.85)
+    elapsed = time.time() - started
+    result('1,500 templated pages produce 1,500 issues quickly',
+           len(detector.get_issues()) == 1500 and elapsed < 30 and not detector.duplication_truncated,
+           f'{len(detector.get_issues())} issues in {elapsed:.1f}s')
+
+
+def test_export_formats_apply_to_every_data_type():
+    """The format picked in Settings must apply to URLs, links AND issues.
+    Links/issues used to fall back to CSV for XML, and "Excel (XLSX)" was
+    offered in the UI but never implemented, so it silently produced CSV."""
+    import base64
+    import json
+    import types
+    import xml.etree.ElementTree as ET
+    from io import BytesIO
+
+    os.environ.setdefault('LOCAL_MODE', 'true')
+    import main as app_module
+    from openpyxl import load_workbook
+    from src.core.issue_detector import IssueDetector
+
+    urls = [{'url': 'https://e.au/', 'status_code': 200, 'title': 'Home', 'h2': ['a', 'b']},
+            {'url': 'https://e.au/p', 'status_code': 404, 'title': 'Gone', 'h2': []}]
+    links = [{'source_url': 'https://e.au/', 'target_url': 'https://e.au/p', 'anchor_text': 'p',
+              'is_internal': True, 'target_domain': 'e.au', 'target_status': 404, 'placement': 'body'}]
+    issues = [{'url': 'https://e.au/p', 'type': 'error', 'category': 'HTTP', 'issue': 'Not Found',
+               'details': '404'}]
+    formats = (('csv', 'text/csv'), ('json', 'application/json'),
+               ('xml', 'application/xml'), ('xlsx', app_module.XLSX_MIMETYPE))
+
+    def parses(fmt, data, expect_rows):
+        if fmt == 'xlsx':
+            rows = list(load_workbook(BytesIO(data)).active.iter_rows(values_only=True))
+            return len(rows) == expect_rows + 1
+        if fmt == 'xml':
+            root = ET.fromstring(data)
+            return len(root[0]) == expect_rows
+        if fmt == 'json':
+            return len(json.loads(data)['data']) == expect_rows
+        return data.decode().count('\n') == expect_rows + 1
+
+    client = app_module.app.test_client()
+    with client.session_transaction() as s:
+        s['user_id'] = 1
+        s['username'] = 'fixture'
+        s['tier'] = 'admin'
+
+    # Streaming endpoint (active or loaded crawl): seed this session's crawler.
+    client.get('/api/crawl_status?stats_only=1')
+    with client.session_transaction() as s:
+        crawler = app_module.crawler_instances[s['session_id']]['crawler']
+    crawler.crawl_results = urls
+    crawler.issue_detector = IssueDetector()
+    crawler.issue_detector.detected_issues = list(issues)
+    crawler.link_manager = types.SimpleNamespace(all_links=[dict(l) for l in links])
+    for data_type, expect_rows in (('urls', 2), ('links', 1), ('issues', 1)):
+        for fmt, mimetype in formats:
+            resp = client.get(f'/api/export_stream?format={fmt}&type={data_type}&fields=url,title,h2')
+            disposition = resp.headers.get('Content-Disposition', '')
+            ok = (resp.status_code == 200 and resp.mimetype == mimetype
+                  and disposition.endswith('.' + fmt) and parses(fmt, resp.data, expect_rows))
+            result(f'streaming {data_type} export honours {fmt}', ok,
+                   f'{resp.status_code} {resp.mimetype} {disposition}')
+
+    # Legacy endpoint (crawl loaded from a file, data lives in the browser).
+    for fmt, mimetype in formats:
+        resp = client.post('/api/export_data', json={
+            'format': fmt, 'fields': ['url', 'title', 'links_detailed', 'issues_detected'],
+            'localData': {'urls': urls, 'links': links, 'issues': issues}})
+        body = resp.get_json() or {}
+        files = body.get('files') or []
+        ok = body.get('success') and len(files) == 3 and all(
+            f['mimetype'] == mimetype and f['filename'].endswith('.' + fmt) for f in files)
+        if ok and fmt == 'xlsx':
+            ok = all(f.get('encoding') == 'base64'
+                     and parses('xlsx', base64.b64decode(f['content']), 1 if 'export' not in f['filename'] else 2)
+                     for f in files)
+        result(f'legacy export honours {fmt} for every file', bool(ok),
+               ', '.join(f['filename'] for f in files) if files else str(body)[:120])
+
+
 def _playwright_available():
     try:
         import playwright  # noqa: F401
@@ -392,6 +520,8 @@ TESTS = (
     test_event_ordering_under_polling,
     test_discovered_counts_image_rows,
     test_js_response_time_excludes_render_wait,
+    test_duplicate_detection_is_linear,
+    test_export_formats_apply_to_every_data_type,
 )
 
 

web/static/js/app.js

@@ -1583,16 +1583,24 @@ async function exportData() {
         exportBtn.textContent = 'Exporting...';
     }
     try {
-        // Get current settings to determine export format and fields
-        const settingsResponse = await fetch('/api/get_settings');
-        const settingsData = await settingsResponse.json();
+        // Export with the settings this browser is showing. settings.js keeps
+        // them in localStorage and syncs to the server in the background; the
+        // server copy can lag (a failed sync, a busy database), which used to
+        // make exports ignore the format picked in Settings. Only fall back to
+        // the server copy when nothing is loaded here.
+        let settings = (typeof currentSettings !== 'undefined' && currentSettings
+                        && Object.keys(currentSettings).length) ? currentSettings : null;
+        if (!settings) {
+            const settingsResponse = await fetch('/api/get_settings');
+            const settingsData = await settingsResponse.json();
 
-        if (!settingsData.success) {
-            showNotification('Failed to get export settings', 'error');
-            return;
+            if (!settingsData.success) {
+                showNotification('Failed to get export settings', 'error');
+                return;
+            }
+            settings = settingsData.settings;
         }
 
-        const settings = settingsData.settings;
         const exportFormat = settings.exportFormat || 'csv';
         const exportFields = settings.exportFields || ['url', 'status_code', 'title', 'meta_description', 'h1'];
 
@@ -1714,10 +1722,19 @@ async function legacyExportData(exportFormat, exportFields) {
         return;
     }
 
+    // Binary formats (xlsx) arrive base64-encoded; text formats as-is
+    const toBlob = (file) => {
+        if (file.encoding === 'base64') {
+            const bytes = Uint8Array.from(atob(file.content), c => c.charCodeAt(0));
+            return new Blob([bytes], { type: file.mimetype });
+        }
+        return new Blob([file.content], { type: file.mimetype });
+    };
+
     if (exportData.multiple_files && exportData.files) {
         exportData.files.forEach((file, index) => {
             setTimeout(() => {
-                const blob = new Blob([file.content], { type: file.mimetype });
+                const blob = toBlob(file);
                 const url = window.URL.createObjectURL(blob);
                 const a = document.createElement('a');
                 a.style.display = 'none';
@@ -1732,7 +1749,7 @@ async function legacyExportData(exportFormat, exportFields) {
 
         showNotification(`Exporting ${exportData.files.length} files...`, 'success');
     } else {
-        const blob = new Blob([exportData.content], { type: exportData.mimetype });
+        const blob = toBlob(exportData);
         const url = window.URL.createObjectURL(blob);
         const a = document.createElement('a');
         a.style.display = 'none';

web/static/js/settings.js

@@ -529,7 +529,11 @@ function saveSettings() {
     .then(response => response.json())
     .then(data => {
         if (!data.success) {
-            console.warn('Backend sync failed:', data.error);
+            // The server copy drives crawls started later, so a failed sync
+            // must not hide behind the "saved" notification above.
+            const reason = data.message || data.error || 'unknown error';
+            console.warn('Backend sync failed:', reason);
+            showNotification('Settings saved in this browser, but the server did not accept them: ' + reason, 'warning');
         }
 
         // Update crawler with new settings if it's running
@@ -539,6 +543,7 @@ function saveSettings() {
     })
     .catch(error => {
         console.error('Error syncing settings to backend:', error);
+        showNotification('Settings saved in this browser, but syncing them to the server failed', 'warning');
     });
 }