Initial upload

CONTRIBUTING.md

@@ -0,0 +1,58 @@
+# Contributing
+
+Thanks for considering contributing to LibreCrawl. Here's how you can help.
+
+## Ways to contribute
+
+- **Report bugs**: If something doesn't work, open an issue
+- **Suggest features**: Ideas for improvements are welcome
+- **Fix issues**: Browse open issues and submit pull requests
+- **Improve docs**: Help make the documentation clearer
+
+## Getting set up
+
+1. Fork the repository
+2. Clone your fork locally
+3. Install dependencies: `pip install -r requirements.txt`
+4. Make your changes
+5. Test that everything still works
+6. Submit a pull request
+
+## Code style
+
+- Follow existing code patterns
+- Add comments for complex logic
+- Test your changes before submitting
+
+## Reporting bugs
+
+When reporting bugs, include:
+
+- What you were trying to do
+- What happened instead
+- Steps to reproduce the issue
+- Your operating system and Python version
+- Any error messages
+
+## Suggesting features
+
+For feature requests:
+
+- Explain what you want to accomplish
+- Describe how you think it should work
+- Consider whether it fits with the project's goals
+
+## Pull requests
+
+- Keep changes focused on one thing
+- Write a clear description of what you changed
+- Include tests if applicable
+- Make sure the app still runs after your changes
+
+## Questions
+
+If you have questions about contributing, feel free to open an issue to ask.
+
+## Code of conduct
+
+Be respectful and constructive. We want this to be a welcoming project for everyone.

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 Phiality
+Copyright (c) 2024 LibreCrawl
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

README.md

@@ -1,2 +1,81 @@
 # LibreCrawl
-Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees.
+
+A desktop web crawler for SEO analysis and website auditing.
+
+## What it does
+
+LibreCrawl crawls websites and gives you detailed information about pages, links, SEO elements, and performance. It's built as a desktop app using Python and a web interface.
+
+## Features
+
+- Crawl websites with configurable depth and limits
+- Extract SEO data (titles, meta descriptions, headings, etc.)
+- Analyze internal and external links
+- Track analytics and social media tags
+- PageSpeed Insights integration
+- Export data to CSV, JSON, or XML
+- Save and load crawl sessions
+- Real-time crawling progress
+
+## Getting started
+
+### Requirements
+
+- Python 3.8 or later
+- Windows, macOS, or Linux
+
+### Installation
+
+1. Clone or download this repository
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+3. Run the application:
+```bash
+python main.py
+```
+
+The app will open in a desktop window.
+
+### Basic usage
+
+1. Enter a website URL in the input field
+2. Click "Start" to begin crawling
+3. View results in the different tabs (Overview, Internal, External, etc.)
+4. Use "Export" to save data or "Save Crawl" to resume later
+
+## Configuration
+
+Click "Settings" to configure:
+
+- **Crawler settings**: depth, URL limits, delays
+- **Request settings**: user agent, timeouts, proxy
+- **Filters**: file types to include/exclude
+- **Export options**: formats and fields to export
+
+For PageSpeed analysis, you can add a Google API key in Settings > Requests for higher rate limits.
+
+## Export formats
+
+- **CSV**: Spreadsheet-friendly format
+- **JSON**: Structured data with all details
+- **XML**: Markup format for other tools
+
+## Known limitations
+
+- PageSpeed API has rate limits (works better with API key)
+- Large sites may take time to crawl completely
+- Some dynamic content requires JavaScript (not currently supported)
+
+## Files
+
+- `main.py` - Main application and Flask server
+- `src/crawler.py` - Core crawling engine
+- `src/settings_manager.py` - Configuration management
+- `web/` - Frontend interface files
+
+## License
+
+MIT License - see LICENSE file for details.

main.py

@@ -0,0 +1,290 @@
+import webview
+import threading
+import time
+import csv
+import json
+import xml.etree.ElementTree as ET
+from io import StringIO
+from flask import Flask, render_template, request, jsonify
+from src.crawler import WebCrawler
+from src.settings_manager import SettingsManager
+
+app = Flask(__name__, template_folder='web/templates', static_folder='web/static')
+
+# Global instances
+crawler = WebCrawler()
+settings_manager = SettingsManager()
+
+def generate_csv_export(urls, fields):
+    """Generate CSV export content"""
+    output = StringIO()
+    writer = csv.DictWriter(output, fieldnames=fields)
+    writer.writeheader()
+
+    for url_data in urls:
+        row = {}
+        for field in fields:
+            value = url_data.get(field, '')
+
+            # Handle complex data types for CSV
+            if field == 'analytics' and isinstance(value, dict):
+                analytics_list = []
+                if value.get('gtag') or value.get('ga4_id'): analytics_list.append('GA4')
+                if value.get('google_analytics'): analytics_list.append('GA')
+                if value.get('gtm_id'): analytics_list.append('GTM')
+                if value.get('facebook_pixel'): analytics_list.append('FB')
+                if value.get('hotjar'): analytics_list.append('HJ')
+                if value.get('mixpanel'): analytics_list.append('MP')
+                row[field] = ', '.join(analytics_list)
+            elif field == 'og_tags' and isinstance(value, dict):
+                row[field] = f"{len(value)} tags" if value else ''
+            elif field == 'twitter_tags' and isinstance(value, dict):
+                row[field] = f"{len(value)} tags" if value else ''
+            elif field == 'json_ld' and isinstance(value, list):
+                row[field] = f"{len(value)} scripts" if value else ''
+            elif field == 'images' and isinstance(value, list):
+                row[field] = f"{len(value)} images" if value else ''
+            elif field == 'internal_links' and isinstance(value, (int, float)):
+                row[field] = f"{int(value)} internal links" if value else '0 internal links'
+            elif field == 'external_links' and isinstance(value, (int, float)):
+                row[field] = f"{int(value)} external links" if value else '0 external links'
+            elif field == 'h2' and isinstance(value, list):
+                row[field] = ', '.join(value[:3]) + ('...' if len(value) > 3 else '')
+            elif field == 'h3' and isinstance(value, list):
+                row[field] = ', '.join(value[:3]) + ('...' if len(value) > 3 else '')
+            elif isinstance(value, (dict, list)):
+                row[field] = str(value)
+            else:
+                row[field] = value
+
+        writer.writerow(row)
+
+    return output.getvalue()
+
+def generate_json_export(urls, fields):
+    """Generate JSON export content"""
+    filtered_urls = []
+    for url_data in urls:
+        filtered_data = {}
+        for field in fields:
+            value = url_data.get(field, '')
+            # Keep complex data structures intact in JSON
+            filtered_data[field] = value
+        filtered_urls.append(filtered_data)
+
+    return json.dumps({
+        'export_date': time.strftime('%Y-%m-%d %H:%M:%S'),
+        'total_urls': len(filtered_urls),
+        'fields': fields,
+        'data': filtered_urls
+    }, indent=2, default=str)
+
+def generate_xml_export(urls, fields):
+    """Generate XML export content"""
+    root = ET.Element('librecrawl_export')
+    root.set('export_date', time.strftime('%Y-%m-%d %H:%M:%S'))
+    root.set('total_urls', str(len(urls)))
+
+    urls_element = ET.SubElement(root, 'urls')
+
+    for url_data in urls:
+        url_element = ET.SubElement(urls_element, 'url')
+        for field in fields:
+            field_element = ET.SubElement(url_element, field)
+            field_element.text = str(url_data.get(field, ''))
+
+    return ET.tostring(root, encoding='unicode')
+
+def generate_links_csv_export(links):
+    """Generate CSV export for links data"""
+    output = StringIO()
+    fieldnames = ['source_url', 'target_url', 'anchor_text', 'is_internal', 'target_domain', 'target_status']
+    writer = csv.DictWriter(output, fieldnames=fieldnames)
+    writer.writeheader()
+
+    for link in links:
+        row = {
+            'source_url': link.get('source_url', ''),
+            'target_url': link.get('target_url', ''),
+            'anchor_text': link.get('anchor_text', ''),
+            'is_internal': 'Yes' if link.get('is_internal') else 'No',
+            'target_domain': link.get('target_domain', ''),
+            'target_status': link.get('target_status', 'Not crawled')
+        }
+        writer.writerow(row)
+
+    return output.getvalue()
+
+def generate_links_json_export(links):
+    """Generate JSON export for links data"""
+    return json.dumps(links, indent=2)
+
+@app.route('/')
+def index():
+    return render_template('index.html')
+
+@app.route('/api/start_crawl', methods=['POST'])
+def start_crawl():
+    data = request.get_json()
+    url = data.get('url')
+
+    if not url:
+        return jsonify({'success': False, 'error': 'URL is required'})
+
+    # Apply current settings to crawler before starting
+    try:
+        crawler_config = settings_manager.get_crawler_config()
+        crawler.update_config(crawler_config)
+    except Exception as e:
+        print(f"Warning: Could not apply settings: {e}")
+
+    success, message = crawler.start_crawl(url)
+    return jsonify({'success': success, 'message': message})
+
+@app.route('/api/stop_crawl', methods=['POST'])
+def stop_crawl():
+    success, message = crawler.stop_crawl()
+    return jsonify({'success': success, 'message': message})
+
+@app.route('/api/crawl_status')
+def crawl_status():
+    return jsonify(crawler.get_status())
+
+@app.route('/api/get_settings')
+def get_settings():
+    try:
+        settings = settings_manager.get_settings()
+        return jsonify({'success': True, 'settings': settings})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/save_settings', methods=['POST'])
+def save_settings():
+    try:
+        data = request.get_json()
+        success, message = settings_manager.save_settings(data)
+        return jsonify({'success': success, 'message': message})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/reset_settings', methods=['POST'])
+def reset_settings():
+    try:
+        success, message = settings_manager.reset_settings()
+        return jsonify({'success': success, 'message': message})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/update_crawler_settings', methods=['POST'])
+def update_crawler_settings():
+    try:
+        # Get current settings and update crawler configuration
+        crawler_config = settings_manager.get_crawler_config()
+        crawler.update_config(crawler_config)
+        return jsonify({'success': True, 'message': 'Crawler settings updated'})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/pause_crawl', methods=['POST'])
+def pause_crawl():
+    try:
+        success, message = crawler.pause_crawl()
+        return jsonify({'success': success, 'message': message})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/resume_crawl', methods=['POST'])
+def resume_crawl():
+    try:
+        success, message = crawler.resume_crawl()
+        return jsonify({'success': success, 'message': message})
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+@app.route('/api/export_data', methods=['POST'])
+def export_data():
+    try:
+        data = request.get_json()
+        export_format = data.get('format', 'csv')
+        export_fields = data.get('fields', ['url', 'status_code', 'title'])
+
+        # Get current crawl results
+        crawl_data = crawler.get_status()
+        urls = crawl_data.get('urls', [])
+        links = crawl_data.get('links', [])
+
+        if not urls:
+            return jsonify({'success': False, 'error': 'No data to export'})
+
+        # Check if links_detailed export is requested
+        if 'links_detailed' in export_fields:
+            # Export links data separately
+            if not links:
+                return jsonify({'success': False, 'error': 'No links data to export'})
+
+            if export_format == 'csv':
+                content = generate_links_csv_export(links)
+                mimetype = 'text/csv'
+                filename = f'librecrawl_links_{int(time.time())}.csv'
+            elif export_format == 'json':
+                content = generate_links_json_export(links)
+                mimetype = 'application/json'
+                filename = f'librecrawl_links_{int(time.time())}.json'
+            else:
+                content = generate_links_csv_export(links)
+                mimetype = 'text/csv'
+                filename = f'librecrawl_links_{int(time.time())}.csv'
+        else:
+            # Generate regular export content
+            if export_format == 'csv':
+                content = generate_csv_export(urls, export_fields)
+                mimetype = 'text/csv'
+                filename = f'librecrawl_export_{int(time.time())}.csv'
+            elif export_format == 'json':
+                content = generate_json_export(urls, export_fields)
+                mimetype = 'application/json'
+                filename = f'librecrawl_export_{int(time.time())}.json'
+            elif export_format == 'xml':
+                content = generate_xml_export(urls, export_fields)
+                mimetype = 'application/xml'
+                filename = f'librecrawl_export_{int(time.time())}.xml'
+            else:
+                return jsonify({'success': False, 'error': 'Unsupported export format'})
+
+        return jsonify({
+            'success': True,
+            'content': content,
+            'mimetype': mimetype,
+            'filename': filename
+        })
+
+    except Exception as e:
+        return jsonify({'success': False, 'error': str(e)})
+
+def start_flask():
+    app.run(host='127.0.0.1', port=5000, debug=False, use_reloader=False)
+
+def main():
+    # Start Flask in a separate thread
+    flask_thread = threading.Thread(target=start_flask, daemon=True)
+    flask_thread.start()
+
+    # Give Flask a moment to start
+    import time
+    time.sleep(1)
+
+    # Create webview window
+    webview.create_window(
+        title='LibreCrawl - SEO Spider',
+        url='http://127.0.0.1:5000',
+        width=1400,
+        height=900,
+        min_size=(1000, 600),
+        resizable=True
+    )
+
+    # Start webview (this will block until window is closed)
+    webview.start(debug=False)
+
+if __name__ == '__main__':
+    main()

requirements.txt

@@ -0,0 +1,5 @@
+pywebview==4.4.1
+requests==2.31.0
+beautifulsoup4==4.12.2
+urllib3==2.0.7
+flask==2.3.3

src/settings_manager.py

@@ -0,0 +1,229 @@
+import json
+import os
+from pathlib import Path
+
+class SettingsManager:
+    def __init__(self):
+        self.settings_file = Path.home() / '.librecrawl' / 'settings.json'
+        self.settings_dir = self.settings_file.parent
+
+        # Default settings
+        self.default_settings = {
+            # Crawler settings
+            'maxDepth': 3,
+            'maxUrls': 1000,
+            'crawlDelay': 1,
+            'followRedirects': True,
+            'crawlExternalLinks': False,
+
+            # Request settings
+            'userAgent': 'LibreCrawl/1.0 (Web Crawler)',
+            'timeout': 10,
+            'retries': 3,
+            'acceptLanguage': 'en-US,en;q=0.9',
+            'respectRobotsTxt': True,
+            'allowCookies': True,
+            'discoverSitemaps': True,
+            'enablePageSpeed': False,
+            'googleApiKey': '',
+
+            # Filter settings
+            'includeExtensions': 'html,htm,php,asp,aspx,jsp',
+            'excludeExtensions': 'pdf,doc,docx,zip,exe,dmg',
+            'includePatterns': '',
+            'excludePatterns': '',
+            'maxFileSize': 50,
+
+            # Export settings
+            'exportFormat': 'csv',
+            'exportFields': ['url', 'status_code', 'title', 'meta_description', 'h1'],
+
+            # Advanced settings
+            'concurrency': 5,
+            'memoryLimit': 512,
+            'logLevel': 'INFO',
+            'saveSession': False,
+            'enableProxy': False,
+            'proxyUrl': '',
+            'customHeaders': ''
+        }
+
+        self.current_settings = self.load_settings()
+
+    def ensure_settings_dir(self):
+        """Ensure the settings directory exists"""
+        self.settings_dir.mkdir(parents=True, exist_ok=True)
+
+    def load_settings(self):
+        """Load settings from file or return defaults"""
+        try:
+            if self.settings_file.exists():
+                with open(self.settings_file, 'r', encoding='utf-8') as f:
+                    saved_settings = json.load(f)
+                    # Merge with defaults to ensure all keys are present
+                    settings = {**self.default_settings}
+                    settings.update(saved_settings)
+                    return settings
+            else:
+                return self.default_settings.copy()
+        except Exception as e:
+            print(f"Error loading settings: {e}")
+            return self.default_settings.copy()
+
+    def save_settings(self, settings):
+        """Save settings to file"""
+        try:
+            self.ensure_settings_dir()
+
+            # Validate settings before saving
+            if not self.validate_settings(settings):
+                return False, "Invalid settings provided"
+
+            # Update current settings
+            self.current_settings = {**self.default_settings}
+            self.current_settings.update(settings)
+
+            # Save to file
+            with open(self.settings_file, 'w', encoding='utf-8') as f:
+                json.dump(self.current_settings, f, indent=2, ensure_ascii=False)
+
+            return True, "Settings saved successfully"
+
+        except Exception as e:
+            return False, f"Error saving settings: {str(e)}"
+
+    def get_settings(self):
+        """Get current settings"""
+        return self.current_settings.copy()
+
+    def get_setting(self, key, default=None):
+        """Get a specific setting value"""
+        return self.current_settings.get(key, default)
+
+    def update_setting(self, key, value):
+        """Update a specific setting"""
+        if key in self.default_settings:
+            self.current_settings[key] = value
+            return self.save_settings(self.current_settings)
+        return False, f"Unknown setting key: {key}"
+
+    def reset_settings(self):
+        """Reset settings to defaults"""
+        return self.save_settings(self.default_settings)
+
+    def validate_settings(self, settings):
+        """Validate settings values"""
+        try:
+            # Check required keys exist
+            for key in self.default_settings:
+                if key not in settings:
+                    return False
+
+            # Validate numeric ranges
+            numeric_validations = {
+                'maxDepth': (1, 10),
+                'maxUrls': (1, 50000),
+                'crawlDelay': (0, 60),
+                'timeout': (1, 120),
+                'retries': (0, 10),
+                'maxFileSize': (1, 1000),
+                'concurrency': (1, 50),
+                'memoryLimit': (64, 4096)
+            }
+
+            for key, (min_val, max_val) in numeric_validations.items():
+                if key in settings:
+                    value = settings[key]
+                    if not isinstance(value, (int, float)) or value < min_val or value > max_val:
+                        return False
+
+            # Validate string fields are not empty where required
+            required_strings = ['userAgent']
+            for key in required_strings:
+                if key in settings and not settings[key].strip():
+                    return False
+
+            # Validate export fields is a list
+            if 'exportFields' in settings and not isinstance(settings['exportFields'], list):
+                return False
+
+            # Validate proxy URL if proxy is enabled
+            if settings.get('enableProxy') and settings.get('proxyUrl'):
+                try:
+                    from urllib.parse import urlparse
+                    result = urlparse(settings['proxyUrl'])
+                    if not all([result.scheme, result.netloc]):
+                        return False
+                except:
+                    return False
+
+            return True
+
+        except Exception:
+            return False
+
+    def get_crawler_config(self):
+        """Get settings formatted for the crawler"""
+        settings = self.get_settings()
+
+        return {
+            'max_depth': settings['maxDepth'],
+            'max_urls': settings['maxUrls'],
+            'delay': settings['crawlDelay'],
+            'follow_redirects': settings['followRedirects'],
+            'crawl_external': settings['crawlExternalLinks'],
+            'user_agent': settings['userAgent'],
+            'timeout': settings['timeout'],
+            'retries': settings['retries'],
+            'accept_language': settings['acceptLanguage'],
+            'respect_robots': settings['respectRobotsTxt'],
+            'allow_cookies': settings['allowCookies'],
+            'include_extensions': [ext.strip() for ext in settings['includeExtensions'].split(',') if ext.strip()],
+            'exclude_extensions': [ext.strip() for ext in settings['excludeExtensions'].split(',') if ext.strip()],
+            'include_patterns': [p.strip() for p in settings['includePatterns'].split('\n') if p.strip()],
+            'exclude_patterns': [p.strip() for p in settings['excludePatterns'].split('\n') if p.strip()],
+            'max_file_size': settings['maxFileSize'] * 1024 * 1024,  # Convert MB to bytes
+            'concurrency': settings['concurrency'],
+            'memory_limit': settings['memoryLimit'] * 1024 * 1024,  # Convert MB to bytes
+            'log_level': settings['logLevel'],
+            'enable_proxy': settings['enableProxy'],
+            'proxy_url': settings['proxyUrl'] if settings['enableProxy'] else None,
+            'custom_headers': self._parse_custom_headers(settings['customHeaders']),
+            'discover_sitemaps': settings['discoverSitemaps'],
+            'enable_pagespeed': settings['enablePageSpeed'],
+            'google_api_key': settings['googleApiKey']
+        }
+
+    def _parse_custom_headers(self, headers_text):
+        """Parse custom headers from text format"""
+        headers = {}
+        if headers_text:
+            for line in headers_text.split('\n'):
+                line = line.strip()
+                if ':' in line:
+                    key, value = line.split(':', 1)
+                    headers[key.strip()] = value.strip()
+        return headers
+
+    def export_settings(self, file_path):
+        """Export settings to a file"""
+        try:
+            with open(file_path, 'w', encoding='utf-8') as f:
+                json.dump(self.current_settings, f, indent=2, ensure_ascii=False)
+            return True, "Settings exported successfully"
+        except Exception as e:
+            return False, f"Error exporting settings: {str(e)}"
+
+    def import_settings(self, file_path):
+        """Import settings from a file"""
+        try:
+            with open(file_path, 'r', encoding='utf-8') as f:
+                imported_settings = json.load(f)
+
+            if self.validate_settings(imported_settings):
+                return self.save_settings(imported_settings)
+            else:
+                return False, "Invalid settings file format"
+
+        except Exception as e:
+            return False, f"Error importing settings: {str(e)}"

web/static/js/settings.js

@@ -0,0 +1,418 @@
+// Settings Management
+let currentSettings = {};
+let defaultSettings = {
+    // Crawler settings
+    maxDepth: 3,
+    maxUrls: 1000,
+    crawlDelay: 1,
+    followRedirects: true,
+    crawlExternalLinks: false,
+
+    // Request settings
+    userAgent: 'LibreCrawl/1.0 (Web Crawler)',
+    timeout: 10,
+    retries: 3,
+    acceptLanguage: 'en-US,en;q=0.9',
+    respectRobotsTxt: true,
+    allowCookies: true,
+    discoverSitemaps: true,
+    enablePageSpeed: false,
+    googleApiKey: '',
+
+    // Filter settings
+    includeExtensions: 'html,htm,php,asp,aspx,jsp',
+    excludeExtensions: 'pdf,doc,docx,zip,exe,dmg',
+    includePatterns: '',
+    excludePatterns: '',
+    maxFileSize: 50,
+
+    // Export settings
+    exportFormat: 'csv',
+    exportFields: ['url', 'status_code', 'title', 'meta_description', 'h1', 'word_count', 'response_time', 'analytics', 'og_tags', 'json_ld', 'internal_links', 'external_links', 'images'],
+
+    // Advanced settings
+    concurrency: 5,
+    memoryLimit: 512,
+    logLevel: 'INFO',
+    saveSession: false,
+    enableProxy: false,
+    proxyUrl: '',
+    customHeaders: ''
+};
+
+// Initialize settings when page loads
+document.addEventListener('DOMContentLoaded', function() {
+    loadSettings();
+    setupSettingsEventHandlers();
+});
+
+function setupSettingsEventHandlers() {
+    // Proxy checkbox handler
+    const enableProxyCheckbox = document.getElementById('enableProxy');
+    if (enableProxyCheckbox) {
+        enableProxyCheckbox.addEventListener('change', function() {
+            const proxySettings = document.getElementById('proxySettings');
+            if (proxySettings) {
+                proxySettings.style.display = this.checked ? 'block' : 'none';
+            }
+        });
+    }
+}
+
+function openSettings() {
+    // Load current settings into form
+    populateSettingsForm();
+
+    // Show modal
+    document.getElementById('settingsModal').style.display = 'flex';
+
+    // Focus first input
+    const firstInput = document.querySelector('.settings-tab-content.active input, .settings-tab-content.active select');
+    if (firstInput) {
+        setTimeout(() => firstInput.focus(), 100);
+    }
+}
+
+function closeSettings() {
+    document.getElementById('settingsModal').style.display = 'none';
+}
+
+function switchSettingsTab(tabName) {
+    // Remove active class from all tabs and content
+    document.querySelectorAll('.settings-tab-btn').forEach(btn => {
+        btn.classList.remove('active');
+    });
+    document.querySelectorAll('.settings-tab-content').forEach(content => {
+        content.classList.remove('active');
+    });
+
+    // Add active class to selected tab and content
+    event.target.classList.add('active');
+    document.getElementById(tabName + '-settings').classList.add('active');
+}
+
+function populateSettingsForm() {
+    // Populate all form fields with current settings
+    Object.keys(currentSettings).forEach(key => {
+        const element = document.getElementById(key);
+        if (element) {
+            if (element.type === 'checkbox') {
+                element.checked = currentSettings[key];
+            } else {
+                element.value = currentSettings[key];
+            }
+        }
+    });
+
+    // Handle export fields checkboxes
+    const exportFieldsCheckboxes = document.querySelectorAll('input[name="exportFields"]');
+    exportFieldsCheckboxes.forEach(checkbox => {
+        checkbox.checked = currentSettings.exportFields.includes(checkbox.value);
+    });
+
+    // Show/hide proxy settings
+    const enableProxy = currentSettings.enableProxy;
+    const proxySettings = document.getElementById('proxySettings');
+    if (proxySettings) {
+        proxySettings.style.display = enableProxy ? 'block' : 'none';
+    }
+}
+
+function collectSettingsFromForm() {
+    const settings = {};
+
+    // Collect regular form fields
+    const formFields = [
+        'maxDepth', 'maxUrls', 'crawlDelay', 'followRedirects', 'crawlExternalLinks',
+        'userAgent', 'timeout', 'retries', 'acceptLanguage', 'respectRobotsTxt', 'allowCookies', 'discoverSitemaps', 'enablePageSpeed', 'googleApiKey',
+        'includeExtensions', 'excludeExtensions', 'includePatterns', 'excludePatterns', 'maxFileSize',
+        'exportFormat', 'concurrency', 'memoryLimit', 'logLevel', 'saveSession',
+        'enableProxy', 'proxyUrl', 'customHeaders'
+    ];
+
+    formFields.forEach(fieldId => {
+        const element = document.getElementById(fieldId);
+        if (element) {
+            if (element.type === 'checkbox') {
+                settings[fieldId] = element.checked;
+            } else if (element.type === 'number') {
+                settings[fieldId] = parseFloat(element.value) || 0;
+            } else {
+                settings[fieldId] = element.value;
+            }
+        }
+    });
+
+    // Collect export fields
+    const exportFieldsCheckboxes = document.querySelectorAll('input[name="exportFields"]:checked');
+    settings.exportFields = Array.from(exportFieldsCheckboxes).map(cb => cb.value);
+
+    return settings;
+}
+
+function saveSettings() {
+    // Collect settings from form
+    const newSettings = collectSettingsFromForm();
+
+    // Validate settings
+    const validation = validateSettings(newSettings);
+    if (!validation.valid) {
+        alert('Settings validation failed: ' + validation.errors.join(', '));
+        return;
+    }
+
+    // Save to backend
+    fetch('/api/save_settings', {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json',
+        },
+        body: JSON.stringify(newSettings)
+    })
+    .then(response => response.json())
+    .then(data => {
+        if (data.success) {
+            currentSettings = { ...newSettings };
+            closeSettings();
+            showNotification('Settings saved successfully', 'success');
+
+            // Update crawler with new settings if it's running
+            if (crawlState.isRunning) {
+                updateCrawlerSettings();
+            }
+        } else {
+            showNotification('Failed to save settings: ' + data.error, 'error');
+        }
+    })
+    .catch(error => {
+        console.error('Error saving settings:', error);
+        showNotification('Error saving settings', 'error');
+    });
+}
+
+function resetSettings() {
+    if (confirm('Are you sure you want to reset all settings to their default values?')) {
+        currentSettings = { ...defaultSettings };
+        populateSettingsForm();
+        showNotification('Settings reset to defaults', 'info');
+    }
+}
+
+function validateSettings(settings) {
+    const errors = [];
+
+    // Validate numeric ranges
+    if (settings.maxDepth < 1 || settings.maxDepth > 10) {
+        errors.push('Max depth must be between 1 and 10');
+    }
+
+    if (settings.maxUrls < 1 || settings.maxUrls > 50000) {
+        errors.push('Max URLs must be between 1 and 50,000');
+    }
+
+    if (settings.crawlDelay < 0 || settings.crawlDelay > 60) {
+        errors.push('Crawl delay must be between 0 and 60 seconds');
+    }
+
+    if (settings.timeout < 1 || settings.timeout > 120) {
+        errors.push('Timeout must be between 1 and 120 seconds');
+    }
+
+    if (settings.retries < 0 || settings.retries > 10) {
+        errors.push('Retries must be between 0 and 10');
+    }
+
+    if (settings.maxFileSize < 1 || settings.maxFileSize > 1000) {
+        errors.push('Max file size must be between 1 and 1000 MB');
+    }
+
+    if (settings.concurrency < 1 || settings.concurrency > 50) {
+        errors.push('Concurrency must be between 1 and 50');
+    }
+
+    if (settings.memoryLimit < 64 || settings.memoryLimit > 4096) {
+        errors.push('Memory limit must be between 64 and 4096 MB');
+    }
+
+    // Validate proxy URL if proxy is enabled
+    if (settings.enableProxy && settings.proxyUrl) {
+        try {
+            new URL(settings.proxyUrl);
+        } catch (e) {
+            errors.push('Invalid proxy URL format');
+        }
+    }
+
+    // Validate user agent
+    if (!settings.userAgent.trim()) {
+        errors.push('User agent cannot be empty');
+    }
+
+    // Validate export fields
+    if (settings.exportFields.length === 0) {
+        errors.push('At least one export field must be selected');
+    }
+
+    return {
+        valid: errors.length === 0,
+        errors: errors
+    };
+}
+
+function loadSettings() {
+    // Load settings from backend
+    fetch('/api/get_settings')
+        .then(response => response.json())
+        .then(data => {
+            if (data.success) {
+                currentSettings = { ...defaultSettings, ...data.settings };
+            } else {
+                console.warn('Failed to load settings, using defaults');
+                currentSettings = { ...defaultSettings };
+            }
+        })
+        .catch(error => {
+            console.error('Error loading settings:', error);
+            currentSettings = { ...defaultSettings };
+        });
+}
+
+function updateCrawlerSettings() {
+    // Send updated settings to crawler
+    fetch('/api/update_crawler_settings', {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json',
+        },
+        body: JSON.stringify(currentSettings)
+    })
+    .then(response => response.json())
+    .then(data => {
+        if (data.success) {
+            console.log('Crawler settings updated');
+        } else {
+            console.warn('Failed to update crawler settings:', data.error);
+        }
+    })
+    .catch(error => {
+        console.error('Error updating crawler settings:', error);
+    });
+}
+
+function exportSettings() {
+    // Create downloadable settings file
+    const settingsBlob = new Blob([JSON.stringify(currentSettings, null, 2)], {
+        type: 'application/json'
+    });
+
+    const url = URL.createObjectURL(settingsBlob);
+    const a = document.createElement('a');
+    a.href = url;
+    a.download = 'librecrawl-settings.json';
+    document.body.appendChild(a);
+    a.click();
+    document.body.removeChild(a);
+    URL.revokeObjectURL(url);
+}
+
+function importSettings(event) {
+    const file = event.target.files[0];
+    if (!file) return;
+
+    const reader = new FileReader();
+    reader.onload = function(e) {
+        try {
+            const importedSettings = JSON.parse(e.target.result);
+
+            // Validate imported settings
+            const validation = validateSettings(importedSettings);
+            if (!validation.valid) {
+                alert('Invalid settings file: ' + validation.errors.join(', '));
+                return;
+            }
+
+            // Merge with defaults to ensure all fields are present
+            currentSettings = { ...defaultSettings, ...importedSettings };
+            populateSettingsForm();
+            showNotification('Settings imported successfully', 'success');
+
+        } catch (error) {
+            alert('Invalid settings file format');
+        }
+    };
+    reader.readAsText(file);
+}
+
+function showNotification(message, type = 'info') {
+    // Create notification element
+    const notification = document.createElement('div');
+    notification.className = `notification notification-${type}`;
+    notification.textContent = message;
+
+    // Style the notification
+    notification.style.cssText = `
+        position: fixed;
+        top: 20px;
+        right: 20px;
+        padding: 12px 20px;
+        border-radius: 6px;
+        color: white;
+        font-weight: 500;
+        z-index: 1001;
+        max-width: 300px;
+        box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
+        transition: all 0.3s ease;
+    `;
+
+    // Set background color based on type
+    switch (type) {
+        case 'success':
+            notification.style.background = 'linear-gradient(135deg, #10b981, #059669)';
+            break;
+        case 'error':
+            notification.style.background = 'linear-gradient(135deg, #ef4444, #dc2626)';
+            break;
+        case 'warning':
+            notification.style.background = 'linear-gradient(135deg, #f59e0b, #d97706)';
+            break;
+        default:
+            notification.style.background = 'linear-gradient(135deg, #8b5cf6, #7c3aed)';
+    }
+
+    // Add to page
+    document.body.appendChild(notification);
+
+    // Remove after 3 seconds
+    setTimeout(() => {
+        notification.style.opacity = '0';
+        notification.style.transform = 'translateX(100%)';
+        setTimeout(() => {
+            if (notification.parentNode) {
+                notification.parentNode.removeChild(notification);
+            }
+        }, 300);
+    }, 3000);
+}
+
+// Close modal when clicking outside
+document.addEventListener('click', function(event) {
+    const modal = document.getElementById('settingsModal');
+    if (event.target === modal) {
+        closeSettings();
+    }
+});
+
+// Close modal with Escape key
+document.addEventListener('keydown', function(event) {
+    if (event.key === 'Escape') {
+        const modal = document.getElementById('settingsModal');
+        if (modal.style.display === 'flex') {
+            closeSettings();
+        }
+    }
+});
+
+// Export current settings object for use by other modules
+window.getCurrentSettings = function() {
+    return currentSettings;
+};

web/templates/index.html

@@ -0,0 +1,682 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>LibreCrawl - SEO Spider</title>
+    <link rel="stylesheet" href="/static/css/styles.css">
+    <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
+</head>
+<body>
+    <div class="app-container">
+        <!-- Header -->
+        <header class="header">
+            <div class="header-content">
+                <h1 class="app-title">LibreCrawl</h1>
+                <div class="header-actions">
+                    <button class="btn btn-secondary" onclick="openSettings()">Settings</button>
+                    <button class="btn btn-secondary" onclick="exportData()">Export</button>
+                    <button class="btn btn-success" id="saveCrawlBtn" onclick="saveCrawl()" disabled>Save Crawl</button>
+                    <button class="btn btn-primary" id="loadCrawlBtn" onclick="loadCrawl()">Load Crawl</button>
+                </div>
+            </div>
+        </header>
+
+        <!-- URL Input Section -->
+        <section class="url-section">
+            <div class="url-container">
+                <div class="input-group">
+                    <label for="urlInput" class="input-label">URL to Crawl</label>
+                    <input
+                        type="url"
+                        id="urlInput"
+                        class="url-input"
+                        placeholder="https://example.com"
+                        onkeypress="handleUrlKeypress(event)"
+                    >
+                </div>
+                <div class="control-buttons">
+                    <button id="startBtn" class="btn btn-primary" onclick="toggleCrawl()">
+                        <svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
+                            <path d="M8 5v14l11-7z"/>
+                        </svg>
+                        Start
+                    </button>
+                    <button id="stopBtn" class="btn btn-danger" onclick="stopCrawl()" disabled>
+                        <svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
+                            <rect x="6" y="6" width="12" height="12"/>
+                        </svg>
+                        Stop
+                    </button>
+                    <button id="clearBtn" class="btn btn-secondary" onclick="clearCrawlData()">
+                        <svg width="16" height="16" viewBox="0 0 24 24" fill="currentColor">
+                            <path d="M19 7l-.867 12.142A2 2 0 0116.138 21H7.862a2 2 0 01-1.995-1.858L5 7m5 4v6m4-6v6m1-10V4a1 1 0 00-1-1h-4a1 1 0 00-1 1v3M4 7h16"/>
+                        </svg>
+                        Clear
+                    </button>
+                </div>
+            </div>
+            <div class="progress-container" id="progressContainer" style="display: none;">
+                <div class="progress-bar">
+                    <div class="progress-fill" id="progressFill"></div>
+                </div>
+                <span class="progress-text" id="progressText">Initializing...</span>
+            </div>
+        </section>
+
+        <!-- Main Content -->
+        <main class="main-content">
+            <!-- Sidebar -->
+            <aside class="sidebar">
+                <div class="sidebar-section">
+                    <h3 class="sidebar-title">Filters</h3>
+                    <div class="filter-tree">
+                        <div class="filter-group">
+                            <div class="filter-item" onclick="toggleFilter('internal')">
+                                <span class="filter-name">Internal</span>
+                                <span class="filter-count" id="internal-count">0</span>
+                            </div>
+                            <div class="filter-item" onclick="toggleFilter('external')">
+                                <span class="filter-name">External</span>
+                                <span class="filter-count" id="external-count">0</span>
+                            </div>
+                        </div>
+
+                        <div class="filter-group">
+                            <div class="filter-header">Response Codes</div>
+                            <div class="filter-subgroup">
+                                <div class="filter-item" onclick="toggleFilter('2xx')">
+                                    <span class="filter-name">2xx Success</span>
+                                    <span class="filter-count" id="2xx-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('3xx')">
+                                    <span class="filter-name">3xx Redirect</span>
+                                    <span class="filter-count" id="3xx-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('4xx')">
+                                    <span class="filter-name">4xx Client Error</span>
+                                    <span class="filter-count" id="4xx-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('5xx')">
+                                    <span class="filter-name">5xx Server Error</span>
+                                    <span class="filter-count" id="5xx-count">0</span>
+                                </div>
+                            </div>
+                        </div>
+
+                        <div class="filter-group">
+                            <div class="filter-header">Content Type</div>
+                            <div class="filter-subgroup">
+                                <div class="filter-item" onclick="toggleFilter('html')">
+                                    <span class="filter-name">HTML</span>
+                                    <span class="filter-count" id="html-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('css')">
+                                    <span class="filter-name">CSS</span>
+                                    <span class="filter-count" id="css-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('js')">
+                                    <span class="filter-name">JavaScript</span>
+                                    <span class="filter-count" id="js-count">0</span>
+                                </div>
+                                <div class="filter-item" onclick="toggleFilter('images')">
+                                    <span class="filter-name">Images</span>
+                                    <span class="filter-count" id="images-count">0</span>
+                                </div>
+                            </div>
+                        </div>
+                    </div>
+                </div>
+
+                <div class="sidebar-section">
+                    <h3 class="sidebar-title">Statistics</h3>
+                    <div class="stats-container">
+                        <div class="stat-item">
+                            <span class="stat-label">URLs Discovered</span>
+                            <span class="stat-value" id="discoveredCount">0</span>
+                        </div>
+                        <div class="stat-item">
+                            <span class="stat-label">URLs Crawled</span>
+                            <span class="stat-value" id="crawledCount">0</span>
+                        </div>
+                        <div class="stat-item">
+                            <span class="stat-label">Crawl Depth</span>
+                            <span class="stat-value" id="crawlDepth">0</span>
+                        </div>
+                        <div class="stat-item">
+                            <span class="stat-label">Speed</span>
+                            <span class="stat-value" id="crawlSpeed">0 URLs/sec</span>
+                        </div>
+                    </div>
+                </div>
+            </aside>
+
+            <!-- Content Area -->
+            <div class="content-area">
+                <div class="tab-container">
+                    <div class="tab-header">
+                        <button class="tab-btn active" onclick="switchTab('overview')">Overview</button>
+                        <button class="tab-btn" onclick="switchTab('internal')">Internal</button>
+                        <button class="tab-btn" onclick="switchTab('external')">External</button>
+                        <button class="tab-btn" onclick="switchTab('status-codes')">Status Codes</button>
+                        <button class="tab-btn" onclick="switchTab('links')">Links</button>
+                        <button class="tab-btn" onclick="switchTab('issues')">Issues</button>
+                        <button class="tab-btn" onclick="switchTab('pagespeed')">PageSpeed</button>
+                    </div>
+
+                    <div class="tab-content">
+                        <div id="overview-tab" class="tab-pane active">
+                            <div class="table-container">
+                                <table class="data-table" id="overviewTable">
+                                    <thead>
+                                        <tr>
+                                            <th>Address</th>
+                                            <th>Status</th>
+                                            <th>Title</th>
+                                            <th>Meta Desc</th>
+                                            <th>H1</th>
+                                            <th>Words</th>
+                                            <th>Response (ms)</th>
+                                            <th>Analytics</th>
+                                            <th>OG Tags</th>
+                                            <th>JSON-LD</th>
+                                            <th>Links (Int/Ext)</th>
+                                            <th>Images</th>
+                                            <th>Details</th>
+                                        </tr>
+                                    </thead>
+                                    <tbody id="overviewTableBody">
+                                        <!-- Data will be populated here -->
+                                    </tbody>
+                                </table>
+                            </div>
+                        </div>
+
+                        <div id="internal-tab" class="tab-pane">
+                            <div class="table-container">
+                                <table class="data-table" id="internalTable">
+                                    <thead>
+                                        <tr>
+                                            <th>Address</th>
+                                            <th>Status</th>
+                                            <th>Content Type</th>
+                                            <th>Size</th>
+                                            <th>Title</th>
+                                        </tr>
+                                    </thead>
+                                    <tbody id="internalTableBody">
+                                        <!-- Data will be populated here -->
+                                    </tbody>
+                                </table>
+                            </div>
+                        </div>
+
+                        <div id="external-tab" class="tab-pane">
+                            <div class="table-container">
+                                <table class="data-table" id="externalTable">
+                                    <thead>
+                                        <tr>
+                                            <th>Address</th>
+                                            <th>Status</th>
+                                            <th>Content Type</th>
+                                            <th>Size</th>
+                                            <th>Title</th>
+                                        </tr>
+                                    </thead>
+                                    <tbody id="externalTableBody">
+                                        <!-- Data will be populated here -->
+                                    </tbody>
+                                </table>
+                            </div>
+                        </div>
+
+                        <div id="status-codes-tab" class="tab-pane">
+                            <div class="table-container">
+                                <table class="data-table" id="statusCodesTable">
+                                    <thead>
+                                        <tr>
+                                            <th>Status Code</th>
+                                            <th>Status</th>
+                                            <th>Count</th>
+                                            <th>Percentage</th>
+                                        </tr>
+                                    </thead>
+                                    <tbody id="statusCodesTableBody">
+                                        <!-- Data will be populated here -->
+                                    </tbody>
+                                </table>
+                            </div>
+                        </div>
+
+                        <div id="links-tab" class="tab-pane">
+                            <div class="table-container">
+                                <div class="links-sections">
+                                    <div class="links-section">
+                                        <h3 class="links-section-title">🔗 Internal Links</h3>
+                                        <div class="table-container">
+                                            <table class="data-table" id="internalLinksTable">
+                                                <thead>
+                                                    <tr>
+                                                        <th>Source URL</th>
+                                                        <th>Target URL</th>
+                                                        <th>Anchor Text</th>
+                                                        <th>Status</th>
+                                                    </tr>
+                                                </thead>
+                                                <tbody id="internalLinksTableBody">
+                                                    <!-- Internal links will be populated here -->
+                                                </tbody>
+                                            </table>
+                                        </div>
+                                    </div>
+
+                                    <div class="links-section">
+                                        <h3 class="links-section-title">🌐 External Links</h3>
+                                        <div class="table-container">
+                                            <table class="data-table" id="externalLinksTable">
+                                                <thead>
+                                                    <tr>
+                                                        <th>Source URL</th>
+                                                        <th>Target URL</th>
+                                                        <th>Anchor Text</th>
+                                                        <th>Domain</th>
+                                                    </tr>
+                                                </thead>
+                                                <tbody id="externalLinksTableBody">
+                                                    <!-- External links will be populated here -->
+                                                </tbody>
+                                            </table>
+                                        </div>
+                                    </div>
+                                </div>
+                            </div>
+                        </div>
+
+                        <div id="issues-tab" class="tab-pane">
+                            <div class="table-container">
+                                <table class="data-table" id="issuesTable">
+                                    <thead>
+                                        <tr>
+                                            <th>Issue Type</th>
+                                            <th>URL</th>
+                                            <th>Description</th>
+                                            <th>Severity</th>
+                                        </tr>
+                                    </thead>
+                                    <tbody id="issuesTableBody">
+                                        <!-- Issues will be populated here -->
+                                    </tbody>
+                                </table>
+                                <div class="empty-state" id="issuesEmptyState">
+                                    <div class="empty-state-content">
+                                        <svg width="64" height="64" viewBox="0 0 24 24" fill="none" stroke="currentColor">
+                                            <circle cx="12" cy="12" r="10"/>
+                                            <path d="M12 8v4M12 16h0"/>
+                                        </svg>
+                                        <h3>No Issues Found</h3>
+                                        <p>Start crawling to detect SEO issues and problems on your website.</p>
+                                    </div>
+                                </div>
+                            </div>
+                        </div>
+
+                        <div id="pagespeed-tab" class="tab-pane">
+                            <div class="pagespeed-container">
+                                <div class="pagespeed-header">
+                                    <h3>PageSpeed Insights Results</h3>
+                                    <p class="pagespeed-description">Performance analysis for homepage and category pages (powered by Google PageSpeed Insights)</p>
+                                </div>
+                                <div id="pagespeedResults" class="pagespeed-results">
+                                    <div class="pagespeed-placeholder">
+                                        <p>🚀 PageSpeed analysis will appear here after crawl completion</p>
+                                        <p>Enable PageSpeed Analysis in settings to get Core Web Vitals and performance scores</p>
+                                    </div>
+                                </div>
+                            </div>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </main>
+
+        <!-- Status Bar -->
+        <footer class="status-bar">
+            <span id="statusText">Ready</span>
+            <span id="crawlTime">00:00</span>
+        </footer>
+    </div>
+
+    <!-- Settings Modal -->
+    <div id="settingsModal" class="modal" style="display: none;">
+        <div class="modal-content">
+            <div class="modal-header">
+                <h2>Settings</h2>
+                <button class="modal-close" onclick="closeSettings()">&times;</button>
+            </div>
+            <div class="modal-body">
+                <div class="settings-tabs">
+                    <button class="settings-tab-btn active" onclick="switchSettingsTab('crawler')">Crawler</button>
+                    <button class="settings-tab-btn" onclick="switchSettingsTab('requests')">Requests</button>
+                    <button class="settings-tab-btn" onclick="switchSettingsTab('filters')">Filters</button>
+                    <button class="settings-tab-btn" onclick="switchSettingsTab('export')">Export</button>
+                    <button class="settings-tab-btn" onclick="switchSettingsTab('advanced')">Advanced</button>
+                </div>
+
+                <!-- Crawler Settings -->
+                <div id="crawler-settings" class="settings-tab-content active">
+                    <h3>Crawler Configuration</h3>
+
+                    <div class="setting-group">
+                        <label for="maxDepth">Maximum Crawl Depth</label>
+                        <input type="number" id="maxDepth" value="3" min="1" max="10">
+                        <span class="setting-help">How deep to crawl from the starting URL</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="maxUrls">Maximum URLs to Crawl</label>
+                        <input type="number" id="maxUrls" value="1000" min="1" max="50000">
+                        <span class="setting-help">Stop crawling after this many URLs</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="crawlDelay">Crawl Delay (seconds)</label>
+                        <input type="number" id="crawlDelay" value="1" min="0" max="60" step="0.1">
+                        <span class="setting-help">Delay between requests to be respectful</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="followRedirects" checked>
+                            Follow Redirects
+                        </label>
+                        <span class="setting-help">Follow 3xx redirects automatically</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="crawlExternalLinks">
+                            Crawl External Links
+                        </label>
+                        <span class="setting-help">Include external domains in crawl</span>
+                    </div>
+                </div>
+
+                <!-- Request Settings -->
+                <div id="requests-settings" class="settings-tab-content">
+                    <h3>HTTP Request Configuration</h3>
+
+                    <div class="setting-group">
+                        <label for="userAgent">User Agent</label>
+                        <input type="text" id="userAgent" value="LibreCrawl/1.0 (Web Crawler)">
+                        <span class="setting-help">User agent string sent with requests</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="timeout">Request Timeout (seconds)</label>
+                        <input type="number" id="timeout" value="10" min="1" max="120">
+                        <span class="setting-help">How long to wait for a response</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="retries">Retry Attempts</label>
+                        <input type="number" id="retries" value="3" min="0" max="10">
+                        <span class="setting-help">Number of retries for failed requests</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="acceptLanguage">Accept Language</label>
+                        <input type="text" id="acceptLanguage" value="en-US,en;q=0.9">
+                        <span class="setting-help">Preferred language for responses</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="respectRobotsTxt" checked>
+                            Respect robots.txt
+                        </label>
+                        <span class="setting-help">Check robots.txt before crawling</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="allowCookies" checked>
+                            Allow Cookies
+                        </label>
+                        <span class="setting-help">Accept and send cookies</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="discoverSitemaps" checked>
+                            Discover Sitemaps
+                        </label>
+                        <span class="setting-help">Automatically find and parse sitemap.xml files (including nested sitemaps)</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="enablePageSpeed">
+                            Enable PageSpeed Analysis
+                        </label>
+                        <span class="setting-help">Run Google PageSpeed Insights on homepage and 2 category pages after crawl completes</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="googleApiKey">Google PageSpeed API Key</label>
+                        <input type="password" id="googleApiKey" placeholder="Enter your Google PageSpeed Insights API key">
+                        <span class="setting-help">
+                            API key for higher rate limits (25k requests/day vs limited without key).
+                            <a href="https://developers.google.com/speed/docs/insights/v5/get-started" target="_blank" style="color: #8b5cf6;">Get API key here</a>
+                        </span>
+                    </div>
+                </div>
+
+                <!-- Filter Settings -->
+                <div id="filters-settings" class="settings-tab-content">
+                    <h3>Content Filters</h3>
+
+                    <div class="setting-group">
+                        <label for="includeExtensions">Include File Extensions</label>
+                        <input type="text" id="includeExtensions" value="html,htm,php,asp,aspx,jsp">
+                        <span class="setting-help">Comma-separated list of extensions to crawl</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="excludeExtensions">Exclude File Extensions</label>
+                        <input type="text" id="excludeExtensions" value="pdf,doc,docx,zip,exe,dmg">
+                        <span class="setting-help">Comma-separated list of extensions to skip</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="includePatterns">Include URL Patterns</label>
+                        <textarea id="includePatterns" rows="3" placeholder="One pattern per line (regex supported)"></textarea>
+                        <span class="setting-help">Only crawl URLs matching these patterns</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="excludePatterns">Exclude URL Patterns</label>
+                        <textarea id="excludePatterns" rows="3" placeholder="One pattern per line (regex supported)"></textarea>
+                        <span class="setting-help">Skip URLs matching these patterns</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="maxFileSize">Maximum File Size (MB)</label>
+                        <input type="number" id="maxFileSize" value="50" min="1" max="1000">
+                        <span class="setting-help">Skip files larger than this size</span>
+                    </div>
+                </div>
+
+                <!-- Export Settings -->
+                <div id="export-settings" class="settings-tab-content">
+                    <h3>Export Configuration</h3>
+
+                    <div class="setting-group">
+                        <label for="exportFormat">Default Export Format</label>
+                        <select id="exportFormat">
+                            <option value="csv">CSV</option>
+                            <option value="xlsx">Excel (XLSX)</option>
+                            <option value="json">JSON</option>
+                            <option value="xml">XML</option>
+                        </select>
+                        <span class="setting-help">Default format for data exports</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label>Export Fields</label>
+                        <div class="checkbox-grid">
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="url" checked>
+                                URL
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="status_code" checked>
+                                Status Code
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="title" checked>
+                                Title
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="meta_description" checked>
+                                Meta Description
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="h1" checked>
+                                H1 Tags
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="word_count">
+                                Word Count
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="content_type">
+                                Content Type
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="response_time">
+                                Response Time
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="canonical_url">
+                                Canonical URL
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="og_tags">
+                                OpenGraph Tags
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="twitter_tags">
+                                Twitter Cards
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="json_ld">
+                                JSON-LD Data
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="analytics">
+                                Analytics Tracking
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="internal_links">
+                                Internal Links Count
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="external_links">
+                                External Links Count
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="images">
+                                Images Count
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="links_detailed">
+                                Detailed Links Data (separate file)
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="lang">
+                                Language
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="charset">
+                                Charset
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="viewport">
+                                Viewport
+                            </label>
+                            <label class="checkbox-label">
+                                <input type="checkbox" name="exportFields" value="robots">
+                                Robots Meta
+                            </label>
+                        </div>
+                    </div>
+                </div>
+
+                <!-- Advanced Settings -->
+                <div id="advanced-settings" class="settings-tab-content">
+                    <h3>Advanced Configuration</h3>
+
+                    <div class="setting-group">
+                        <label for="concurrency">Concurrent Requests</label>
+                        <input type="number" id="concurrency" value="5" min="1" max="50">
+                        <span class="setting-help">Number of simultaneous requests (higher = faster but more resource intensive)</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="memoryLimit">Memory Limit (MB)</label>
+                        <input type="number" id="memoryLimit" value="512" min="64" max="4096">
+                        <span class="setting-help">Maximum memory usage for crawl data</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="logLevel">Log Level</label>
+                        <select id="logLevel">
+                            <option value="DEBUG">Debug</option>
+                            <option value="INFO" selected>Info</option>
+                            <option value="WARNING">Warning</option>
+                            <option value="ERROR">Error</option>
+                        </select>
+                        <span class="setting-help">Logging verbosity level</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="saveSession">
+                            Save Session Data
+                        </label>
+                        <span class="setting-help">Automatically save crawl data between sessions</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label class="checkbox-label">
+                            <input type="checkbox" id="enableProxy">
+                            Enable Proxy
+                        </label>
+                        <span class="setting-help">Route requests through a proxy server</span>
+                    </div>
+
+                    <div class="setting-group" id="proxySettings" style="display: none;">
+                        <label for="proxyUrl">Proxy URL</label>
+                        <input type="text" id="proxyUrl" placeholder="http://proxy.example.com:8080">
+                        <span class="setting-help">Proxy server URL with port</span>
+                    </div>
+
+                    <div class="setting-group">
+                        <label for="customHeaders">Custom Headers</label>
+                        <textarea id="customHeaders" rows="4" placeholder="Header-Name: Header-Value&#10;Another-Header: Another-Value"></textarea>
+                        <span class="setting-help">Additional HTTP headers to send (one per line)</span>
+                    </div>
+                </div>
+            </div>
+            <div class="modal-footer">
+                <button class="btn btn-secondary" onclick="resetSettings()">Reset to Defaults</button>
+                <button class="btn btn-secondary" onclick="closeSettings()">Cancel</button>
+                <button class="btn btn-primary" onclick="saveSettings()">Save Settings</button>
+            </div>
+        </div>
+    </div>
+
+    <script src="/static/js/app.js"></script>
+    <script src="/static/js/settings.js"></script>
+</body>
+</html>