mirror of
https://github.com/kemalai/FreeCrawl-SEO-Tool.git
synced 2026-10-18 00:39:38 +02:00
feat: manifest parse + tls cert chain + security policy
This commit is contained in:
kemalai committed
1 parent
fe35b1699f
commit
4caae06805
20 files changed
+644
-19
No files matched your search
+184
@@ -0,0 +1,184 @@
|
||||
# Security Policy
|
||||
|
||||
The FreeCrawl SEO Tool team takes security seriously. We appreciate responsible
|
||||
disclosure of vulnerabilities and aim to triage all reports promptly.
|
||||
|
||||
---
|
||||
|
||||
## Supported Versions
|
||||
|
||||
Only the latest published release on the `main` branch receives security
|
||||
fixes. Older versions are not patched — if you are on an older release,
|
||||
upgrading is the recommended remediation path.
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | ------------------ |
|
||||
| Latest stable (currently `0.3.x`) | Yes |
|
||||
| Older `0.x` releases | No |
|
||||
|
||||
Release artifacts are published at:
|
||||
<https://github.com/kemalai/FreeCrawl-SEO-Tool/releases>
|
||||
|
||||
---
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
**Please do not open public GitHub issues for security vulnerabilities.**
|
||||
Public disclosure before a fix is published puts users at risk.
|
||||
|
||||
### Preferred channel — GitHub Security Advisories (private)
|
||||
|
||||
Report privately via GitHub's built-in coordinated-disclosure flow:
|
||||
|
||||
1. Go to <https://github.com/kemalai/FreeCrawl-SEO-Tool/security/advisories>
|
||||
2. Click **"Report a vulnerability"**
|
||||
3. Fill in the advisory form with reproduction steps and impact
|
||||
|
||||
This creates a private discussion visible only to the maintainers. We will
|
||||
coordinate a fix, prepare a CVE if applicable, and credit the reporter
|
||||
(unless you request anonymity) when the patched release goes public.
|
||||
|
||||
### Alternative — Email
|
||||
|
||||
If you cannot use GitHub Security Advisories, email the maintainer at:
|
||||
|
||||
**`kemalacarofficial@gmail.com`**
|
||||
|
||||
Use the subject prefix `[FreeCrawl Security]` so the report is routed
|
||||
correctly. PGP encryption is not currently set up; do not include
|
||||
exploit code in plain email — link to a private gist or password-protect
|
||||
an attachment instead.
|
||||
|
||||
---
|
||||
|
||||
## What to Include in a Report
|
||||
|
||||
To help us triage quickly, please include as much of the following as
|
||||
possible:
|
||||
|
||||
- **Affected component** — crawler engine (`@freecrawl/core`), database
|
||||
layer (`@freecrawl/db`), desktop shell, MCP server, or CLI
|
||||
- **Affected version(s)** — output of `Help → About`, or the `version`
|
||||
field in `package.json`
|
||||
- **Operating system + architecture** — Windows 11 / macOS 14 (Intel or
|
||||
ARM64) / Linux distro
|
||||
- **Reproduction steps** — minimal sequence that triggers the issue
|
||||
- **Proof-of-concept** — screenshot, video, or minimal code
|
||||
- **Impact assessment** — confidentiality / integrity / availability
|
||||
exposure, privilege escalation, data exfiltration, etc.
|
||||
- **Suggested fix** if you have one (entirely optional)
|
||||
|
||||
---
|
||||
|
||||
## Response Timeline
|
||||
|
||||
We aim for the following timeline. Real-world response times may vary
|
||||
because the project is currently maintained by a single contributor.
|
||||
|
||||
| Stage | Target time |
|
||||
| ---------------------- | ---------------- |
|
||||
| Initial acknowledgement | within 5 business days |
|
||||
| Triage + severity assessment | within 14 business days |
|
||||
| Patch development | severity-dependent (critical: ≤ 30 days, high: ≤ 60 days, medium: ≤ 90 days) |
|
||||
| Public disclosure | coordinated with reporter — typically after a patched release is published |
|
||||
|
||||
If you do not receive an acknowledgement within 7 business days, please
|
||||
follow up — your initial report may have been lost in spam filtering.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
|
||||
The following are considered valid security issues:
|
||||
|
||||
- **Remote code execution** in the crawler, parser, or MCP server
|
||||
triggered by malicious server responses (HTML, sitemap XML, robots.txt,
|
||||
HTTP headers, redirect chains, manifest JSON, etc.)
|
||||
- **Path traversal / arbitrary file write** through project-file
|
||||
(`.seoproject`) handling, export paths, or import flows
|
||||
- **SQL injection** in the SQLite layer (`@freecrawl/db`) — all user
|
||||
input must flow through prepared statements
|
||||
- **Cross-site scripting (XSS)** in the renderer process when displaying
|
||||
crawled content (titles, anchor text, meta descriptions, custom
|
||||
extraction results)
|
||||
- **Sandbox escape** from the renderer to the main process bypassing
|
||||
contextBridge isolation
|
||||
- **Credential leakage** — API keys, project paths, or crawl data
|
||||
accidentally written to logs, telemetry, or remote services
|
||||
- **Denial-of-service** on a normal site crawl that the user could
|
||||
reasonably expect to complete (e.g. catastrophic memory growth on a
|
||||
modestly-sized site, parser infinite loops, deadlocks under normal
|
||||
concurrency)
|
||||
- **Supply-chain compromise** of distributed installers or the GitHub
|
||||
Actions release workflow
|
||||
|
||||
### Out of scope
|
||||
|
||||
- **DoS via clearly-malicious target sites** — if a user manually points
|
||||
the crawler at a hostile server, transient resource exhaustion is
|
||||
expected; report only if it persists after the crawl is stopped
|
||||
- **Issues that require physical access** to the user's machine
|
||||
- **Self-XSS** that requires the user to paste attacker-controlled
|
||||
content into devtools
|
||||
- **Reports based on outdated versions** without a current-version
|
||||
reproduction
|
||||
- **Best-practice / hardening suggestions without a concrete impact**
|
||||
(e.g. "you should add a CSP header") — please open a regular GitHub
|
||||
issue or pull request instead
|
||||
- **Vulnerabilities in third-party dependencies** that are already
|
||||
publicly disclosed and have an upstream patch in flight — we
|
||||
appreciate the heads-up, but the report should ideally be filed with
|
||||
the upstream project; we will pull in patched versions on the next
|
||||
routine release
|
||||
- **Social-engineering attacks** against the maintainer
|
||||
|
||||
---
|
||||
|
||||
## Distribution & Code-Signing
|
||||
|
||||
FreeCrawl SEO Tool releases are distributed exclusively through GitHub
|
||||
Releases. The artifacts are produced by an automated GitHub Actions
|
||||
workflow defined in `.github/workflows/release.yml`. SHA512 digests are
|
||||
published alongside each release in `latest.yml` / `latest-mac.yml`.
|
||||
|
||||
Code-signing for Windows installers is being established through
|
||||
[SignPath.io](https://signpath.io)'s open-source signing program. Once
|
||||
active, all `.exe` artifacts will be signed and the application will
|
||||
verify the publisher signature before applying any in-app update. macOS
|
||||
artifacts are currently unsigned; users should verify the SHA512 digest
|
||||
manually until Apple Developer ID notarization is in place.
|
||||
|
||||
If you encounter an installer that fails signature verification or
|
||||
appears to come from any source other than the official GitHub Releases
|
||||
page above, **do not run it** — please report it via the channels above.
|
||||
|
||||
---
|
||||
|
||||
## Cryptographic Material & Secrets
|
||||
|
||||
- **No secrets are bundled in the application.** API keys for optional
|
||||
third-party integrations (Search Console, Analytics, AI providers,
|
||||
etc.) are stored locally via the OS-native keychain when configured by
|
||||
the user, and are never transmitted to FreeCrawl-controlled servers.
|
||||
- **No telemetry is collected** — the application does not phone home,
|
||||
beacon, or report usage data. The only outbound network requests are
|
||||
the crawls the user explicitly initiates and (when enabled) the
|
||||
on-demand "Check for Updates" call to the GitHub Releases API.
|
||||
- **Project files (`.seoproject`)** are SQLite databases containing
|
||||
crawled content. They are not encrypted at rest in the current
|
||||
release — users handling sensitive data should rely on full-disk
|
||||
encryption at the OS level.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
We are happy to credit security researchers who report valid issues
|
||||
through the channels above, both in the affected release's CHANGELOG
|
||||
and (for higher-severity issues) in the published GitHub Security
|
||||
Advisory. If you prefer to remain anonymous, please mention that in
|
||||
your report.
|
||||
|
||||
Thank you for helping keep FreeCrawl SEO Tool and its users safe.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/cli",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/desktop",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"description": "FreeCrawl SEO — desktop application",
|
||||
"main": "./out/main/index.js",
|
||||
|
||||
@@ -2134,6 +2134,8 @@ function registerIpc(): void {
|
||||
subject: null,
|
||||
signatureAlgorithm: null,
|
||||
protocol: null,
|
||||
chainLength: null,
|
||||
chainSubjects: null,
|
||||
probeStatus: -1,
|
||||
probeError: null,
|
||||
probedAt: null,
|
||||
|
||||
@@ -861,6 +861,14 @@ function NameValueView({ row }: { row: CrawlUrlRow }) {
|
||||
['TLS Cert Issuer', cert?.issuer ?? null],
|
||||
['TLS Cert Subject', cert?.subject ?? null],
|
||||
['TLS Cert Signature Alg', cert?.signatureAlgorithm ?? null],
|
||||
[
|
||||
'TLS Cert Chain',
|
||||
cert?.chainLength
|
||||
? cert.chainSubjects && cert.chainSubjects.length > 0
|
||||
? `${cert.chainLength} cert(s): ${cert.chainSubjects.join(' ← ')}`
|
||||
: `${cert.chainLength} cert(s)`
|
||||
: null,
|
||||
],
|
||||
['TLS Cert Valid From', cert?.validFrom ?? null],
|
||||
['TLS Cert Valid To', cert?.validTo ?? null],
|
||||
['TLS Cert Status', certExpiryLabel],
|
||||
@@ -903,6 +911,11 @@ function NameValueView({ row }: { row: CrawlUrlRow }) {
|
||||
['Apple Touch Icon', row.appleTouchIcon],
|
||||
['Android / PWA Icon', row.androidIcon],
|
||||
['Web Manifest', row.manifestUrl],
|
||||
['Manifest Theme Color', row.manifestThemeColor],
|
||||
['Manifest Short Name', row.manifestShortName],
|
||||
['Manifest Display Mode', row.manifestDisplay],
|
||||
['Manifest Scope', row.manifestScope],
|
||||
['Manifest Icon Count', row.manifestIconCount > 0 ? row.manifestIconCount : null],
|
||||
['RSS / Atom Feed', row.feedUrl],
|
||||
['Title Tag Count', row.titleCount > 1 ? row.titleCount : null],
|
||||
['Empty-Alt Images', row.imagesEmptyAlt > 0 ? row.imagesEmptyAlt : null],
|
||||
|
||||
@@ -85,6 +85,7 @@ interface FormState {
|
||||
// duplicates
|
||||
nearDuplicateHammingThreshold: string;
|
||||
duplicatesOnlyIndexable: boolean;
|
||||
dedupePreNormalize: boolean;
|
||||
// custom extraction
|
||||
customExtractionRules: CustomExtractionRule[];
|
||||
// webhook
|
||||
@@ -319,6 +320,7 @@ function configToForm(c: CrawlConfig): FormState {
|
||||
processPriority: c.processPriority,
|
||||
nearDuplicateHammingThreshold: String(c.nearDuplicateHammingThreshold),
|
||||
duplicatesOnlyIndexable: c.duplicatesOnlyIndexable,
|
||||
dedupePreNormalize: c.dedupePreNormalize ?? true,
|
||||
customExtractionRules: (c.customExtractionRules ?? []).map((r) => ({ ...r })),
|
||||
webhookUrl: c.webhookUrl ?? '',
|
||||
auth: { ...(c.auth ?? { type: 'none' }) },
|
||||
@@ -474,6 +476,7 @@ export function SettingsDialog({ open, onClose }: Props) {
|
||||
),
|
||||
),
|
||||
duplicatesOnlyIndexable: form.duplicatesOnlyIndexable,
|
||||
dedupePreNormalize: form.dedupePreNormalize,
|
||||
customExtractionRules: form.customExtractionRules
|
||||
.filter((r) => r.name.trim() && r.selector.trim())
|
||||
.slice(0, 10),
|
||||
@@ -1919,6 +1922,13 @@ function DuplicatesPanel({ form, update }: PanelProps) {
|
||||
info="When on, pages with noindex / canonicalised / robots-blocked indexability are excluded from clustering — the Near-Duplicate report then surfaces only issues that affect search visibility."
|
||||
example="ON for SEO audits (the typical case). Turn OFF to also cluster paginated / canonical-blocked variants for completeness."
|
||||
/>
|
||||
<Bool
|
||||
label="Normalise URLs before duplicate-URL check"
|
||||
checked={form.dedupePreNormalize}
|
||||
onChange={(v) => update('dedupePreNormalize', v)}
|
||||
info="When ON (default), the Duplicate URL filter compares URLs after lowercasing the host, dropping the query string, and trimming the trailing slash — the canonical SEO behaviour. When OFF, comparison is byte-exact, so the filter only fires on rows that share an identical raw URL string (rare since URLs are deduped at insert time)."
|
||||
example="ON for SEO audits. OFF only when you specifically need to inspect raw-URL collisions (e.g. case-sensitive filesystem CMSes)."
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="rounded border border-surface-800 bg-surface-950/40 p-3 text-[10px] text-surface-500">
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/mcp-server",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
Generated
+8
-8
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "freecrawl-seo",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "freecrawl-seo",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"license": "MIT",
|
||||
"workspaces": [
|
||||
"apps/*",
|
||||
@@ -25,7 +25,7 @@
|
||||
},
|
||||
"apps/cli": {
|
||||
"name": "@freecrawl/cli",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"dependencies": {
|
||||
"@freecrawl/core": "*",
|
||||
"@freecrawl/db": "*",
|
||||
@@ -42,7 +42,7 @@
|
||||
},
|
||||
"apps/desktop": {
|
||||
"name": "@freecrawl/desktop",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"dependencies": {
|
||||
"@freecrawl/core": "*",
|
||||
"@freecrawl/db": "*",
|
||||
@@ -84,7 +84,7 @@
|
||||
},
|
||||
"apps/mcp-server": {
|
||||
"name": "@freecrawl/mcp-server",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"dependencies": {
|
||||
"@freecrawl/db": "*",
|
||||
"@freecrawl/shared-types": "*"
|
||||
@@ -8732,7 +8732,7 @@
|
||||
},
|
||||
"packages/core": {
|
||||
"name": "@freecrawl/core",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"dependencies": {
|
||||
"@freecrawl/db": "*",
|
||||
"@freecrawl/shared-types": "*",
|
||||
@@ -8758,7 +8758,7 @@
|
||||
},
|
||||
"packages/db": {
|
||||
"name": "@freecrawl/db",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"dependencies": {
|
||||
"@freecrawl/shared-types": "*"
|
||||
},
|
||||
@@ -8770,7 +8770,7 @@
|
||||
},
|
||||
"packages/shared-types": {
|
||||
"name": "@freecrawl/shared-types",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"devDependencies": {
|
||||
"rimraf": "^6.0.1",
|
||||
"typescript": "^5.7.0"
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "freecrawl-seo",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"description": "Open-source desktop SEO crawler — a free alternative to Screaming Frog",
|
||||
"license": "MIT",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/core",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -658,6 +658,9 @@ export class Crawler extends EventEmitter {
|
||||
await yieldToEventLoop();
|
||||
this.setOp('post-crawl:tls-probes');
|
||||
await this.runTlsCertProbes();
|
||||
await yieldToEventLoop();
|
||||
this.setOp('post-crawl:manifest-probes');
|
||||
await this.runManifestProbes();
|
||||
this.running = false;
|
||||
this.stopMemoryMonitor();
|
||||
// Release per-URL dedup sets — at 1M URLs this is ~80–120 MB of string
|
||||
@@ -838,6 +841,76 @@ export class Crawler extends EventEmitter {
|
||||
* crawls have at most a few unique hosts and the cost is dominated by
|
||||
* the handshake round-trip, not throughput.
|
||||
*/
|
||||
/**
|
||||
* Post-crawl pass: GET each declared `<link rel="manifest">` URL once
|
||||
* and stamp parsed fields onto every URL that referenced it. Manifest
|
||||
* URLs are usually site-wide singletons (one fetch covers thousands
|
||||
* of pages) so concurrency is small. Failures store an error string
|
||||
* in the per-URL `manifest_json` column so the next crawl re-probes.
|
||||
*/
|
||||
private async runManifestProbes(): Promise<void> {
|
||||
if (!this.config.probeManifestJson) return;
|
||||
if (this.stopped) return;
|
||||
let urls: string[] = [];
|
||||
try {
|
||||
urls = this.db.unprobedManifestUrls(2_000);
|
||||
} catch (err) {
|
||||
this.emit(
|
||||
'info',
|
||||
`manifest probe skipped (DB query failed): ${
|
||||
err instanceof Error ? err.message : String(err)
|
||||
}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (urls.length === 0) return;
|
||||
this.emit('info', `Probing ${urls.length} web manifest(s)…`);
|
||||
|
||||
const { probeManifest } = await import('./manifest-probe.js');
|
||||
const concurrency = 4;
|
||||
let cursor = 0;
|
||||
let probed = 0;
|
||||
let failed = 0;
|
||||
|
||||
const worker = async (): Promise<void> => {
|
||||
while (!this.stopped) {
|
||||
const idx = cursor++;
|
||||
if (idx >= urls.length) return;
|
||||
const url = urls[idx];
|
||||
if (!url) return;
|
||||
try {
|
||||
const result = await probeManifest(url, {
|
||||
userAgent: this.resolveUserAgent(url),
|
||||
acceptLanguage: this.config.acceptLanguage,
|
||||
customHeaders: this.config.customHeaders,
|
||||
auth: this.config.auth,
|
||||
});
|
||||
this.db.setManifestForReferrers({
|
||||
manifestUrl: url,
|
||||
rawJson: result.error ? null : result.rawJson,
|
||||
themeColor: result.themeColor,
|
||||
shortName: result.shortName,
|
||||
display: result.display,
|
||||
scope: result.scope,
|
||||
iconCount: result.iconCount,
|
||||
});
|
||||
if (result.error) failed++;
|
||||
probed++;
|
||||
} catch {
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const workers: Promise<void>[] = [];
|
||||
for (let i = 0; i < concurrency; i++) workers.push(worker());
|
||||
await Promise.all(workers);
|
||||
this.emit(
|
||||
'info',
|
||||
`Manifest probe complete: ${probed - failed}/${probed} parsed, ${failed} failed`,
|
||||
);
|
||||
}
|
||||
|
||||
private async runTlsCertProbes(): Promise<void> {
|
||||
if (!this.config.probeTlsCerts) return;
|
||||
if (this.stopped) return;
|
||||
@@ -888,6 +961,8 @@ export class Crawler extends EventEmitter {
|
||||
subject: info.subject,
|
||||
signatureAlgorithm: info.signatureAlgorithm,
|
||||
protocol: info.protocol,
|
||||
chainLength: info.chainLength,
|
||||
chainSubjects: info.chainSubjects,
|
||||
probeStatus: info.error ? 0 : 200,
|
||||
probeError: info.error,
|
||||
});
|
||||
@@ -908,6 +983,8 @@ export class Crawler extends EventEmitter {
|
||||
subject: null,
|
||||
signatureAlgorithm: null,
|
||||
protocol: null,
|
||||
chainLength: null,
|
||||
chainSubjects: null,
|
||||
probeStatus: 0,
|
||||
probeError: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
import { fetch as undiciFetch } from 'undici';
|
||||
import { defaultRequestHeaders } from './http-client.js';
|
||||
import type { CrawlConfig } from '@freecrawl/shared-types';
|
||||
|
||||
/**
|
||||
* Parsed Web App Manifest fields surfaced for the Detail panel + audits.
|
||||
* Captures the most-frequently-checked PWA / install metadata; the raw
|
||||
* JSON body is preserved separately so power users can read every
|
||||
* custom field without us needing a column for each.
|
||||
*/
|
||||
export interface ManifestProbeResult {
|
||||
/** Raw JSON body, capped at 4 KB to keep the column tiny on hostile manifests. */
|
||||
rawJson: string | null;
|
||||
themeColor: string | null;
|
||||
shortName: string | null;
|
||||
display: string | null;
|
||||
scope: string | null;
|
||||
iconCount: number;
|
||||
/** Free-form error string when the fetch / parse fails. Null on success. */
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
const RAW_JSON_CAP = 4096;
|
||||
const PROBE_TIMEOUT_MS = 8_000;
|
||||
|
||||
/**
|
||||
* Fetch a Web App Manifest URL and surface a small set of canonical
|
||||
* fields. We deliberately keep this best-effort:
|
||||
* - JSON parse failures don't throw — caller stores `error`
|
||||
* - 4xx / 5xx still resolve with `error` set so the result can be
|
||||
* persisted (otherwise we'd re-probe on every post-crawl pass)
|
||||
* - Body cap of 4 KB protects against accidentally-streamed huge
|
||||
* manifests (some CMSes proxy a 5 MB JSON-LD by mistake)
|
||||
*/
|
||||
export async function probeManifest(
|
||||
manifestUrl: string,
|
||||
config: Pick<CrawlConfig, 'userAgent' | 'acceptLanguage' | 'customHeaders' | 'auth'>,
|
||||
): Promise<ManifestProbeResult> {
|
||||
const empty: ManifestProbeResult = {
|
||||
rawJson: null,
|
||||
themeColor: null,
|
||||
shortName: null,
|
||||
display: null,
|
||||
scope: null,
|
||||
iconCount: 0,
|
||||
error: null,
|
||||
};
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), PROBE_TIMEOUT_MS);
|
||||
try {
|
||||
const res = await undiciFetch(manifestUrl, {
|
||||
method: 'GET',
|
||||
headers: defaultRequestHeaders(
|
||||
config.userAgent,
|
||||
config.acceptLanguage,
|
||||
config.customHeaders,
|
||||
config.auth,
|
||||
),
|
||||
redirect: 'follow',
|
||||
signal: controller.signal,
|
||||
});
|
||||
if (!res.ok) {
|
||||
return { ...empty, error: `HTTP ${res.status}` };
|
||||
}
|
||||
const text = await res.text();
|
||||
const truncated = text.slice(0, RAW_JSON_CAP);
|
||||
let parsed: Record<string, unknown> | null = null;
|
||||
try {
|
||||
const body = JSON.parse(text) as unknown;
|
||||
if (body && typeof body === 'object' && !Array.isArray(body)) {
|
||||
parsed = body as Record<string, unknown>;
|
||||
} else {
|
||||
return { ...empty, rawJson: truncated, error: 'manifest is not a JSON object' };
|
||||
}
|
||||
} catch (err) {
|
||||
return {
|
||||
...empty,
|
||||
rawJson: truncated,
|
||||
error: `JSON parse error: ${err instanceof Error ? err.message : String(err)}`,
|
||||
};
|
||||
}
|
||||
const stringField = (k: string): string | null => {
|
||||
const v = parsed![k];
|
||||
return typeof v === 'string' && v.trim() ? v.trim() : null;
|
||||
};
|
||||
const icons = parsed['icons'];
|
||||
return {
|
||||
rawJson: truncated,
|
||||
themeColor: stringField('theme_color'),
|
||||
shortName: stringField('short_name'),
|
||||
display: stringField('display'),
|
||||
scope: stringField('scope'),
|
||||
iconCount: Array.isArray(icons) ? icons.length : 0,
|
||||
error: null,
|
||||
};
|
||||
} catch (err) {
|
||||
return {
|
||||
...empty,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,20 @@ export interface TlsCertInfo {
|
||||
signatureAlgorithm: string | null;
|
||||
/** Negotiated TLS protocol version, e.g. `TLSv1.3`. */
|
||||
protocol: string | null;
|
||||
/**
|
||||
* Number of certificates the server presented in its chain (leaf
|
||||
* inclusive). 1 means leaf-only / self-signed; 2 means leaf + root
|
||||
* with no intermediate; ≥3 indicates one or more intermediate CAs.
|
||||
* Null on probe failure.
|
||||
*/
|
||||
chainLength: number | null;
|
||||
/**
|
||||
* Distinguished names of the certs in the chain, leaf-first. Capped
|
||||
* at 5 entries to keep the column tiny — corporate PKIs occasionally
|
||||
* present long chains and we don't need the full path for the audit.
|
||||
* Null on probe failure.
|
||||
*/
|
||||
chainSubjects: string[] | null;
|
||||
/** Empty on success. Free-form error string when the connect/handshake fails. */
|
||||
error: string | null;
|
||||
}
|
||||
@@ -44,6 +58,8 @@ export async function probeTlsCert(
|
||||
subject: null,
|
||||
signatureAlgorithm: null,
|
||||
protocol: null,
|
||||
chainLength: null,
|
||||
chainSubjects: null,
|
||||
error: null,
|
||||
};
|
||||
|
||||
@@ -76,14 +92,37 @@ export async function probeTlsCert(
|
||||
() => {
|
||||
clearTimeout(timer);
|
||||
try {
|
||||
// `getPeerCertificate(false)` returns just the leaf cert — what
|
||||
// we need for expiry/issuer. `(true)` would walk the chain but
|
||||
// we don't surface chain info in V1.
|
||||
const peer = socket.getPeerCertificate(false) as PeerCertificate;
|
||||
// `getPeerCertificate(true)` returns the leaf cert with its
|
||||
// `issuerCertificate` chain attached so we can surface chain
|
||||
// length + intermediate CA subjects alongside the leaf
|
||||
// expiry/issuer/signature data.
|
||||
const peer = socket.getPeerCertificate(true) as PeerCertificate;
|
||||
if (!peer || Object.keys(peer).length === 0) {
|
||||
finish({ ...empty, error: 'peer certificate unavailable' });
|
||||
return;
|
||||
}
|
||||
// Walk the chain via `issuerCertificate` until we hit a
|
||||
// self-signed cert (issuerCertificate === self) or run out.
|
||||
// Capped at 5 hops to defend against weird PKIs that loop.
|
||||
const chainSubjects: string[] = [];
|
||||
let chainLength = 0;
|
||||
{
|
||||
let cursor: PeerCertificate | undefined = peer;
|
||||
const seen = new Set<PeerCertificate>();
|
||||
while (cursor && !seen.has(cursor) && chainLength < 5) {
|
||||
seen.add(cursor);
|
||||
chainLength++;
|
||||
const sub = formatDistinguishedName(
|
||||
(cursor as { subject?: Record<string, string> }).subject ?? null,
|
||||
);
|
||||
if (sub) chainSubjects.push(sub);
|
||||
const next: PeerCertificate | undefined = (
|
||||
cursor as { issuerCertificate?: PeerCertificate }
|
||||
).issuerCertificate;
|
||||
if (!next || next === cursor) break;
|
||||
cursor = next;
|
||||
}
|
||||
}
|
||||
const validTo = (peer as { valid_to?: string }).valid_to ?? null;
|
||||
const validFrom = (peer as { valid_from?: string }).valid_from ?? null;
|
||||
let daysUntilExpiry: number | null = null;
|
||||
@@ -111,6 +150,8 @@ export async function probeTlsCert(
|
||||
(peer as { signatureAlgorithm?: string }).signatureAlgorithm ??
|
||||
null,
|
||||
protocol,
|
||||
chainLength,
|
||||
chainSubjects: chainSubjects.length > 0 ? chainSubjects : null,
|
||||
error: null,
|
||||
});
|
||||
} catch (err) {
|
||||
|
||||
@@ -100,6 +100,11 @@ const XML_COLUMNS: (keyof CrawlUrlRow)[] = [
|
||||
'appleTouchIcon',
|
||||
'androidIcon',
|
||||
'manifestUrl',
|
||||
'manifestThemeColor',
|
||||
'manifestShortName',
|
||||
'manifestDisplay',
|
||||
'manifestScope',
|
||||
'manifestIconCount',
|
||||
'feedUrl',
|
||||
'metaRefresh',
|
||||
'metaRefreshUrl',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/db",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1082,6 +1082,60 @@ const MIGRATIONS: Migration[] = [
|
||||
db.exec('ALTER TABLE urls ADD COLUMN android_icon TEXT');
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 51,
|
||||
name: 'add_manifest_json_and_cert_chain_columns',
|
||||
// V1 Faz 1 final bundle:
|
||||
// urls.manifest_json — raw parsed manifest JSON (string,
|
||||
// ~2 KB cap recommended at write time)
|
||||
// so the Detail panel can show every
|
||||
// custom field without us shaping
|
||||
// a column for each.
|
||||
// urls.manifest_theme_color — `theme_color` shorthand for the
|
||||
// Stack / Server color tinting in
|
||||
// the URL Details panel.
|
||||
// urls.manifest_short_name — `short_name` (home-screen label).
|
||||
// urls.manifest_display — `display` (e.g. `standalone`).
|
||||
// urls.manifest_scope — `scope` (PWA install scope URL).
|
||||
// urls.manifest_icon_count — number of icons in the parsed manifest.
|
||||
//
|
||||
// host_certs.chain_length — total certificates in the peer-cert
|
||||
// chain (1 = self-signed, 2 = leaf
|
||||
// + root, ≥3 = intermediate(s) +
|
||||
// root). Surfaces partial / missing
|
||||
// chain misconfigurations.
|
||||
// host_certs.chain_subjects — JSON-stringified subject DN list,
|
||||
// root-first order (max 5 hops).
|
||||
up: (db) => {
|
||||
const urlCols = db.prepare('PRAGMA table_info(urls)').all() as unknown as {
|
||||
name: string;
|
||||
}[];
|
||||
const hasUrl = (n: string) => urlCols.some((c) => c.name === n);
|
||||
if (!hasUrl('manifest_json'))
|
||||
db.exec('ALTER TABLE urls ADD COLUMN manifest_json TEXT');
|
||||
if (!hasUrl('manifest_theme_color'))
|
||||
db.exec('ALTER TABLE urls ADD COLUMN manifest_theme_color TEXT');
|
||||
if (!hasUrl('manifest_short_name'))
|
||||
db.exec('ALTER TABLE urls ADD COLUMN manifest_short_name TEXT');
|
||||
if (!hasUrl('manifest_display'))
|
||||
db.exec('ALTER TABLE urls ADD COLUMN manifest_display TEXT');
|
||||
if (!hasUrl('manifest_scope'))
|
||||
db.exec('ALTER TABLE urls ADD COLUMN manifest_scope TEXT');
|
||||
if (!hasUrl('manifest_icon_count'))
|
||||
db.exec(
|
||||
'ALTER TABLE urls ADD COLUMN manifest_icon_count INTEGER NOT NULL DEFAULT 0',
|
||||
);
|
||||
|
||||
const certCols = db
|
||||
.prepare('PRAGMA table_info(host_certs)')
|
||||
.all() as unknown as { name: string }[];
|
||||
const hasCert = (n: string) => certCols.some((c) => c.name === n);
|
||||
if (!hasCert('chain_length'))
|
||||
db.exec('ALTER TABLE host_certs ADD COLUMN chain_length INTEGER');
|
||||
if (!hasCert('chain_subjects'))
|
||||
db.exec('ALTER TABLE host_certs ADD COLUMN chain_subjects TEXT');
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export function runMigrations(db: DatabaseSync): void {
|
||||
|
||||
@@ -153,6 +153,12 @@ interface UrlRowDb {
|
||||
og_site_name: string | null;
|
||||
og_locale: string | null;
|
||||
android_icon: string | null;
|
||||
manifest_json: string | null;
|
||||
manifest_theme_color: string | null;
|
||||
manifest_short_name: string | null;
|
||||
manifest_display: string | null;
|
||||
manifest_scope: string | null;
|
||||
manifest_icon_count: number;
|
||||
}
|
||||
|
||||
interface ImageRowDb {
|
||||
@@ -3749,6 +3755,8 @@ export class ProjectDb {
|
||||
subject: string | null;
|
||||
signatureAlgorithm: string | null;
|
||||
protocol: string | null;
|
||||
chainLength?: number | null;
|
||||
chainSubjects?: string[] | null;
|
||||
probeStatus: number;
|
||||
probeError: string | null;
|
||||
}): void {
|
||||
@@ -3757,10 +3765,12 @@ export class ProjectDb {
|
||||
`INSERT INTO host_certs (
|
||||
host, port, valid_from, valid_to, days_until_expiry,
|
||||
issuer, subject, signature_algorithm, protocol,
|
||||
chain_length, chain_subjects,
|
||||
probe_status, probe_error, probed_at
|
||||
) VALUES (
|
||||
:host, :port, :valid_from, :valid_to, :days_until_expiry,
|
||||
:issuer, :subject, :signature_algorithm, :protocol,
|
||||
:chain_length, :chain_subjects,
|
||||
:probe_status, :probe_error, CURRENT_TIMESTAMP
|
||||
)
|
||||
ON CONFLICT(host) DO UPDATE SET
|
||||
@@ -3771,6 +3781,8 @@ export class ProjectDb {
|
||||
subject = excluded.subject,
|
||||
signature_algorithm = excluded.signature_algorithm,
|
||||
protocol = excluded.protocol,
|
||||
chain_length = excluded.chain_length,
|
||||
chain_subjects = excluded.chain_subjects,
|
||||
probe_status = excluded.probe_status,
|
||||
probe_error = excluded.probe_error,
|
||||
probed_at = CURRENT_TIMESTAMP`,
|
||||
@@ -3785,11 +3797,74 @@ export class ProjectDb {
|
||||
subject: input.subject,
|
||||
signature_algorithm: input.signatureAlgorithm,
|
||||
protocol: input.protocol,
|
||||
chain_length: input.chainLength ?? null,
|
||||
chain_subjects:
|
||||
input.chainSubjects && input.chainSubjects.length > 0
|
||||
? JSON.stringify(input.chainSubjects)
|
||||
: null,
|
||||
probe_status: input.probeStatus,
|
||||
probe_error: input.probeError,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Return distinct manifest URLs referenced by indexable HTML 2xx
|
||||
* pages that don't yet have manifest data filled in. The post-crawl
|
||||
* `runManifestProbes` pass walks this list once.
|
||||
*/
|
||||
unprobedManifestUrls(limit = 1000): string[] {
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT DISTINCT manifest_url AS u
|
||||
FROM urls
|
||||
WHERE manifest_url IS NOT NULL
|
||||
AND manifest_url != ''
|
||||
AND is_external = 0
|
||||
AND content_kind = 'html'
|
||||
AND status_code >= 200 AND status_code < 300
|
||||
AND (manifest_json IS NULL OR manifest_json = '')
|
||||
LIMIT ?`,
|
||||
)
|
||||
.all(limit) as { u: string }[];
|
||||
return rows.map((r) => r.u).filter((u) => typeof u === 'string' && u.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stamp parsed manifest fields onto every URL that referenced this
|
||||
* `manifest_url`. A single manifest is typically shared site-wide,
|
||||
* so this is a one-fetch fan-out write.
|
||||
*/
|
||||
setManifestForReferrers(input: {
|
||||
manifestUrl: string;
|
||||
rawJson: string | null;
|
||||
themeColor: string | null;
|
||||
shortName: string | null;
|
||||
display: string | null;
|
||||
scope: string | null;
|
||||
iconCount: number;
|
||||
}): void {
|
||||
this.db
|
||||
.prepare(
|
||||
`UPDATE urls
|
||||
SET manifest_json = :json,
|
||||
manifest_theme_color = :theme,
|
||||
manifest_short_name = :short_name,
|
||||
manifest_display = :display,
|
||||
manifest_scope = :scope,
|
||||
manifest_icon_count = :icons
|
||||
WHERE manifest_url = :url`,
|
||||
)
|
||||
.run({
|
||||
url: input.manifestUrl,
|
||||
json: input.rawJson,
|
||||
theme: input.themeColor,
|
||||
short_name: input.shortName,
|
||||
display: input.display,
|
||||
scope: input.scope,
|
||||
icons: input.iconCount,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up the cached TLS cert info for a single URL — joins the URL's
|
||||
* host (parsed from the stored URL) against `host_certs`. Returns null
|
||||
@@ -3804,6 +3879,8 @@ export class ProjectDb {
|
||||
subject: string | null;
|
||||
signatureAlgorithm: string | null;
|
||||
protocol: string | null;
|
||||
chainLength: number | null;
|
||||
chainSubjects: string[] | null;
|
||||
probeStatus: number;
|
||||
probeError: string | null;
|
||||
probedAt: string | null;
|
||||
@@ -3812,6 +3889,7 @@ export class ProjectDb {
|
||||
.prepare(
|
||||
`SELECT hc.host, hc.valid_from, hc.valid_to, hc.days_until_expiry,
|
||||
hc.issuer, hc.subject, hc.signature_algorithm, hc.protocol,
|
||||
hc.chain_length, hc.chain_subjects,
|
||||
hc.probe_status, hc.probe_error, hc.probed_at
|
||||
FROM urls u
|
||||
JOIN host_certs hc
|
||||
@@ -3829,12 +3907,25 @@ export class ProjectDb {
|
||||
subject: string | null;
|
||||
signature_algorithm: string | null;
|
||||
protocol: string | null;
|
||||
chain_length: number | null;
|
||||
chain_subjects: string | null;
|
||||
probe_status: number;
|
||||
probe_error: string | null;
|
||||
probed_at: string | null;
|
||||
}
|
||||
| undefined;
|
||||
if (!row) return null;
|
||||
let chainSubjects: string[] | null = null;
|
||||
if (row.chain_subjects) {
|
||||
try {
|
||||
const parsed = JSON.parse(row.chain_subjects) as unknown;
|
||||
if (Array.isArray(parsed) && parsed.every((s) => typeof s === 'string')) {
|
||||
chainSubjects = parsed as string[];
|
||||
}
|
||||
} catch {
|
||||
/* leave null on parse failure */
|
||||
}
|
||||
}
|
||||
return {
|
||||
host: row.host,
|
||||
validFrom: row.valid_from,
|
||||
@@ -3844,6 +3935,8 @@ export class ProjectDb {
|
||||
subject: row.subject,
|
||||
signatureAlgorithm: row.signature_algorithm,
|
||||
protocol: row.protocol,
|
||||
chainLength: row.chain_length ?? null,
|
||||
chainSubjects,
|
||||
probeStatus: row.probe_status,
|
||||
probeError: row.probe_error,
|
||||
probedAt: row.probed_at,
|
||||
@@ -4412,6 +4505,12 @@ export class ProjectDb {
|
||||
ogSiteName: r.og_site_name ?? null,
|
||||
ogLocale: r.og_locale ?? null,
|
||||
androidIcon: r.android_icon ?? null,
|
||||
manifestJson: r.manifest_json ?? null,
|
||||
manifestThemeColor: r.manifest_theme_color ?? null,
|
||||
manifestShortName: r.manifest_short_name ?? null,
|
||||
manifestDisplay: r.manifest_display ?? null,
|
||||
manifestScope: r.manifest_scope ?? null,
|
||||
manifestIconCount: r.manifest_icon_count ?? 0,
|
||||
crawledAt: r.crawled_at,
|
||||
});
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@freecrawl/shared-types",
|
||||
"version": "0.3.2",
|
||||
"version": "0.3.3",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -268,6 +268,18 @@ export interface CrawlUrlRow {
|
||||
ogLocale: string | null;
|
||||
/** First `<link rel="icon" sizes>` ≥ 192px (or `sizes="any"`). PWA / Android home-screen icon. */
|
||||
androidIcon: string | null;
|
||||
/** Raw parsed Web App Manifest JSON (4 KB cap). Filled by post-crawl `runManifestProbes`. */
|
||||
manifestJson: string | null;
|
||||
/** Manifest `theme_color` field (hex / CSS color). */
|
||||
manifestThemeColor: string | null;
|
||||
/** Manifest `short_name` (home-screen label). */
|
||||
manifestShortName: string | null;
|
||||
/** Manifest `display` mode (`fullscreen` / `standalone` / `minimal-ui` / `browser`). */
|
||||
manifestDisplay: string | null;
|
||||
/** Manifest `scope` (PWA install scope URL). */
|
||||
manifestScope: string | null;
|
||||
/** Number of icons in the parsed manifest's `icons` array. */
|
||||
manifestIconCount: number;
|
||||
/** Total user-facing form inputs (input/textarea/select, excluding hidden/submit/button/image/reset). */
|
||||
formInputCount: number;
|
||||
/** Form inputs without label / aria-label / title (WCAG 1.3.1, 4.1.2 violation). */
|
||||
@@ -708,6 +720,24 @@ export interface CrawlConfig {
|
||||
* connect per host — typically a handful of probes for a site crawl.
|
||||
*/
|
||||
probeTlsCerts: boolean;
|
||||
/**
|
||||
* After the HTML crawl finishes, GET each unique declared
|
||||
* `<link rel="manifest">` once and stamp the parsed `theme_color`,
|
||||
* `short_name`, `display`, `scope`, and icon count onto every page
|
||||
* that referenced it. Raw JSON body is preserved (4 KB cap) so the
|
||||
* Detail panel can show every custom field. Default `true` — one
|
||||
* fetch per unique manifest URL, typically site-wide singleton.
|
||||
*/
|
||||
probeManifestJson: boolean;
|
||||
/**
|
||||
* When the post-norm "Duplicate URL" issue filter runs, normalise
|
||||
* URLs (lowercase, strip query, trim trailing slash) before
|
||||
* comparing. Default `true` — the canonical SEO behaviour. Set to
|
||||
* `false` for a byte-exact match (rarely useful; surfaces nothing
|
||||
* because URLs are already deduped at insert time, but kept as a
|
||||
* config knob for symmetry with the URL-rewrite preview).
|
||||
*/
|
||||
dedupePreNormalize: boolean;
|
||||
/**
|
||||
* Hostnames (lowercase, no scheme/port) that should be treated as
|
||||
* "same host" for scope purposes — used to keep CDN-served subdomains
|
||||
@@ -1411,6 +1441,8 @@ export const DEFAULT_CRAWL_CONFIG: CrawlConfig = {
|
||||
probeImageSizes: true,
|
||||
largeImageBytes: 102_400,
|
||||
probeTlsCerts: true,
|
||||
probeManifestJson: true,
|
||||
dedupePreNormalize: true,
|
||||
cdnHosts: [],
|
||||
maxLinksPerPage: 100,
|
||||
maxResponseTimeMs: 0,
|
||||
|
||||
@@ -427,6 +427,10 @@ export interface UrlCertInfoResult {
|
||||
subject: string | null;
|
||||
signatureAlgorithm: string | null;
|
||||
protocol: string | null;
|
||||
/** Total certificates in the peer chain (1 = self-signed, 2 = leaf+root, ≥3 = with intermediates). */
|
||||
chainLength: number | null;
|
||||
/** Distinguished names of the certs in the chain, leaf-first, capped at 5 entries. */
|
||||
chainSubjects: string[] | null;
|
||||
/** 200 = handshake OK + cert read, 0 = error/timeout, -1 = no probe yet. */
|
||||
probeStatus: number;
|
||||
probeError: string | null;
|
||||
|
||||
Reference in new issue
Block a user