CI/CD Integration

Run FreeCrawl's headless CLI as a quality gate in your pipeline — crawl a deploy, fail the build on broken links, validate the sitemap, and catch SEO regressions between releases.

Two ready-to-adapt templates live next to this file:

File	Platform
`github-actions-example.yml`	GitHub Actions
`gitlab-ci-example.yml`	GitLab CI/CD

Both crawl TARGET_URL, fail when 4xx/5xx URLs exceed BROKEN_LINK_THRESHOLD, validate the sitemap (advisory), and upload the crawl as an artifact.

How the CLI is built in CI

The runner clones the source and builds only the CLI and the packages it imports:

npm ci --ignore-scripts        # skip the Playwright download — undici crawl needs no Chromium
npx tsc -b apps/cli            # builds shared-types → db → core → cli via project references
node apps/cli/dist/index.js <url> [options]

npm run build:cli compiles only apps/cli and does not build the workspace packages the CLI imports, so the crawl would fail to resolve @freecrawl/core at runtime. Use npx tsc -b apps/cli, which follows the project references and builds the dependency graph in order — while skipping the heavy Electron desktop build. Drop --ignore-scripts only if you enable JS rendering via a --config file (that path needs Chromium).

Exit-code contract

Every command returns a CI-friendly exit code, so most gates need no scripting:

Command	`0`	`1`	`2`
`<url>` (crawl)	clean — no 4xx/5xx	a 4xx/5xx was found, the seed was unreachable, or nothing was fetched	—
`compare <before> <after>`	no differences	at least one diff category is non-zero	a project failed to open
`validate-sitemap <url>`	valid	validation warnings	fetch failure
`audit-robots <url>`	URL allowed	URL disallowed	invalid URL

Gate variants

Zero-tolerance (no script). Let the crawl's own exit code fail the step — any 4xx/5xx fails the build:

- run: node apps/cli/dist/index.js "$TARGET_URL" --max 5000

Threshold (what the templates do). Tolerate up to N broken URLs. Run the crawl with --json (append || true so the CLI's exit code doesn't pre-empt the gate), then count 4xx/5xx from the summary's byStatus map:

node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --json > summary.json || true
node -e "
  const s = require('./summary.json');
  const by = (s.summary && s.summary.byStatus) || {};
  let n = 0;
  for (const [code, count] of Object.entries(by)) if (Number(code) >= 400) n += count;
  if (n > Number(process.env.BROKEN_LINK_THRESHOLD)) { console.error('too many broken:', n); process.exit(1); }
"

Reading byStatus (a { '200': 1234, '404': 5, … } map) is robust against the full dump's column layout and avoids parsing a multi-MB file just to count.

Regression gating with `compare`

Fail a deploy when SEO metrics regress against a known-good baseline. Keep a baseline .seoproject (from a previous green run, restored from cache or an artifact store), crawl the new deploy into a fresh project, then diff:

node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --db new.seoproject
node apps/cli/dist/index.js compare baseline.seoproject new.seoproject
# exit 1 → a page changed status / title / meta / h1 / canonical / indexability

compare runs the same engine as the desktop Compare dialog and exits 1 on any non-zero diff category, so it drops straight into a pipeline gate. Promote new.seoproject to the new baseline when the pipeline passes.

Reproducible config

Export your settings once from the desktop app (Settings → Export Settings) and commit the JSON, then feed it to every run so CI crawls exactly like your desktop audits:

node apps/cli/dist/index.js "$TARGET_URL" --config freecrawl.config.json --max 5000

Per-flag overrides still win over the file, so --max can differ per run while the shared config stays fixed.

4.6 KiB Raw Permalink Blame History