mirror of
https://github.com/kemalai/FreeCrawl-SEO-Tool.git
synced 2026-10-18 00:39:38 +02:00
101 lines
4.6 KiB
Markdown
101 lines
4.6 KiB
Markdown
# CI/CD Integration
|
|
|
|
Run FreeCrawl's headless CLI as a quality gate in your pipeline — crawl a
|
|
deploy, fail the build on broken links, validate the sitemap, and catch SEO
|
|
regressions between releases.
|
|
|
|
Two ready-to-adapt templates live next to this file:
|
|
|
|
| File | Platform |
|
|
| ------------------------------------------------------------ | -------------- |
|
|
| [`github-actions-example.yml`](./github-actions-example.yml) | GitHub Actions |
|
|
| [`gitlab-ci-example.yml`](./gitlab-ci-example.yml) | GitLab CI/CD |
|
|
|
|
Both crawl `TARGET_URL`, fail when 4xx/5xx URLs exceed `BROKEN_LINK_THRESHOLD`,
|
|
validate the sitemap (advisory), and upload the crawl as an artifact.
|
|
|
|
## How the CLI is built in CI
|
|
|
|
The runner clones the source and builds only the CLI and the packages it
|
|
imports:
|
|
|
|
```bash
|
|
npm ci --ignore-scripts # skip the Playwright download — undici crawl needs no Chromium
|
|
npx tsc -b apps/cli # builds shared-types → db → core → cli via project references
|
|
node apps/cli/dist/index.js <url> [options]
|
|
```
|
|
|
|
> `npm run build:cli` compiles **only** `apps/cli` and does **not** build the
|
|
> workspace packages the CLI imports, so the crawl would fail to resolve
|
|
> `@freecrawl/core` at runtime. Use `npx tsc -b apps/cli`, which follows the
|
|
> project references and builds the dependency graph in order — while skipping
|
|
> the heavy Electron desktop build. Drop `--ignore-scripts` only if you enable
|
|
> JS rendering via a `--config` file (that path needs Chromium).
|
|
|
|
## Exit-code contract
|
|
|
|
Every command returns a CI-friendly exit code, so most gates need no scripting:
|
|
|
|
| Command | `0` | `1` | `2` |
|
|
| -------------------------- | ------------------ | --------------------------------------------------------------------- | ------------------------ |
|
|
| `<url>` (crawl) | clean — no 4xx/5xx | a 4xx/5xx was found, the seed was unreachable, or nothing was fetched | — |
|
|
| `compare <before> <after>` | no differences | at least one diff category is non-zero | a project failed to open |
|
|
| `validate-sitemap <url>` | valid | validation warnings | fetch failure |
|
|
| `audit-robots <url>` | URL allowed | URL disallowed | invalid URL |
|
|
|
|
## Gate variants
|
|
|
|
**Zero-tolerance (no script).** Let the crawl's own exit code fail the step —
|
|
any 4xx/5xx fails the build:
|
|
|
|
```yaml
|
|
- run: node apps/cli/dist/index.js "$TARGET_URL" --max 5000
|
|
```
|
|
|
|
**Threshold (what the templates do).** Tolerate up to N broken URLs. Run the
|
|
crawl with `--json` (append `|| true` so the CLI's exit code doesn't pre-empt
|
|
the gate), then count 4xx/5xx from the summary's `byStatus` map:
|
|
|
|
```bash
|
|
node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --json > summary.json || true
|
|
node -e "
|
|
const s = require('./summary.json');
|
|
const by = (s.summary && s.summary.byStatus) || {};
|
|
let n = 0;
|
|
for (const [code, count] of Object.entries(by)) if (Number(code) >= 400) n += count;
|
|
if (n > Number(process.env.BROKEN_LINK_THRESHOLD)) { console.error('too many broken:', n); process.exit(1); }
|
|
"
|
|
```
|
|
|
|
Reading `byStatus` (a `{ '200': 1234, '404': 5, … }` map) is robust against the
|
|
full dump's column layout and avoids parsing a multi-MB file just to count.
|
|
|
|
## Regression gating with `compare`
|
|
|
|
Fail a deploy when SEO metrics regress against a known-good baseline. Keep a
|
|
baseline `.seoproject` (from a previous green run, restored from cache or an
|
|
artifact store), crawl the new deploy into a fresh project, then diff:
|
|
|
|
```bash
|
|
node apps/cli/dist/index.js "$TARGET_URL" --max 5000 --db new.seoproject
|
|
node apps/cli/dist/index.js compare baseline.seoproject new.seoproject
|
|
# exit 1 → a page changed status / title / meta / h1 / canonical / indexability
|
|
```
|
|
|
|
`compare` runs the same engine as the desktop Compare dialog and exits `1` on
|
|
any non-zero diff category, so it drops straight into a pipeline gate. Promote
|
|
`new.seoproject` to the new baseline when the pipeline passes.
|
|
|
|
## Reproducible config
|
|
|
|
Export your settings once from the desktop app (Settings → Export Settings) and
|
|
commit the JSON, then feed it to every run so CI crawls exactly like your
|
|
desktop audits:
|
|
|
|
```bash
|
|
node apps/cli/dist/index.js "$TARGET_URL" --config freecrawl.config.json --max 5000
|
|
```
|
|
|
|
Per-flag overrides still win over the file, so `--max` can differ per run while
|
|
the shared config stays fixed.
|