Cloudflare Browser Rendering — Crawl Reference

What This Is

We used Cloudflare's Browser Rendering /crawl API to scrape the existing Squarespace site at rootstructureomaha.com before migrating to WordPress. This captured all text content, image URLs, and page structure.

API Details

Endpoint: POST /client/v4/accounts/{account_id}/browser-rendering/crawl
Account ID: 5078281af676ac2003a394b4ac758262
Crawl job ID: 0954cf93-199f-4686-82cf-ed4dbcdd300e
Date: 2026-03-12
Output format: Markdown
Pages crawled: 13

Free Plan Limits

Resource	Limit
Crawl jobs/day	5
Pages per crawl	100
Browser hours	10 min/day
REST API requests	6/min

Other Useful Endpoints

Endpoint	Purpose	Notes
`/json`	AI-powered structured data extraction	Pass a prompt + JSON schema, get clean data back. Needs Browser Rendering Edit token permission.
`/scrape`	CSS selector-based scraping	Target specific elements like `meta[name="description"]`
`/content`	Full raw HTML	Returns complete `<head>` with all meta tags
`/crawl`	Multi-page crawl	What we used. Follows links, returns markdown/HTML/JSON

Token Permissions

The stored token at ~/.cloudflare/credentials has zone management permissions only. For Browser Rendering endpoints (/json, /scrape, /content, /crawl), the token needs Browser Rendering - Edit permission added in the Cloudflare dashboard.

Data Locations

Folder	Contents
`~/.claude/project-notes/root-structure/rootstructure-crawl/`	Raw crawl markdown files (13 pages)
`~/.claude/project-notes/root-structure/rootstructure-images/`	All images organized by category (48 files, 51MB)
`~/.claude/project-notes/root-structure/rootstructure-seo/`	SEO metadata per page (JSON + CSV)

Replicating for Other Clients

This workflow can be used for any Squarespace-to-WordPress migration:

Crawl — Hit /crawl with the client's domain, get markdown content
Extract images — Parse image URLs from markdown, download with curl
Extract SEO — Fetch raw HTML, parse <head> meta tags (or use /json with a prompt if token has permissions)
Import to WP — Browser-based PHP script using wp_insert_post() + media_handle_sideload()
Set Yoast — Write to _yoast_wpseo_* post meta and wpseo_* term meta
Create redirects — Old URL paths → new WordPress paths via Redirection plugin