Scrape Operator

Overview

Deliver corpora from any site type into workspace files (csv/json/xlsx/md/report). When writing Python scrapers, use Scrapling 0.4.14 first (see skill scrapling). Use the scrape tool (Rust / httpx) for one-shot jobs with no custom code. Use browser for JS/scroll/forms. Finish with files + a one-line done summary.

Routing by page type

Site type	Approach
Custom scraper / spider the agent writes	Scrapling 0.4.14 (scrapling skill)
Static docs / blogs, no custom code	scrape crawl or pipeline (Rust when navin-core is built)
Listing + next/page=N	Scrapling spider, or scrape paginate then crawl item links
Sitemap known	Scrapling or sitemap → fetch urls in batches → export
Infinite scroll / lazy JS	browser navigate → scroll_infinite → extract / content → hand off to export
Forms / multi-tab / upload	browser (see playwright-browser)
empty_shell after fetch	escalate browser
captcha / CF / paywall / login	pause - never bypass

Mass crawl checklist

1. diagnose a seed URL.
2. Cap with max_pages (hard max 500) and optional allow / deny regex.
3. Resume long jobs with checkpoint="scrape/checkpoint.json".
4. Prefer pipeline to write exports directly; keep chat short.
5. Dedup is on by content hash; pagination links do not burn depth budget.

Cheat sheet

# Custom code: Scrapling 0.4.14 (Fetcher / StealthyFetcher / Spider / ShopifySpider)
# see skill scrapling and scrapling/references/api.md

scrape(action="paginate", url="https://shop.example/list?page=1", max_pages=30)
scrape(action="crawl", url="https://shop.example/list", max_depth=2, max_pages=200, checkpoint="scrape/ckpt.json", deny="/(cart|login|logout)")
scrape(action="sitemap", url="https://example.com/sitemap.xml", max_pages=200)
scrape(action="pipeline", url="https://example.com", format="xlsx", path="scrape/out.xlsx", max_pages=100)
browser(action="scroll_infinite", index=12)   # after navigate - then extract/content

Rules

• Developing scraper code: Scrapling 0.4.14 first. Do not start with BeautifulSoup or raw Playwright.
• One-shot corpora without new code: scrape tool (Rust hot path, else httpx).
• Escalate to browser for scroll/JS/forms only.
• Respect robots.txt (default on). Never invent rows.
• Never bypass human walls. Write files, not chat dumps.
• Deliver the dataset and the report. Scraper code stays under scrape/build/ and is handed over only when asked for.
• Qualified people/companies from a scrape can be written into the shared Studio CRM with the crm tool after a quality pass.