Website crawling

Website bots

Create a website bot (this creates the bot and starts the first crawl):

Then watch and manage it:

| Call | Does | |---|---| | GET /api/scraper/website-kb/{id}/status/ | Crawl progress | | GET …/{id}/check-updates/ | Which pages changed since the last crawl | | POST …/{id}/re-crawl/ | Re-read the whole site | | POST …/{id}/recrawl-url/ | Re-read one URL ({"url": "…"}) | | POST …/{id}/bulk-recrawl-changes/ | Re-read only changed pages | | POST …/{id}/cancel-crawl/ | Stop | | GET …/{id}/urls/, …/history/, …/graph/ | Pages, past crawls, link graph | | POST /api/scraper/website-kb/convert/ | Turn an existing bot into a website bot |

One-off scraping

- POST /api/scraper/scrape/ — structured scrape of a page - POST /api/scraper/build-kb/ / build-kb-full-site/ — scrape straight into a bot - POST /api/scraper/thumbnail/ — page screenshot

Scrape jobs

Long crawls run as jobs: GET /api/scraper/scrape-jobs/, …/{id}/status/, …/{id}/pages/, …/{id}/cancel/, and …/{id}/ingest/ to load the results into a bot.