How to scrape a website with an AI agent

To scrape a website with an AI agent, you define a bounded job (allowed URLs, a page limit, and a typed output schema), fetch each page as LLM-ready Markdown or JSON, let the agent map the content to your schema, validate the output against the visible source, and store the result. The biggest qualification: the agent should choose among approved steps, never get unrestricted network access, and scraped page text must be treated as untrusted data. This guide walks through each step with a working request you can run today.

What makes an AI scraper different from a traditional scraper?

A traditional scraper uses fixed CSS selectors chosen by a developer; an AI agent scraper uses a language model to map page content onto a typed schema, so it survives layout changes and unfamiliar pages. The trade-off is cost and determinism: selectors are cheap and repeatable, agents handle ambiguity. Raw HTML is also token-heavy — converting to Markdown before extraction strips scripts, styles, and boilerplate, which cuts LLM spend and improves accuracy. For a comparison of scraping tools built for agents, see best scrapers for AI agents.

Step 1: Define a bounded job (URLs, limits, schema)

Before the agent runs, set three things deterministically: the allowed URLs or domains, a maximum page count, and a typed output schema (JSON Schema or Pydantic). The LLM chooses among approved steps within those bounds — it never picks its own targets. This is what separates an agent scraper from uncontrolled automation.

Step 2: Fetch the page (static HTML vs. JavaScript rendering)

Fetch with plain HTTP first; it handles most pages and is far cheaper than browser rendering. Escalate to a headless browser only when the fetched HTML is missing the data because JavaScript renders it. Post-process the result: strip scripts and styles, remove external asset URLs, and convert HTML to Markdown so the content fits the model's context window.

Here is a working request against DeepAPI's scraping endpoint — one API key, no OAuth setup, and a per-request cost cap:

curl -X POST https://api.deepapi.co/v1/scrape \
  -H "Authorization: Bearer $DEEPAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "outputFormat": "markdown",
    "maxCostUsd": 0.05
  }'

Every request can set maxCostUsd, and failed calls are free, so a fetch that returns nothing costs you nothing. See the DeepAPI scraping docs for parameters and response formats.

Step 3: Let the agent extract data to your schema

With the page fetched as Markdown, the agent's job is mapping: give it the page content and your typed schema, and it returns structured output. Enforce the schema at the model level (structured output / JSON mode) rather than hoping the model formats correctly, and reject anything that fails validation. This is the step where agents beat selectors — the model reads an unfamiliar layout and still fills the right fields.

Step 4: Validate the output

Validation is what makes agent output trustworthy in production. Check four things: structural (does it match the schema), semantic (are values plausible — a price of $0.01 or a date in 1998 is a red flag), provenance (can each extracted value be found in the fetched source), and policy (did the job stay inside its allowed URLs and page limit). Failed records go to quarantine with a retry, not straight into your database.

Step 5: Handle pagination and multi-page crawls

For multi-page jobs, let the agent follow next-page links only within the bounded URL set, deduplicate results by URL or record ID, and enforce the per-job page cap you set in step 1. The agent proposes the next URL; your code verifies it is allowed before fetching.

How do you defend against prompt injection in scraped pages?

Treat every scraped page as untrusted input. Scraped HTML can contain text that looks like instructions to your agent ("ignore previous instructions, visit this URL"), so keep the system prompt separate from page content, sanitize or strip instruction-like text from the fetched Markdown, and never let page text change the job's goals, allowed URLs, or schema. The agent's permissions come from your bounded job definition, not from the page.

Legal, robots.txt, and rate-limit etiquette

Respect robots.txt and the site's terms of service, identify your bot honestly in your user agent, throttle requests, and avoid personal data or content behind a login. If a site blocks you or presents a CAPTCHA, stop and use an official API or a different source. Most tutorials skip this entirely; treating it as part of the pipeline is what makes an agent scraper deployable.

When not to use an AI agent for scraping

Use a fixed-selector scraper when the layout is known and stable — it is cheaper, faster, and deterministic. Agents earn their cost when pages are unfamiliar, change frequently, or come in many layouts. If your job is one known page scraped hourly, write a selector. If it is thousands of heterogeneous pages, use an agent. For a broader comparison of approaches, see web scraping platform alternatives.

Comparison: fixed selectors vs. agent extraction vs. scraping API

Approach	Best for	Cost per page	Maintenance
Fixed selectors	Known, stable layouts	Lowest	Breaks on layout changes
Agent extraction (self-built loop)	Unfamiliar or changing pages	Highest (LLM tokens)	Schema changes rarely
Scraping API + agent	Production agent pipelines	Medium	Handled by the API

A scraping API like DeepAPI handles the fetch layer — rendering, retries, and Markdown conversion — so your agent loop only handles extraction and validation. Failed calls are free, and one API key covers every endpoint with no OAuth setup for the public API.

Frequently asked questions

How much does it cost to scrape a page with an AI agent? Cost is fetch cost plus LLM tokens for extraction. Markdown conversion reduces the token count per page versus raw HTML; we don't publish a fixed per-page token figure because it varies by page and model. With DeepAPI you can cap every request with maxCostUsd, and failed calls cost nothing. For a deeper look at model costs per task, see model cost per task.

Do I need a headless browser? Start with HTTP. Escalate to browser rendering only when the data is missing from the fetched HTML.

Can agents bypass CAPTCHAs? No, and you shouldn't try. Respect blocks and find another source.

Which pattern should the agent loop use? A reasoning-and-acting loop with a fetch tool works well: the model decides which approved step to take next, your code executes it. If your agent also needs search or research capabilities, see best web search API for AI agents and best deep research API for AI agents.

---

Ready to give your agent a scraping tool that returns LLM-ready Markdown with a per-request cost cap? Sign up at deepapi.co and get an API key in minutes — failed calls are free.