Guides · Updated 2026-10-19
Best web scraping API for AI agents: pick by the job, 6 options compared
Disclosure: DartzDabz is a ScraperAPI affiliate. If you sign up through our ScraperAPI links, we may earn a commission at no extra cost to you. That's why ScraperAPI is our featured pick, alongside the reasons below. It doesn't change our automated trust scores, and no other tool on this page paid to be included.
There is no single best web scraping API for AI agents; pick by the job. To fetch pages that block plain requests, use a proxy-and-render API such as ScraperAPI, a tool we earn an affiliate commission on. For search plus scrape in one API, try Firecrawl or Exa. For ready-made site scrapers, Apify. For multi-step browsing, Browser Use or Playwright MCP.
What is the best web scraping API for AI agents?
It depends on what your agent has to do. If it must read specific pages that block bots, a proxy-and-render API such as ScraperAPI does the fetching for you in one call. If it has to find pages first, Firecrawl and Exa combine search with page contents. If a ready-made scraper for one site already exists, Apify's marketplace saves you writing it. If the agent has to click, log in or fill forms, it needs a real browser: Browser Use in the cloud or Playwright MCP on your machine. The table compares all six on type, best use, free tier, entry price and our trust score.
| Tool | Best for | Type | Free tier | Entry price | DartzDabz score | Listing |
|---|---|---|---|---|---|---|
| ScraperAPI Affiliate link, #ad | Fetching specific pages that block plain requests | Scraping API with proxies, rendering and anti-bot handling; remote MCP server | Pricing page: 7-day trial with 5,000 API credits, no credit card. The same page's FAQ also describes a free plan with 1,000 API credits and 5 concurrent connections. Both statements are ScraperAPI's own; we quote both. | $49/month (Hobby, 100,000 credits); $44.10/month billed annually | Curated listing, not scored | ScraperAPI · Try ScraperAPI |
| Firecrawl | Search plus scrape in one API, and crawling whole sites into Markdown | Search, scrape, crawl and map API with LLM-ready output; official MCP server | 1,000 credits/month, no card | Hobby $19/month billed monthly or $16/month billed annually (5,000 credits) | Official · needs auth (not checked yet) | Firecrawl |
| Apify | Site-specific scrapers you don't want to build (social, maps, ecommerce) | Marketplace of ready-made scrapers (Actors); MCP server; agent payments via x402 | $5 platform usage/month, no card | Starter $19/month ($17/month billed annually) | 100/100 (A) | Apify |
| Exa | Finding pages first, then reading their contents | Search engine API plus page contents for known URLs; hosted MCP server; also listed as x402 endpoints | $10 credits, reset monthly, no payment method | Pay as you go: Search from $4/1k requests; Contents $1/1k pages | 81/100 (B) | Exa |
| Browser Use | Multi-step tasks where the agent must click, log in or fill forms | Cloud browsers and browser agents; also an x402 endpoint | $1 once, no card | Pay as you go; minimum top-up $5 | 100/100 (A) | Browser Use |
| Playwright MCP (Microsoft) | Free, local browsing for coding agents; testing and exploratory automation | Open-source MCP server that drives a local browser | Free (open source) | Free; you pay for your own compute | Official · local server (not checked) | Playwright MCP (Microsoft) |
Browse every search and web data tool in our agent tool directory or the search and web data tools. one sentence at the end of the 'best' H2: 'Other names you'll see, such as Bright Data, Browserbase, Zyte and Tavily, aren't in the DartzDabz directory yet, so we haven't compared them here.' For pay-per-call search and page readers ranked by today's score, see web search and scraping tools for agents.
Sources (checked 2026-10-19): www.scraperapi.com/pricing/ unverified · www.firecrawl.dev/pricing unverified · apify.com/pricing unverified · exa.ai/pricing unverified · browser-use.com/pricing unverified · github.com/microsoft/playwright-mcp unverified
What does an AI agent need from a scraping API?
Five things matter more for agents than for a classic scraper. It has to get the page despite blocks, which means proxies, JavaScript rendering and anti-bot handling. It should return text a model can use, ideally Markdown, not raw HTML that wastes tokens. It needs a tool interface the agent can call, such as an MCP server or a plain HTTP API. Cost has to be predictable per page, because agents retry and loop. And it should let you stay within the site's rules: robots.txt, rate limits and terms. DartzDabz's own crawler honors robots.txt and caps requests per host, and opted-out tools show as opted out. See which tools real agents use in the setups other builders run.
Sources (checked 2026-10-19): /about/ · docs.scraperapi.com/integrations/llm-integrations/mcp-server/claude-code unverified · www.firecrawl.dev/ unverified
ScraperAPI: our featured pick for pages that block plain requests
Why ScraperAPI is featured: we earn an affiliate commission on it, and it is built for one specific job: fetching pages that block plain requests. Per its own docs, one request handles proxy rotation, JavaScript rendering and CAPTCHA and anti-bot bypass, it returns Markdown for LLMs, it has structured endpoints for sites such as Amazon and Google, and it publishes a remote MCP server. We have not benchmarked it against the others, so this is not a claim that it is the best for every job.
ScraperAPI turns a hard fetch into one GET request: your API key plus the target URL, and ScraperAPI says it handles proxy rotation, JavaScript rendering and CAPTCHA and anti-bot bypass behind it. Its output_format parameter returns Markdown, text or JSON, and it publishes a remote MCP server for Claude Code and other clients. The trade-off is credits: a standard page costs 1, Google 25 and LinkedIn 30, sites behind Cloudflare and similar protection add 10, and unused credits don't roll over. ScraperAPI's pricing page says: 'Start collecting data with our 7-day trial and 5,000 API credits. No credit card required.' The FAQ on the same page says ScraperAPI 'offers a free plan where you can sign up here and get 1,000 free API credits (with a maximum of 5 concurrent connections)'. Both checked Oct 19, 2026. Paid plans start at $49 a month.
Try ScraperAPI (affiliate link) Affiliate link: we may earn a commission. It never affects scores. Disclosure
ScraperAPI: strengths and weak spots
- One GET to api.scraperapi.com with an API key and a target URL; ScraperAPI says proxy rotation, JavaScript rendering and CAPTCHA/anti-bot bypass happen behind that call.
- output_format can return Markdown, text or JSON (JSON for a list of supported domains), which suits LLM context.
- Remote MCP server at https://mcp.scraperapi.com/mcp with documented setup for Claude Code; the scrape tool takes render, country_code, premium, ultra_premium and device_type options.
- Structured-data endpoints return JSON for high-demand targets such as Amazon, Google Search, Walmart, eBay and Redfin, per ScraperAPI's product overview.
- All plans list JS rendering, premium proxies, JSON auto-parsing, CAPTCHA and anti-bot detection, automatic retries and unlimited bandwidth; the pricing page also states a 99.9% uptime guarantee and a 7-day no-questions refund.
- Watch: Credits vary by target: a standard page is 1 credit, Amazon 5, Google and Bing 25, LinkedIn 30, and sites behind Cloudflare, DataDome or PerimeterX add 10, so hard targets cost far more than the plan's headline credit count suggests.
- Watch: Credits reset at renewal and do not roll over; by default scraping pauses when credits run out.
- Watch: Paid plans start at $49 a month (Hobby, 100,000 credits; $44.10 a month billed annually), and Hobby and Startup geotargeting is US and EU only.
- Watch: It fetches the pages you point it at; it is not a general web search index, though it has structured endpoints for Google Search results.
Connect in Claude Code: claude mcp add scraperapi -- npx mcp-remote https://mcp.scraperapi.com/mcp --header "Authorization: Bearer {YOUR_API_KEY}"
Sources (checked 2026-10-19): www.scraperapi.com/pricing/ unverified · docs.scraperapi.com/integrations/llm-integrations/mcp-server/claude-code unverified · docs.scraperapi.com/getting-started/quick-start/send-your-first-request unverified · www.scraperapi.com/docs/ai/product-overview unverified
Firecrawl: search, scrape and crawl in one API
Firecrawl covers search, scrape, crawl and map in one API; /scrape returns Markdown by default and renders JavaScript automatically. The free plan gives 1,000 credits a month with no card, and a basic scrape is 1 credit per page. Its MCP server can even start keyless, within daily limits. Watch two billing rules: a page that returns an error such as 403 or 404 still costs 1 credit, and JSON or other extraction formats add 4 credits per page. Hobby is $19 a month, or $16 billed annually, for 5,000 credits. See Firecrawl's official MCP server.
Firecrawl: strengths and weak spots
- One API for search, scrape, crawl, map and interact; /scrape returns Markdown by default, renders JavaScript automatically, and the hosted version handles proxies and rendering, per Firecrawl's site.
- /crawl follows links across a whole site with depth and path limits, and Firecrawl says it respects robots.txt rules for its FirecrawlAgent.
- Free plan: 1,000 credits every month, refreshed monthly, no card; scrapes that return no result are not charged.
- MCP server with three ways in: a keyless 'Try instantly' mode for Search, Scrape and Parse within daily limits, OAuth sign-in, or an API key.
- Pay-as-you-go top-ups in $5 increments on paid plans, with a monthly cap you set.
- Watch: A page that returns an error status such as 403 or 404 is still returned and costs 1 credit.
- Watch: The JSON, Question and Highlight formats add 4 credits per page on Scrape and Crawl.
- Watch: Credits roll over only on Scale and Enterprise; Hobby, Standard and Growth credits do not.
- Watch: Anti-bot handling is described in general terms (proxy management, an Enhanced mode) rather than per-site credit rules, so test your hardest target sites on the free plan before you rely on them.
Connect in Claude Code: claude mcp add --transport http firecrawl-official https://mcp.firecrawl.dev/v2/mcp (API-key connections use a bearer token; OAuth URL: https://mcp.firecrawl.dev/v2/mcp-oauth)
Sources (checked 2026-10-19): www.firecrawl.dev/pricing unverified · docs.firecrawl.dev/mcp-server unverified · www.firecrawl.dev/ unverified
Apify: ready-made scrapers for specific sites
Apify is a marketplace of ready-to-run scrapers called Actors. Through its MCP server, an agent can search the store, read an Actor's input schema and run it, and the default tools include a web-fetch Actor that Apify says renders JavaScript and bypasses anti-bot protection. Agents can even pay without an account through x402 at agi.apify.com. The catch is predictability: most Actors set their own pay-per-event prices, others bill compute units and proxies, so run one test job before you budget. The free plan includes $5 of usage a month. See Apify's x402 listing.
Apify: strengths and weak spots
- Apify Store is a marketplace of ready-to-run Actors; the MCP server at mcp.apify.com lets an agent search for an Actor, read its input schema and call it on demand.
- The default MCP tools include apify/rag-web-browser and apify/web-fetch, which Apify says renders JavaScript and bypasses anti-bot protection.
- Agents can pay without an account: through x402 or MPP at agi.apify.com, an agent buys a temporary token with a fixed spend cap.
- Free plan with $5 of platform usage a month, no credit card.
- Watch: Cost depends on the Actor: most use pay-per-event pricing set by the Actor's developer, others bill compute units ($0.20 per CU on Free and Starter) plus proxies and storage, so a run's cost is hard to predict before a test run.
- Watch: On the free plan, access is blocked until the next cycle once the $5 is used; unused prepaid usage expires each cycle.
- Watch: Running Actors and reading results always needs authentication; call-actor returns a run status and storage IDs, so the agent needs a second call to read results.
Connect in Claude Code: MCP URL: https://mcp.apify.com (OAuth) or with an Authorization: Bearer <APIFY_TOKEN> header
Sources (checked 2026-10-19): apify.com/pricing unverified · docs.apify.com/platform/integrations/mcp unverified
Exa: search first, then read the pages
Exa is a search API with a contents endpoint that returns page text, highlights or summaries for known URLs. It is pay-as-you-go: search from $4 per 1,000 requests and contents at $1 per 1,000 pages per content type, with $10 of free credit that resets monthly. It has a hosted MCP server, and the DartzDabz directory also lists Exa's search and contents endpoints as x402 pay-per-call APIs. Its pricing page doesn't describe proxies or anti-bot handling, so use it to find and read pages, and a scraping API for sites that fight bots. See Exa in the directory.
Exa: strengths and weak spots
- Pay-as-you-go with no subscription and no minimum spend; /contents is $1 per 1,000 pages per content type, and /search starts at $4 per 1,000 requests.
- The free tier gives $10 in credits at signup and resets the free balance to $10 on the first of each month, no payment method required.
- Hosted MCP server at https://mcp.exa.ai/mcp and an agent skill (npx skills add exa-labs/agent-skills).
- The directory also lists Exa's search and contents endpoints as x402 pay-per-call APIs.
- Watch: Built around search and page contents; its pricing page doesn't describe proxies, JavaScript rendering or CAPTCHA handling, so for pages that actively block bots, a scraping API is the safer bet.
- Watch: /contents bills each content type separately, so text plus highlights for one page counts as two.
- Watch: Deep Search and Agent runs cost more ($12–15 per 1,000 Deep Search requests; Agent from $0.012 to $1.00 per fixed-effort run).
Connect in Claude Code: MCP URL: https://mcp.exa.ai/mcp
Sources (checked 2026-10-19): exa.ai/pricing unverified
Browser Use and Playwright MCP: when the agent has to click
Some jobs need a real browser: logging in, paging through results, filling forms. Browser Use runs cloud browsers at $0.02 an hour plus $5 per GB of residential proxy, and its browser agents bill model tokens plus 20%; its own benchmark lists tasks taking minutes. Playwright MCP is Microsoft's free, open-source server that drives a local browser through the accessibility tree, with one-line setup for Claude Code, Codex and Cursor. It runs on your machine, so proxies and blocking are up to you, and its README says it is not a security boundary. See Browser Use's x402 endpoint.
Browser Use: strengths and weak spots
- Cloud browsers at $0.02 an hour, plus $5 per GB of residential proxy traffic (on by default) or $0.20 per GB with direct or your own proxy.
- Browser agents bill model tokens plus a 20% fee, plus browser time and traffic; no subscription, and credits never expire.
- $1 of free credit once per person, with no card, for eligible Google, GitHub or Microsoft signups.
- Its x402 sessions endpoint is listed in the DartzDabz directory for agents that pay per call.
- Watch: Per-task cost depends on the model and the task: Browser Use's own benchmark table lists recorded agent costs from about $0.06 to $8.87 per task, and says those figures exclude browser hosting and aren't Cloud prices.
- Watch: The same benchmark table lists tasks taking minutes, not seconds, so it's overkill for fetching a single page.
- Watch: Minimum top-up is $5.
Connect in Claude Code: x402 endpoint: https://x402.api.browser-use.com/api/v3/sessions (listed price $1 per call in our directory)
Sources (checked 2026-10-19): browser-use.com/pricing unverified · github.com/microsoft/playwright-mcp unverified
How do you connect a scraping API to Claude Code or Cursor?
Add it as an MCP server, then ask the agent to fetch a URL. In Claude Code that is one command per server, shown below as each vendor documents it. Cursor and Codex take the same server URLs in their MCP settings. Add one server, run /mcp to check it connected, try one page, then add the next, so tool descriptions don't crowd the context. Keep API keys in your client's config, never in a prompt. See best MCP servers for Claude Code.
ScraperAPI (remote MCP, from ScraperAPI docs):
claude mcp add scraperapi -- npx mcp-remote https://mcp.scraperapi.com/mcp --header "Authorization: Bearer {YOUR_API_KEY}"Firecrawl (remote MCP, API key or OAuth):
claude mcp add --transport http firecrawl-official https://mcp.firecrawl.dev/v2/mcp
Playwright MCP (local, from the README):
claude mcp add playwright npx @playwright/mcp@latest
Sources (checked 2026-10-19): docs.scraperapi.com/integrations/llm-integrations/mcp-server/claude-code unverified · docs.firecrawl.dev/mcp-server unverified · github.com/microsoft/playwright-mcp unverified
What does web scraping cost for an AI agent?
Every vendor here bills in its own unit, so compare per page of your real targets, not per plan. ScraperAPI charges credits that vary by site, from 1 for a standard page to 30 for LinkedIn, plus 10 for bot-protected sites. Firecrawl charges 1 credit per basic page and more for extraction formats. Exa charges $1 per 1,000 pages of contents. Apify depends on the Actor. Browser Use bills browser hours, proxy traffic and model tokens. Playwright MCP is free, but you pay for the machine. All figures are vendor list prices checked October 19, 2026. See how much it costs to run AI agents.
| Tool | Billing unit | Example unit price | Unused credits roll over? | Source (checked 2026-10-19) |
|---|---|---|---|---|
| ScraperAPI | API credits per request | 1 per standard page; 5 Amazon; 25 Google and Bing; 30 LinkedIn; +10 when bypassing bot protection | No: "your ScraperAPI credit balance resets when your subscription renews" (ScraperAPI pricing FAQ, checked Oct 19, 2026) | www.scraperapi.com/pricing/ unverified |
| Firecrawl | Credits per page | 1 per basic scrape (still charged on 403/404); +4 per page for JSON and other extraction formats | not published | www.firecrawl.dev/pricing unverified |
| Apify | Set per Actor (pay-per-event), or compute units plus proxies | Depends on the Actor | not published | apify.com/pricing unverified |
| Exa | Per 1,000 requests or pages | Search from $4 per 1,000 requests; contents $1 per 1,000 pages per content type | not published ($10 free credit resets monthly) | exa.ai/pricing unverified |
| Browser Use | Browser hours, proxy traffic and model tokens | $0.02 per browser hour; $5 per GB residential proxy; model tokens plus 20% | not published | browser-use.com/pricing unverified |
| Playwright MCP | None (open source) | Free; you pay for the machine it runs on | n/a | github.com/microsoft/playwright-mcp unverified |
Sources (checked 2026-10-19): www.scraperapi.com/pricing/ unverified · www.firecrawl.dev/pricing unverified · exa.ai/pricing unverified · apify.com/pricing unverified · browser-use.com/pricing unverified · github.com/microsoft/playwright-mcp unverified
Is web scraping with AI agents allowed?
It depends on the site, the data and where you are, and this is not legal advice. Many platforms set their own rules in their terms; Fiverr, for example, prohibits scraping its platform. The practical rules we follow for the DartzDabz crawler are a good floor: check robots.txt, rate-limit each host, don't collect personal data you don't need, and stop when a site asks you to. A scraping API changes how a page is fetched, not whether you're allowed to fetch it.
Sources (checked 2026-10-19): /about/ · /trends/make-money-with-grok-bot/
Web scraping APIs for AI agents: FAQ
Is there a free web scraping API for AI agents?
Several have free tiers, each checked on October 19, 2026. Firecrawl gives 1,000 credits a month, Exa $10 of credit that resets monthly, and Apify $5 of usage a month, all with no card. ScraperAPI's pricing page lists a 7-day trial with 5,000 API credits, while the FAQ on the same page describes a free plan with 1,000 free API credits, and Playwright MCP is free and open source.
What is the difference between a scraping API and a search API?
A scraping API fetches a page you already know and returns its content, often handling proxies and JavaScript for you. A search API, such as Exa's, finds pages for a query first. Firecrawl and Exa offer both search and page contents.
Do I need an MCP server to scrape with an AI agent?
No. Every tool here has a plain HTTP API your code can call. An MCP server lets an agent like Claude Code or Cursor call the scraper as a tool on its own, and ScraperAPI, Firecrawl, Apify, Exa and Playwright all publish one.
Can an AI agent scrape sites protected by Cloudflare?
Some scraping APIs say they can. ScraperAPI says it bypasses protection such as Cloudflare, DataDome and PerimeterX and charges 10 extra credits per request when it does. Success isn't guaranteed, and the site's terms still apply.
Which scraping API returns Markdown for LLMs?
ScraperAPI's output_format parameter returns Markdown, text or JSON. Firecrawl returns Markdown by default, and Exa's contents endpoint returns page text, highlights or summaries. Markdown usually costs fewer tokens than raw HTML.
How much does ScraperAPI cost?
Paid plans start at $49 a month for 100,000 credits on Hobby, or $44.10 a month billed annually. Credits depend on the site: 1 for a standard page, 5 for Amazon, 25 for Google and 30 for LinkedIn. Unused credits don't roll over.
Is ScraperAPI the best scraping API for AI agents?
It is our featured pick for fetching pages that block plain requests, and we are its affiliate. We haven't benchmarked it against the others. For search, crawling whole sites or multi-step browsing, another tool on this page may fit better.
How does DartzDabz score these tools?
MCP servers and x402 APIs get an automated live check every day and a 0 to 100 trust and readiness score from a published method. Curated listings such as ScraperAPI aren't scored, and scores are point-in-time checks, not endorsements or security audits. See how DartzDabz scores tools.
Built an agent that scrapes with one of these? Submit your setup.
Affiliate disclosure: DartzDabz earns a commission if you buy ScraperAPI through links marked 'affiliate' on this page, at no extra cost to you. We picked the tools on this page from our own directory, and no vendor paid to be included or to change a score. Prices and plan details are each vendor's own, checked 2026-10-19; they change, so confirm on the vendor's page before you buy. See our full disclosure.
Sources
- ScraperAPI pricing and FAQ (checked 2026-10-19)
- ScraperAPI docs, MCP server for Claude Code (checked 2026-10-19)
- ScraperAPI docs, send your first request (checked 2026-10-19)
- ScraperAPI homepage (checked 2026-10-19)
- ScraperAPI product overview (AI grounding page) (checked 2026-10-19)
- Firecrawl pricing (checked 2026-10-19)
- Firecrawl docs, MCP server (checked 2026-10-19)
- Firecrawl homepage (checked 2026-10-19)
- Apify pricing (checked 2026-10-19)
- Apify docs, MCP server (checked 2026-10-19)
- Exa pricing (checked 2026-10-19)
- Browser Use pricing (checked 2026-10-19)
- Microsoft Playwright MCP README (checked 2026-10-19)
- DartzDabz, about and scoring (checked 2026-10-19)
- DartzDabz counts.json (checked 2026-10-19)
- DartzDabz tools-search.json (trust scores) (checked 2026-10-19)
- DartzDabz, make money with Grok Bot (cites Fiverr Community Standards) (checked 2026-10-19)
Get the free DartzDabz newsletter
Free email with new and notable agent tools, the biggest trust-score changes, a rising trend, and one setup you can copy. No spam; unsubscribe anytime.