← All leaderboards
Site crawling
What this board tests: enumerate and fetch many pages across a site. Every provider below ran the identical corpus on 2026-06-22. The composite blends measured quality, latency p50, cost per successful call and error rate. Method and corpora: how we measure.
| Rank | Provider | Composite | Coverage | Cost | Latency p50 | Field note |
|---|---|---|---|---|---|---|
| 1 |  Firecrawl |
8.61/10 | 0.92 | $0.024 / useful | 16.3 s | Leads for clean, path-scoped enumeration. |
| 2 |  Spider |
8.54/10 | 0.77 | $6.2e-05 / useful | 2.9 s | Quick at recursive crawling. |
| 3 |  Tavily |
8.38/10 | 0.78 | $0.014 / useful | 174 ms | Performs well through its /map variant. |
| 4 |  Bright Data |
4.47/10 | 0.10 | $0.01 / useful | 6.5 s | No link discovery: url_collection hands back just the seed. |
| 5 |  Oxylabs |
3.18/10 | 0.27 | $0.13 / useful | 35.5 s | Path-scoped enumeration is beyond the AI-prompt crawler. |
Low scores are published alongside high ones — a ranking you can only trust if nothing gets hidden. The gateway's fee sits on top-ups, not per call, so the order above earns us nothing either way.