Try a request
# 1. start the job — returns an id, not the pages
curl https://api.nativeport.ai/firecrawl/v2/crawl \
-H "Authorization: Bearer $NATIVEPORT_API_KEY" \
-H "Content-Type: application/json" \
--data '{"url": "https://docs.example.com", "limit": 100,
"scrapeOptions": {"formats": ["markdown"]}}'
# 2. poll it until status is "completed"
curl https://api.nativeport.ai/firecrawl/v2/crawl/8f2a1c04-6b3e-4f77-9a15 \
-H "Authorization: Bearer $NATIVEPORT_API_KEY"{
"status": "completed",
"total": 87,
"completed": 87,
"data": [
{
"markdown": "# Getting started\n\nInstall the client with...",
"metadata": {
"sourceURL": "https://docs.example.com/start",
"title": "Getting started",
"statusCode": 200
}
}
]
}Start a crawl and collect the results
Send a starting URL, a page limit, and the formats you want in scrapeOptions.
The example requests Markdown and starts a job that you can check using the
returned ID.
Each status request reports progress. When the job completes, its results include page content and metadata such as the source URL. The Python example writes the returned Markdown into files.
For a first run, choose a small page limit and inspect the output. A website can contain pages your app does not need, so narrow the scope before collecting more. The number of collected pages and the options you enable affect the cost.
If you only need to discover URLs, use /firecrawl/v2/map first. When you already
have a short list of pages, individual scrape requests may be easier to manage.
The structured extraction guide
shows how to get specific fields from each page.
Crawl a docs site into one file per page
Shows how to start a crawl, check its status, and save returned pages. Replace the example domain and choose a page limit before running it.
import json, os, pathlib, time, urllib.request
BASE = "https://api.nativeport.ai/firecrawl/v2"
AUTH = {"Authorization": f"Bearer {os.environ['NATIVEPORT_API_KEY']}",
"Content-Type": "application/json"}
def call(path, payload=None):
req = urllib.request.Request(
BASE + path,
data=json.dumps(payload).encode() if payload else None,
headers=AUTH,
)
with urllib.request.urlopen(req) as r:
return json.load(r)
job = call("/crawl", {"url": "https://docs.example.com", "limit": 100,
"scrapeOptions": {"formats": ["markdown"]}})
while True:
res = call(f"/crawl/{job['id']}")
if res["status"] == "completed":
break
print(f" {res['completed']}/{res['total']} pages…")
time.sleep(5)
out = pathlib.Path("docs"); out.mkdir(exist_ok=True)
for page in res["data"]:
slug = page["metadata"]["sourceURL"].rstrip("/").rsplit("/", 1)[-1] or "index"
(out / f"{slug}.md").write_text(page["markdown"])
print(f"wrote {len(res['data'])} pages to {out}/")$ python crawl_docs.py 18/87 pages… 61/87 pages… wrote 87 pages to docs/
Consider another approach
Firecrawl starts a job and returns an ID. Check its status to follow progress and retrieve the collected pages.
Compare the synchronous crawl options from Tavily and Spider if you prefer a response in the same request. Check their page limits and crawl controls for your target site.
Use it with NativePort
Run Firecrawl crawls with your NativePort key and shared balance. The same account covers individual page requests and the models you use to work with the collected content.
Before you start
Why do I need to check the job after starting it?
Collecting a website takes time. The first request returns a job ID, and later status requests let you check progress and retrieve results. The total number of requests depends on how long the job takes.
How do I limit what gets collected?
Set limit to cap the number of pages. Firecrawl also supports path and depth controls. Use these to focus the crawl on the part of the site you need, such as its documentation.
How do I estimate the cost?
The number of pages and the scrape options affect the credits used. Start with a small page limit and check the Firecrawl pricing details before increasing it.
Can I collect just the URLs?
Yes. Firecrawl’s map endpoint returns discovered URLs without collecting the body of every page. Use it when you need a URL list or want to choose pages before extracting their content.