Scrape structured data from a website

Try a request

curl

curl https://api.nativeport.ai/firecrawl/v2/scrape \
  -H "Authorization: Bearer $NATIVEPORT_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "url": "https://example.com/product/1",
    "formats": [{
      "type": "json",
      "schema": {
        "type": "object",
        "properties": {
          "name":  {"type": "string"},
          "price": {"type": "number"},
          "stock": {"type": "string"}
        }
      }
    }]
  }'

example response

{
  "success": true,
  "data": {
    "json": {
      "name": "Aeron Chair",
      "price": 1395.00,
      "stock": "in stock"
    },
    "metadata": {
      "sourceURL": "https://example.com/product/1",
      "statusCode": 200
    }
  }
}

Describe the fields you want

The request sends a page URL with a JSON schema. In the example, the schema asks for a product name, price, and stock status. Firecrawl fetches the page and extracts those values into data.json.

Use the schema to describe the information your app needs. This approach can reduce dependence on specific HTML elements when a site changes its layout. It still needs checking: compare results with a few source pages and decide how your app will handle missing or unexpected values.

JSON extraction has its own credit cost in addition to fetching the page. If you only need readable content, requesting Markdown may be sufficient. The formats list also lets you request Markdown alongside JSON in the same response.

For several known URLs, repeat the request for each one as the Python example does. If you first need to discover the pages on a site, start with the website crawling guide.

Pull the same three fields from a list of URLs

Set NATIVEPORT_API_KEY and replace the example URLs with your product pages. The script uses the Python standard library.

python

import json, os, urllib.request

SCHEMA = {
    "type": "object",
    "properties": {
        "name":  {"type": "string"},
        "price": {"type": "number"},
        "stock": {"type": "string"},
    },
}

def scrape(url):
    req = urllib.request.Request(
        "https://api.nativeport.ai/firecrawl/v2/scrape",
        data=json.dumps({
            "url": url,
            "formats": [{"type": "json", "schema": SCHEMA}],
        }).encode(),
        headers={
            "Authorization": f"Bearer {os.environ['NATIVEPORT_API_KEY']}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req) as r:
        return json.load(r)["data"]["json"]

for url in ["https://example.com/product/1", "https://example.com/product/2"]:
    row = scrape(url)
    print(f"{row['name']:<24} {row['price']:>9.2f}  {row['stock']}")

example output

$ python scrape_products.py
Aeron Chair               1395.00  in stock
Embody Chair              1795.00  backordered

Consider another approach

For a single page

Use a scrape request with a JSON schema for the fields you need. Check the extracted data against your source page.

When you need to find pages first

Use a site map or crawl to discover URLs, then extract the fields from the pages you need. The website crawling guide covers discovery and page limits.

Use it with NativePort

Try Firecrawl with your NativePort key and pay for usage from your shared balance. You can compare other scraping providers through the same account.

Before you start

Do I need CSS selectors or XPath?

This example uses a JSON schema to describe the fields. It can reduce the selectors you maintain, but you should still check the extracted values. If you prefer extraction rules, compare the options in the web scraping guide.

How much does extraction cost?

Firecrawl uses credits, and options such as JSON extraction or proxy features can change the amount used. Check the Firecrawl provider page before estimating a batch.

What happens if the site blocks the request?

NativePort returns Firecrawl’s error and status code. Your application decides whether to retry, change the request options, or use another provider.