NativePort
Sign in Get a key
← How-to

How to Scrape a JavaScript-Rendered Page on NativePort with ScrapingBee

Render JS-heavy pages, wait for content that loads late, extract structured fields, and keep credit cost under control through NativePort's ScrapingBee proxy.

A plain HTTP fetch gets you whatever HTML a server sends before any JavaScript runs — for a page that builds its actual content client-side, that’s often close to nothing. ScrapingBee solves this with one endpoint: give it a URL, ask it to render, and it hands back the page’s post-render HTML (or, with the right flags, a screenshot or already-structured JSON) from a real headless browser it runs for you. NativePort proxies that single endpoint under one key, with every rendering control — JS on or off, proxy tier, wait conditions, extraction rules — passed straight through in the query string, untouched.

What you’ll need

  • A NativePort API key. Sign up to get a key and $5 in credits.
  • Python 3, standard library only — urllib.request, no pip install.
  • Your key exported as an environment variable, never hardcoded:
export NATIVEPORT_API_KEY="np_..."

A basic render

import json
import os
import urllib.parse
import urllib.request

GATEWAY = "https://api.nativeport.ai"
API_KEY = os.environ["NATIVEPORT_API_KEY"]


def scrape(url, **params):
    qs = urllib.parse.urlencode({"url": url, **params})
    req = urllib.request.Request(f"{GATEWAY}/scrapingbee?{qs}", method="GET")
    req.add_header("Authorization", f"Bearer {API_KEY}")
    with urllib.request.urlopen(req, timeout=145) as resp:
        return resp.status, dict(resp.headers), resp.read()


status, headers, body = scrape("https://example.com", render_js="true")
print(status, headers.get("Spb-cost"), headers.get("Spb-resolved-url"))
print(body.decode(errors="replace")[:300])

url is the only required parameter; every other ScrapingBee control — render_js, premium_proxy, wait, country_code, and so on — is entirely your business, forwarded exactly as sent. ScrapingBee’s own credential never touches your process: NativePort injects it server-side into the request, so nothing you send needs to (or can) carry your own ScrapingBee key. render_js defaults to true on ScrapingBee’s side even if you omit it, so the example above is explicit mostly for clarity.

A plain scrape returns the rendered page’s raw HTML, Content-Type: text/html. Three response headers are worth reading on every call: Spb-cost (credits this request actually charged), Spb-resolved-url (where you landed, if the page redirected), and Spb-initial-status-code (the status the target page itself returned, useful specifically when it redirected).

Waiting for content that loads late

Rendering doesn’t mean the page is done — a lot of client-side content finishes loading after the initial render. wait_for (a CSS or XPath selector) blocks until that element appears in the DOM; wait (milliseconds, up to 35000) adds a flat pause on top. If you use both, ScrapingBee runs wait_for first, then wait:

status, headers, body = scrape(
    "https://example.com",
    render_js="true",
    wait_for="#content-loaded",
)

For actual interaction — clicking, scrolling, filling a field before capturing — js_scenario takes a stringified JSON list of steps (click, wait, wait_for, fill, scroll_x/scroll_y, evaluate, infinite_scroll), executed after any wait. A whole scenario is capped at 40 seconds; ScrapingBee times the call out if it runs longer.

Structured extraction instead of raw HTML

extract_rules turns the response into JSON built from CSS selectors, so you skip parsing the HTML yourself:

extract_rules = json.dumps({
    "title": "h1",
    "links": {"selector": "a", "output": "@href", "type": "list"},
})

status, headers, body = scrape(
    "https://example.com",
    render_js="true",
    extract_rules=extract_rules,
)
data = json.loads(body)
print(data["title"], len(data["links"]))

A bare selector string ("h1") returns that element’s text. The expanded form ({"selector": ..., "output": ..., "type": ...}) lets you pull an attribute instead of text ("output": "@href") and collect every match rather than just the first ("type": "list").

Cost control

ScrapingBee bills in credits, and the multiplier depends entirely on which rendering/proxy tier you ask for — cost you control directly through the flags you set, not something that varies unpredictably per page:

Configuration Credits
No JS rendering, standard proxy 1
JS rendering (render_js=true, the default) 5
Premium proxy, no JS 10
Premium proxy + JS rendering 25
Stealth proxy (stealth_proxy=true; forces JS rendering on) 75

Only successful requests are billed — ScrapingBee’s own documented rule is that a 200, 404, or 410 response is charged; other failures aren’t. The cheapest correct configuration is the one to reach for by default: skip render_js entirely for pages that don’t need it (static HTML, an API response, a server-rendered page) to drop from 5 credits to 1, and only reach for premium_proxy or stealth_proxy when a page is actually blocking the standard tier — ScrapingBee’s own guidance is to escalate in that order (try without JS first, then wait/wait_for tuning, then premium_proxy, then stealth_proxy as the last resort) rather than defaulting to the most expensive tier up front. block_resources also defaults to true (images and CSS aren’t fetched), which keeps both cost and render time down — but some pages genuinely depend on those resources to render correctly, so set block_resources=false first if a page comes back visibly broken, before reaching for a more expensive proxy tier to fix what’s actually a resource-blocking problem.

Errors you’ll actually hit

  • 402 Payment Required (from NativePort): the NativePort balance is at $0. Distinct from a ScrapingBee-side credit issue — this is your balance, checked before the request leaves the gateway.
  • 401 Unauthorized: the Authorization: Bearer <NATIVEPORT_API_KEY> header is missing, malformed, or doesn’t resolve to an account.
  • 403 Forbidden: the key is valid but the account isn’t active.
  • 400 (from ScrapingBee): malformed request — most often an incorrectly URL-encoded url value. Make sure url is passed through urlencode (as in the examples above), not concatenated raw.
  • 429 (from ScrapingBee): too many concurrent requests for the shared account’s tier. Back off and retry rather than immediately resending.
  • 500 (from ScrapingBee): the render itself failed. Check the response body for ScrapingBee’s own error detail, then work through the escalation path above (block_resources=false, longer wait, premium_proxy, stealth_proxy) rather than retrying the identical request.

Security

Treat NATIVEPORT_API_KEY like any other credential: environment variable or secret store, never a literal string in source, never logged. ScrapingBee’s own key never reaches your process — the gateway injects it server-side, and a client-supplied key in any channel ScrapingBee itself accepts (query param, Authorization header, X-API-KEY header) is stripped rather than honored, so there’s no way to authenticate with a different ScrapingBee credential through this route.

What this costs

NativePort reads the Spb-cost header ScrapingBee returns on every call and meters exactly that, at ScrapingBee’s own per-credit rate, with no markup. Because cost is driven entirely by the flags in your own request (the table above), it’s predictable per call rather than something that varies with page size or content. Current per-credit pricing is on the ScrapingBee provider page. A zero balance answers every request with 402 instead of degrading. Adding credit carries a flat 5.5% fee on top-ups between $10 and $5,000 — never on the calls themselves. Full breakdown: pricing.

Where to go next