Build a reliable Firecrawl scrape client in TypeScript

A small client with production-shaped behavior

By the end, you will have a dependency-free TypeScript function that returns clean page content, preserves the provider’s error message, honors Retry-After, and retries only failures that can plausibly succeed later.

Prerequisites

• Node.js 20 or later, so built-in fetch is available.
• A Firecrawl API key stored in FIRECRAWL_API_KEY.
• A public http:// or https:// page to scrape.

Send a scrape request

Firecrawl’s v2 Scrape endpoint accepts a URL and one or more output formats. This request asks for the main page content as Markdown plus a separate list of discovered links.

const response = await fetch(
  "https://api.firecrawl.dev/v2/scrape",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.FIRECRAWL_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: "https://example.com/docs",
      formats: ["markdown", "links"],
      onlyMainContent: true,
      timeout: 60_000,
    }),
  },
);

const payload = await response.json();

if (!response.ok || !payload.success) {
  throw new Error(payload.error ?? `HTTP ${response.status}`);
}

console.log(payload.data.markdown);

Why these options?

Model both success and failure

A useful client keeps failure data typed instead of collapsing every problem into a generic exception. Preserve the human-readable error and leave room for structured details.

type ScrapeData = {
  markdown?: string;
  links?: string[];
  metadata?: {
    title?: string;
    sourceURL?: string;
    statusCode?: number;
  };
};

type FirecrawlSuccess = {
  success: true;
  data: ScrapeData;
};

type FirecrawlFailure = {
  success: false;
  error: string;
  details?: unknown;
};

type FirecrawlResponse =
  | FirecrawlSuccess
  | FirecrawlFailure;

This union gives the caller a clear narrowing point. It also prevents code from assuming data exists after a failed request.

Retry only transient failures

Authentication, validation, and exhausted-credit errors need a person or configuration change. Timeouts, rate limits, and selected server failures may succeed later. The client below retries the latter, applies capped exponential backoff with jitter, and honors Retry-After when present.

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

const sleep = (ms: number) =>
  new Promise((resolve) => setTimeout(resolve, ms));

function retryDelay(response: Response, attempt: number) {
  const retryAfter = response.headers.get("Retry-After");
  if (retryAfter) return Number(retryAfter) * 1_000;

  const exponential = Math.min(2 ** attempt * 1_000, 30_000);
  return exponential + Math.random() * 500;
}

export async function scrapePage(
  url: string,
  maxAttempts = 4,
): Promise<ScrapeData> {
  const apiKey = process.env.FIRECRAWL_API_KEY;
  if (!apiKey) throw new Error("FIRECRAWL_API_KEY is not set");

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const response = await fetch(
      "https://api.firecrawl.dev/v2/scrape",
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${apiKey}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          url,
          formats: ["markdown", "links"],
          onlyMainContent: true,
          timeout: 60_000,
        }),
      },
    );

    const payload = (await response.json()) as FirecrawlResponse;

    if (response.ok && payload.success) {
      return payload.data;
    }

    const message = payload.success
      ? `HTTP ${response.status}`
      : payload.error;

    const finalAttempt = attempt === maxAttempts - 1;
    if (!RETRYABLE.has(response.status) || finalAttempt) {
      throw new Error(message);
    }

    await sleep(retryDelay(response, attempt));
  }

  throw new Error("Unreachable");
}

Make freshness, cost, and privacy explicit

Use the cache intentionally

Choose a cache age from the product’s freshness requirement instead of forcing every request to recrawl. “Always fresh” is a cost and latency decision, not a neutral default.

Handle concurrency, not just requests per minute

A rate-limit response can reflect a per-minute quota or a concurrent-browser limit. Honor Retry-After and cap your own worker concurrency.

Decide whether content may be cached

When the target or workflow has data-protection concerns, use the provider’s cache and retention controls deliberately. Do not treat scraped content as harmless simply because it arrived through an API.

Log a request identifier

Record the target domain, status, attempt count, duration, and provider request ID when available. Do not log API keys or sensitive page content by default.

Diagnose by status before changing code

Why this guide is structured this way

The reader gets a working request before encountering abstraction. Each later section answers a production question that appears after the first successful call: what can I trust in the response, which failures should I retry, and how do I balance freshness, cost, and privacy?

The code stays dependency-free so the API contract remains visible. This independent candidate sample was written from public provider documentation; verify current behavior in the official Scrape endpoint reference, errors and retry guide, and API documentation.