feat(pdf): page limit for PDF parsing on preview teams by abimaelmartell · Pull Request #4439 · firecrawl/firecrawl

abimaelmartell · 2026-08-28T08:20:23Z

Problem

A single request without an API key can currently parse a PDF of unbounded size — multi-thousand-page documents go through the full GPU pipeline on the free tier. PDFs are a small fraction of keyless requests but a heavily disproportionate share of keyless processing cost, driven by a thin tail of very large documents.

Change

Preview/keyless teams (team_id prefix preview, covering both the legacy preview_<iptoken> and keyless preview_keyless_<ip> shapes) get a PDF page ceiling:

PREVIEW_PDF_MAX_PAGES (default 500 ≈ the free-tier p99; 0 disables). Documents whose effective page count exceeds it are rejected with a new typed error, SCRAPE_PDF_PAGE_LIMIT_EXCEEDED, whose message points at signing up for an API key or passing an in-limit parsers: [{ "type": "pdf", "maxPages": … }].
Enforced on the effective page count, so an explicit maxPages at or under the ceiling still parses the first N pages of a larger document.
Reject, don't truncate: silently capping via maxPages would bypass the PDF result cache (read and write) and hand free-tier users incomplete documents without telling them.
Best-effort by design: if page detection fails, the request proceeds as before.
Keyed teams are completely unaffected.

Applied at the PDF engine entry, so scrape, search, crawl, and parse paths all inherit the same rule.

Tests

New unit suite for the predicate and error serde round-trip; all PDF engine suites pass (151 tests); tsc --noEmit clean.

🤖 Generated with Claude Code

Summary by cubic

Caps free-tier PDF parsing at 500 pages per request so a single keyless scrape can no longer consume unbounded GPU time on very large documents.

Preview/keyless teams (team_id prefix preview) are rejected with the new typed error SCRAPE_PDF_PAGE_LIMIT_EXCEEDED, whose message points at signing up for an API key or passing an in-limit maxPages.
The limit is configurable via PREVIEW_PDF_MAX_PAGES (default 50; 0 disables) and enforced on the effective page count, so an explicit in-limit maxPages still parses the first N pages of a larger document.
Enforcement rejects rather than truncates, since silent truncation via maxPages would bypass the PDF result cache and hand free-tier users incomplete documents.
If page detection fails, the request proceeds as before; keyed teams are unaffected.
The check lives at the PDF engine entry, so scrape, search, crawl, and parse paths share the same rule.

^{Written for commit f56b9ea. Summary will update on new commits.}

Adds PREVIEW_PDF_MAX_PAGES (default 500, 0 disables). Preview/keyless teams (team_id prefix "preview") parsing a PDF whose effective page count exceeds the limit get a typed SCRAPE_PDF_PAGE_LIMIT_EXCEEDED error with guidance instead of a parse. - Enforced on the effective page count, so an explicit maxPages at or under the limit still parses the first N pages of a larger document. - Reject rather than truncate: an implicit maxPages would change cache behavior and silently return incomplete documents. - Best-effort: when page detection fails the request proceeds. - Keyed teams are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cubic-dev-ai

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/api/src/scraper/scrapeURL/engines/pdf/index.ts">

<violation number="1" location="apps/api/src/scraper/scrapeURL/engines/pdf/index.ts:433">
P1: When Rust extraction is enabled and no in-limit `maxPages` is supplied, this guard runs only after `processPdf` has extracted the entire document. A preview request for a multi-thousand-page PDF therefore still incurs the full extraction cost before returning the page-limit error; run metadata-only `detectPdf` first and reject before `processPdf` for over-limit documents.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai · 2026-08-28T08:30:14Z

+    // regardless of extraction path so the policy is one predictable rule.
+    // Best-effort by design: when page detection fails (effectivePageCount
+    // stays 0) the request proceeds and downstream accounting still applies.
+    if (


P1: When Rust extraction is enabled and no in-limit maxPages is supplied, this guard runs only after processPdf has extracted the entire document. A preview request for a multi-thousand-page PDF therefore still incurs the full extraction cost before returning the page-limit error; run metadata-only detectPdf first and reject before processPdf for over-limit documents.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At apps/api/src/scraper/scrapeURL/engines/pdf/index.ts, line 433: <comment>When Rust extraction is enabled and no in-limit `maxPages` is supplied, this guard runs only after `processPdf` has extracted the entire document. A preview request for a multi-thousand-page PDF therefore still incurs the full extraction cost before returning the page-limit error; run metadata-only `detectPdf` first and reject before `processPdf` for over-limit documents.</comment> <file context> @@ -419,6 +424,25 @@ export async function scrapePDF(meta: Meta): Promise<EngineScrapeResult> { + // regardless of extraction path so the policy is one predictable rule. + // Best-effort by design: when page detection fails (effectivePageCount + // stays 0) the request proceeds and downstream accounting still applies. + if ( + exceedsPreviewPdfPageLimit( + meta.internalOptions.teamId, </file context>

cubic-dev-ai

1 existing issue remains and 2 new issues found across 9 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/api/src/scraper/scrapeURL/engines/pdf/index.ts">

<violation number="1" location="apps/api/src/scraper/scrapeURL/engines/pdf/index.ts:434">
P1: When native extraction fails or undercounts, FirePDF can raise `effectivePageCount` above the limit after this guard has already run. Reapply the preview limit after each FirePDF reconciliation, or perform one final check before returning the result.</violation>
</file>

<file name="apps/api/src/config.ts">

<violation number="1" location="apps/api/src/config.ts:232">
P2: The `PREVIEW_PDF_MAX_PAGES` default is 50, but the PR specifies a default of 500 (≈ the free-tier p99) and every other artifact in this change treats 500 as the canonical limit (the tests hardcode `500`, and the error message text reads "500-page limit"). With the current default, preview/keyless requests are rejected at 50 effective pages instead of the intended 500, incorrectly rejecting legitimate mid-size documents. Align the default to 500 or update the PR/tests if 50 is truly intended.</violation>
</file>

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai · 2026-08-28T08:34:52Z

+    // Best-effort by design: when page detection fails (effectivePageCount
+    // stays 0) the request proceeds and downstream accounting still applies.
+    if (
+      exceedsPreviewPdfPageLimit(


P1: When native extraction fails or undercounts, FirePDF can raise effectivePageCount above the limit after this guard has already run. Reapply the preview limit after each FirePDF reconciliation, or perform one final check before returning the result.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At apps/api/src/scraper/scrapeURL/engines/pdf/index.ts, line 434: <comment>When native extraction fails or undercounts, FirePDF can raise `effectivePageCount` above the limit after this guard has already run. Reapply the preview limit after each FirePDF reconciliation, or perform one final check before returning the result.</comment> <file context> @@ -419,6 +424,25 @@ export async function scrapePDF(meta: Meta): Promise<EngineScrapeResult> { + // Best-effort by design: when page detection fails (effectivePageCount + // stays 0) the request proceeds and downstream accounting still applies. + if ( + exceedsPreviewPdfPageLimit( + meta.internalOptions.teamId, + effectivePageCount, </file context>

cubic-dev-ai · 2026-08-28T08:34:52Z

+  // check. Explicit maxPages at or under the ceiling is honored (parsing
+  // the first N pages of a larger document stays allowed).
+  PREVIEW_PDF_MAX_PAGES: emptyStringAsDefault(
+    z.coerce.number().int().nonnegative().default(50),


P2: The PREVIEW_PDF_MAX_PAGES default is 50, but the PR specifies a default of 500 (≈ the free-tier p99) and every other artifact in this change treats 500 as the canonical limit (the tests hardcode 500, and the error message text reads "500-page limit"). With the current default, preview/keyless requests are rejected at 50 effective pages instead of the intended 500, incorrectly rejecting legitimate mid-size documents. Align the default to 500 or update the PR/tests if 50 is truly intended.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At apps/api/src/config.ts, line 232: <comment>The `PREVIEW_PDF_MAX_PAGES` default is 50, but the PR specifies a default of 500 (≈ the free-tier p99) and every other artifact in this change treats 500 as the canonical limit (the tests hardcode `500`, and the error message text reads "500-page limit"). With the current default, preview/keyless requests are rejected at 50 effective pages instead of the intended 500, incorrectly rejecting legitimate mid-size documents. Align the default to 500 or update the PR/tests if 50 is truly intended.</comment> <file context> @@ -223,6 +223,15 @@ const configSchema = z.object({ + // check. Explicit maxPages at or under the ceiling is honored (parsing + // the first N pages of a larger document stays allowed). + PREVIEW_PDF_MAX_PAGES: emptyStringAsDefault( + z.coerce.number().int().nonnegative().default(50), + ), + </file context>

Suggested change

z.coerce.number().int().nonnegative().default(50),

z.coerce.number().int().nonnegative().default(500),

abimaelmartell requested a review from mogery as a code owner August 28, 2026 08:20

abimaelmartell force-pushed the feat/preview-pdf-page-cap branch from 197262d to 05b9b55 Compare August 28, 2026 08:21

abimaelmartell changed the title ~~feat(pdf): free-tier page ceiling for preview/keyless PDF parsing~~ feat(pdf): page limit for PDF parsing on preview teams Aug 28, 2026

abimaelmartell force-pushed the feat/preview-pdf-page-cap branch from 05b9b55 to 4a8a314 Compare August 28, 2026 08:21

abimaelmartell force-pushed the feat/preview-pdf-page-cap branch from 4a8a314 to f56b9ea Compare August 28, 2026 08:23

cubic-dev-ai Bot reviewed Aug 28, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(pdf): page limit for PDF parsing on preview teams - #4439

feat(pdf): page limit for PDF parsing on preview teams#4439
abimaelmartell wants to merge 1 commit into
mainfrom
feat/preview-pdf-page-cap

abimaelmartell commented Aug 28, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	z.coerce.number().int().nonnegative().default(50),
	z.coerce.number().int().nonnegative().default(500),

Uh oh!

Conversation

abimaelmartell commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Tests

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

abimaelmartell commented Aug 28, 2026 •

edited

Loading

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading

cubic-dev-ai Bot Aug 28, 2026 •

edited

Loading