Feature/partner-search: turbopuffer rollout with filters and exact counts - #4333
Feature/partner-search: turbopuffer rollout with filters and exact counts#4333Joker666 wants to merge 30 commits into
Conversation
…full-text-search-rollout
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The provider now narrows on the discrete enrollment fields instead of leaving every filter to the database, which applied them to the already truncated candidate list. A broad query with country=US returned 85 rows out of 7,698 real matches; measured on 5,000 seeded partners, filtering inside the query takes that from 20% to 100% of the true set. Metrics stay out on purpose: they move on every click and conversion, so indexing them would make the document churn continuously. Namespace moves to v3 — the new attributes do not exist on v2 rows.
The partner count API was over-counting when a referral filter was present because the database-only fallback did not include `referredByPartnerId`. This adds that filter to the fallback guard and covers it with a search test so partner counts remain accurate when the provider cannot evaluate the filter.
This change avoids expensive Turbopuffer count aggregations for very short final search tokens, where prefix expansion can make the query too slow. The provider now returns null when it should decline counting, and getPartnersCount falls back to the database count in that case. Tests cover both the fallback path and the short-query guard.
countCandidates returned 0 when the response carried no aggregate, which rendered an unanswered count as an exact empty result now that null means the provider cannot answer. The fallback tests mocked a database count larger than the candidate list they were constrained to, which cannot happen under a real id IN clause and hid the fact that every fallback path is capped at the candidate ceiling. They now assert that constraint instead.
…ch' into feature/partner-full-text-search-rollout
package.json moved back to ^1.35.3 without regenerating the lockfile, so CI's frozen-lockfile install failed before building. The resolved version is unchanged — 1.38.2 satisfies the range — so only the specifier moves.
Both described a two-provider world that this branch removed, and the first also described the wrong filtering model. types.ts said filters run after the truncation and called indexing them "out of scope here" — but that is what this branch implemented, and turbopuffer.ts says the opposite three files away. Anyone reading the constant would have understood the model backwards. It now names what the provider filters before the cut (status, group, country, tags) and what the database filters after it (tenant, explicit partner IDs, referral, metric ranges), matching the schema and the databaseOnlyFilters list. The 999 ceiling is described as inherited from the removed Redis provider rather than derived, since nothing enforces it now. The floor beyond it is framed as the accepted trade it is, with what raising it would cost on every query, rather than as a known limitation inviting a fix. shared.ts loses the Redis-versus-Turbopuffer packaging explanation and the "both providers store" phrasing.
…full-text-search-rollout
…full-text-search-rollout # Conflicts: # apps/web/lib/api/partners/search/providers/shared.ts # apps/web/lib/api/partners/search/providers/turbopuffer.ts # apps/web/lib/api/partners/search/types.ts # apps/web/scripts/dev/seed-100k-partners.ts
House style, matching the same pass on the base branch.
The note carried the Redis provider's history and a long defense of the trade-off. Keeps the filter order, the lower-bound behavior, and what raising the ceiling costs.
ff819a3 to
39dec36
Compare
…full-text-search-rollout
cf77de9 to
613d1c8
Compare
ac86908 to
0af20a5
Compare
debug-partner-search kept its own copy of the namespace name, which stayed on partner-search-v2 when the filterable-attribute change bumped the provider to v3. Its candidate search went to v3 through the provider while its indexed-text lookup went to v2, so --searchOnly reported empty names and zero tokens instead of failing. Export the constant and import it, so the two cannot drift again.
…full-text-search-rollout
The candidate path selects the database route on the trimmed value, so a pasted " pn_… " or " user@x.com " reached the exact match untrimmed and found nothing.
ca9c200 to
5c0eb16
Compare
The digit sniff could classify a permanent failure as transient when an ID or count in the message contained 500 or 429. Typed APIError carries the status; message matching remains only for non-SDK network errors.
…full-text-search-rollout
…full-text-search-rollout # Conflicts: # apps/web/lib/api/partners/search/providers/turbopuffer.ts
Partner search rollout
This narrows the proof of concept to something deployable using Turbopuffer.
What changed
Upstash Redis Search removed, along with
PARTNER_SEARCH_PROVIDER,PARTNER_SEARCH_INDEX_NAME, andwaitForIndexing.@upstash/redisrolls back to^1.35.3— the package stays for caching and rate limiting, only the bump the Redis search API needed is reverted. ThePartnerSearchProviderinterface stays; only the multi-provider registry went.TURBOPUFFER_API_KEYis now the feature flag — unset still falls back to database search, so this ships dark.Exact email and partner ID skip the provider and use the indexes the database already has. An email that matches nothing still falls through to search, because that's usually an address being typed (
steven@dub.coon the way tosteven@dub.com). The ID rule ispn_+ 24 characters minimum, from a full scan of all 626,348 production documents — a fixed-length check would have dropped 1 partner in 280.status,groupId,country, andpartnerTagIdsare indexed and filtered before the ranking truncates. This is the main fix. On 5,000 seeded partners with 1,000 real matches, a filtered search returned 201 rows before, 999 after. Metrics stay database-only — they change on every click, and indexing them would make documents churn continuously.Counts are exact, via one aggregation rather than counting the truncated candidate list ("999 partners" when there are twelve thousand). It returns
number | nulland falls back to the old count when it can't answer.Seed script now varies status, groups, and tags, including nulls and empties. Previously every partner was approved, ungrouped, and untagged.
Testing
110 unit tests, covering the paths that fail quietly: email hit/miss routing, partner IDs at 24 and 25 characters and one short, filters reaching every branch with correct exclusion semantics, an empty filter list not becoming
In [], all four count fallbacks, and tags from other programs being dropped.Known limitation: filter dropdown counts are floors
The exact count above covers the ungrouped total only. The per-value counts in the filter dropdowns (status, group, country, tag) are still grouped from the truncated 999 candidates.
How users see it: on a search matching more than 999 partners, the header reports the real total (say "12,000 partners") while a dropdown shows something like "Approved 730", counted from the top 999 only. The numbers will not add up. Searches matching fewer than 999 are unaffected.
Exact dropdown counts are deferred to a follow-up: they need one aggregation per dropdown, so a single screen can create many Turbopuffer queries.
Addressed limitations