Discovery Platform Architecture and Delivery Design

1. Purpose

This document turns search-prd.txt into an implementation design for an owned discovery platform. It covers the recommended technology stack, system boundaries, high-level and low-level designs, data contracts, feature delivery sequence, safety controls, operations, and decision criteria for evolving the search engine. It extends the existing platform instead of creating a parallel discovery stack: Compass remains the customer-facing discovery service; Atlas remains the catalogue edge; Sesh provides anonymous/session context; Tetris composes CMS-backed content and widgets; Medusa, Storehouse, and Dermadesk remain sources for commerce, availability, and prescription/consultation context. The complete architecture remains in this document. Discovery platform low-level design is an additive companion focused on executable query parsing, TypeSense request construction, ranking execution, event routing, and use-case examples. The first production goal is deliberately narrow:

A user can search for a product, brand, concern, or ingredient; receive safe and relevant results; have the interaction measured; and have a merchandiser correct a bad outcome without an engineering deployment.

The platform is not a search engine replacement alone. It is a product layer around candidate retrieval that owns catalogue quality, safety eligibility, ranking policy, merchandising, attribution, experimentation, and operational visibility.

2. Architecture Decisions

Decision	Recommendation	Reason
Initial search engine	Self-hosted open-source Typesense	Supports typo tolerance, fields, facets, filtering, grouping, autocomplete, and native search analytics while retaining operational/data control.
Search ownership	Extend Compass (FastAPI) as the application-owned Discovery API	Preserves its existing ownership of search, browse, autosuggest, and recommendations while keeping domain rules independent of the engine.
System of record	Existing commerce/catalogue services	Discovery stores an optimized read model, never the authoritative product record.
Operational database	A Compass-owned PostgreSQL schema/database, operated through existing platform patterns	Reliable transactional store for rules, configuration, audits, experiments, and schedules without adding a new service boundary.
Runtime cache	Existing Redis platform	Low-latency configuration cache, idempotency keys, rate limits, and short-lived result caching.
Compute	Existing Compass FastAPI API and worker deployments on Kubernetes	Matches the established Helm, Argo CD, KEDA, Datadog, Prometheus, and Loki platform.
Ingestion transport	SNS/SQS queues	Matches the established asynchronous transport, DLQ, and worker model used by Atlas, Storehouse, Gearbox, and other services.
Phase 1 analytics	Typesense native analytics + PostHog + Compass PostgreSQL	Typesense owns search-native feedback; PostHog owns cross-surface funnels, revenue and experiments; Compass retains inspectable operational evidence.
Data warehouse	Deferred behind concrete cross-system/reporting requirements	Do not block search launch on a new data platform when PostHog and authoritative commerce events meet Phase 1 needs.
Deployment	Existing Helm charts, Argo CD, and DevOps Terraform/traffic-split tooling	Matches the documented production deployment and progressive delivery model.
Observability	Existing Datadog, Prometheus, Grafana, Loki, and PostHog	Covers metrics, tracing, dashboards, logs, and product analytics without creating duplicate tooling.
Semantic retrieval	Deferred; add as a second candidate source only	Lexical search, catalogue attributes, and measurement must be trustworthy first.
Advanced engine	OpenSearch evaluated later through shadow traffic	Use only when concrete relevance, scale, or cost evidence justifies its higher complexity.

3. Why This Stack

3.1 Typesense for the first production version

Typesense matches the Phase 1 requirements with low implementation and operational burden:

Fielded full-text search with adjustable text-match weights.
Typo tolerance, prefix matching, infix matching where appropriate, and search-as-you-type.
Facets, numeric filters, Boolean filters, sorting, and filter counts.
Grouping by canonical product ID to prevent variant duplicates.
Synonym support and curated result overrides where they are useful.
Fast response times without designing shard topology, analyzers, or a complex query DSL from day one.
Native analytics rules for popular/no-hit queries, document counters, and pseudonymous behavioral logs.
One self-contained server binary with a clear Kubernetes deployment model.

Typesense should be treated as a retrieval dependency, not as the discovery platform. Product policy must not be encoded only in its collection schema or its curation features.

3.2 Why not Meilisearch by default

Meilisearch is a strong choice for simple product search. It can be the right option for a lightweight MVP if there is no near-term need for detailed ranking governance or complex discovery controls. This PRD, however, requires a path toward:

Query-, page-, and context-specific merchandising rules.
Explicit rank-layer precedence and per-result explanations.
Product/variant family behavior.
Rich per-page filter and sorting configuration.
Controlled behavioral ranking and experiments.
Hybrid lexical and semantic candidate retrieval.

Typesense gives a better early balance between simple operation and control over retrieval. The platform design below also makes a later move to either Meilisearch or OpenSearch possible.

3.3 Why not OpenSearch initially

OpenSearch is mature and highly capable. It becomes attractive when the business has proven needs for complex custom scoring, advanced analyzers, nested variant documents, very high throughput, or a mature search operations team. It should not be the default starting point because it brings forward substantial work:

Index mappings and analyzer strategy are expensive to change and require safe reindexing.
Production reliability requires shard, replica, snapshot, storage, scaling, and upgrade decisions.
Query DSL, scoring, relevance tuning, and performance diagnostics require specialist expertise.
A resilient managed deployment has a materially higher baseline cost than a small managed Typesense deployment.

None of those costs eliminate the need to build the Discovery API, catalogue transformation, rule store, analytics, safety layer, and inspectors. Those are the durable capabilities in this PRD.

3.4 When to move to OpenSearch

Run an OpenSearch evaluation after Phase 2 or Phase 3 only when one or more evidence-based triggers is met:

Typesense cannot express an important retrieval or ranking requirement without fragile workarounds.
The catalogue or query volume makes Typesense materially less cost-effective.
Strong hybrid retrieval or vector-search quality is a validated requirement.
Explainability requires score details unavailable from the current engine.
The team has committed operational ownership and has people experienced with OpenSearch.

Migration must use dual indexing, shadow queries, golden-query comparison, relevance review, and gradual traffic shifting. The public Discovery API does not change.

4. Guiding Principles

Safety and eligibility are server-side invariants. A client, rule, experiment, pin, model, or future sponsored placement cannot reintroduce an ineligible item.
The catalogue is the quality bottleneck. Attribute coverage and freshness are features, not data-cleanup work outside the platform.
Rules are data, not deployments. Routine merchandising changes are versioned configuration with previews and audit records.
Every result is attributable. The platform records the request, configuration version, returned products, positions, and downstream actions.
Search is lexical first. Exact names, brands, and product families must be dependable before semantic matching is introduced.
Do not create a service per PRD module. Begin with a cohesive Discovery API and a small set of worker processes. Split only at proven scaling or ownership boundaries.
Configuration needs a rollback story. All rule and profile changes must be previewable, schedulable, audited, and reversible.
Defaults must work without personalization. Personalization only reorders already eligible, relevant products and can be disabled per surface.
Production launches are measured and staged. Shadow mode, canaries, regression tests, and health signals are mandatory for material changes.

5. High-Level Design

                              +--------------------------+
                              | Atlas / Commerce         |
                              | product, variant, price, |
                              | stock, listing, taxonomy |
                              +------------+-------------+
                                            |
                              SNS/SQS product and inventory events
                                           |
                                           v
  +--------------------+        +--------------------------+        +-------------------+
  | Merchandising      |------->| Discovery configuration  |<-------| Product / Admin   |
  | console            |        | PostgreSQL + audit log   |        | operators         |
   +--------------------+        +------------+-------------+        +-------------------+
                                            |
                                            | cached configuration
                                           v
+-------------------+         +--------------------------+         +-------------------+
  | Web / mobile apps |-------->| Compass Discovery API    |-------->| Typesense         |
| search, PLP, PDP  |         | orchestration and policy |         | candidate index   |
+-------------------+         +-----+---------+----------+         +-------------------+
                                      |         |
                                      |         +--------------------+
                                      v                              v
                           +---------------------+          +---------------------+
                            | Analytics            |          | Inspector / explain |
                            | Typesense + PostHog |          | query/page debug UI |
                           +----------+----------+          +---------------------+
                                      |
                                      v
                            +---------------------+
                            | Future warehouse    |
                            | only when triggered |
                            +---------------------+

                    +------------------------------+
                    | Compass/Gearbox index workers |
                    | SNS/SQS -> transform          |
                    | -> Typesense upsert/delete    |
                    +------------------------------+

5.1 Runtime request path

Client request
  -> authenticate/session context
  -> request validation and rate limiting
  -> load active ranking profile and applicable rules
  -> normalize query and protect terms
  -> classify intent and extract filters/preferences
  -> check redirect rules
  -> retrieve lexical candidates from engine
  -> apply mandatory eligibility/safety policy
  -> group product variants and choose display variant
  -> score/re-rank using profile and approved boosts
  -> apply merchandiser boost/bury/hide rules
  -> apply deterministic pins
  -> execute fallback ladder if needed
  -> build facets, UI metadata, and search ID
  -> asynchronously persist exposure event
  -> return response

The Discovery API owns every stage except raw candidate retrieval. It can use engine-native capabilities for efficiency, but it must preserve enough information to explain the final result.

6. Service Boundaries

Begin with four deployable applications. Their code may live in one monorepo.

Application	Responsibility	Initial scale model
`Compass API`	Search, autosuggest, browse retrieval, ranking orchestration, eligibility, rules, response generation, event emission	Existing FastAPI deployment, Helm release, and KEDA scaling model
`Compass index worker`	Consume catalogue/inventory changes, validate and transform products, index delta updates, execute full rebuilds	Queue-driven worker deployment with SNS/SQS, retry policy, and DLQ
`Compass admin API`	Internal control-plane endpoints for rules, profiles, previews, audits, and approvals	Begin within Compass; a separate UI can be introduced when operator workflows mature
`Gearbox/analytics jobs`	Typesense analytics review, PostHog dashboard support, demand-gap jobs, reconciliation and quality reports	Existing scheduled/background job patterns; warehouse transforms only after a future adoption decision

Do not make ranking, synonym management, personalization, recommendations, or experiments independent services until they have independent load, release cadence, data storage, or ownership requirements.

7. Infrastructure Design

7.1 AWS resources

Kubernetes
  Compass API deployment and Compass index-worker deployment
  Helm values for DEVELOPMENT, STAGING, and PRODUCTION
  KEDA ScaledObjects for API and queue-driven worker scaling

Platform data stores
  Compass PostgreSQL and Redis
  Self-hosted Typesense 3-node production cluster with persistent data and analytics volumes
  S3 for Typesense snapshots and index rebuild artifacts

Data and events
  SNS topic and SQS index queue + dead-letter queue
  Typesense native analytics and PostHog server/client event delivery

Operations
  AWS Secrets through the existing CSI configuration
  Datadog, Prometheus, Grafana, and Loki
  Argo CD progressive rollout and Terraform traffic-split tooling

7.2 Environments

Maintain at least development, staging, and production.

Development: synthetic catalogue subset, seed configuration, no production personal data.
Staging: production-like schemas and anonymized/sampled catalogue; validates migrations, index builds, and golden-query tests.
Production: separate database, cache, queues, engine collections, credentials, and dashboards.

Every configuration entity contains environment-local IDs; never promote a production rule by copying database rows manually. Use versioned export/import or an approved configuration promotion flow.

7.3 Availability and degraded operation

The discovery-api must return a usable response when a downstream dependency is degraded:

Dependency failure	Expected behavior
Typesense unavailable	Return cached popular search/category fallback where safe; route to browse/category experience; emit incident event.
Redis unavailable	Read configuration from PostgreSQL at a protected rate; disable non-essential result caching.
PostgreSQL unavailable	Use recently loaded, versioned read-only configuration cache; block admin mutations; alert.
Event pipeline unavailable	Queue events locally/durably where possible; never block search responses on analytics.
Catalogue feed delayed	Continue serving last valid index, alert on freshness SLO violation.

No degraded response may violate Rx, listing, or price/availability safety requirements. Cached responses should be limited to contexts where eligibility remains valid.

8. Data Model

7.4 Self-hosted Typesense operations

Production runs a three-node Typesense cluster. Typesense uses Raft consensus and replicates the full dataset to every node; three nodes tolerate one node failure. The Raft peering network remains private, while Compass connects through the internal load-balanced Typesense service and configures individual nodes as fallbacks. Each node requires persistent SSD/NVMe-backed volumes for both the Typesense data directory and analytics directory. The cluster configuration enables native analytics explicitly:

--enable-search-analytics=true
--analytics-dir=/var/lib/typesense-analytics
--analytics-flush-interval=60

The 60-second flush interval is the documented minimum. It gives sufficiently fresh popular-query/no-hit data for operational review without treating analytics as a synchronous request dependency. Benchmark this configuration with production-like search, filter, and indexing traffic because native analytics consumes cluster resources. Operational requirements:

Use a predefined schema and leave display-only fields unindexed to reduce memory use.
Plan keyword-search RAM at roughly 2-3 times the combined size of indexed/searchable/filterable/sortable values, then validate with the real catalogue.
Use collection aliases and versioned collections for incompatible schema changes.
Snapshot through Typesense’s snapshot API and copy the resulting snapshot to S3. Never copy a live data directory directly.
Monitor /health, /metrics.json, and /stats.json; alert before RAM reaches 85% or sustained CPU reaches 90%.
Use bulk imports for indexer batches. Treat 503 Not Ready/Lagging as backpressure, retry with jitter, and keep concurrent bulk imports at or below vCPU count - 2.
Keep analytics collections, analytics directories, API keys, and snapshots isolated per environment.
Set enable_analytics=false on tests and vendor-shadow queries so they do not contaminate production query counts/suggestions.

8.1 Discovery product document

Index at the canonical product-family level, not one document per SKU. This satisfies the PRD requirement to avoid near-duplicate variants while preserving variant choices for the PDP/cart.

type DiscoveryProduct = {
  id: string;                       // canonical product/family ID
  sourceProductId: string;
  sourceVersion: string;
  title: string;
  normalizedTitle: string;
  brandId: string;
  brandName: string;
  categoryIds: string[];
  categoryNames: string[];
  concernIds: string[];
  concernNames: string[];
  ingredientIds: string[];
  ingredientNames: string[];
  productType: string;
  routineStep?: string;
  formats: string[];
  skinHairTypes: string[];
  spf?: number;
  ingredientStrengths: Array<{ ingredientId: string; percentage?: number }>;
  searchableDescription: string;
  tags: string[];

  listable: boolean;
  purchasable: boolean;
  inStock: boolean;
  stockStatus: "in_stock" | "low_stock" | "out_of_stock";
  rxClassification: "none" | "prescription" | "consult_required";
  pregnancyBreastfeedingCautions: string[];

  price: { currency: string; minimum: number; maximum: number };
  rating?: number;
  reviewCount?: number;
  launchedAt?: string;
  commercialFlags: string[];

  displayVariants: Array<{
    variantId: string;
    label: string;
    price: number;
    inStock: boolean;
    attributes: Record<string, string | number | boolean>;
  }>;
  defaultDisplayVariantId: string;
  imageUrl: string;
  productUrl: string;

  updatedAt: string;
};

Required principles:

listable, purchasable, inStock, and rxClassification are distinct values.
Do not infer search facets from ungoverned descriptions at query time.
Attribute IDs are stable canonical IDs; display names can change without breaking rules or URLs.
Variants are included to select a sensible display option, but the top-level product remains the search/browse unit.
Product family, pack, and flavour policies must be explicit. The current open decision recommends grouping size variants but not distinct flavours.

8.2 PostgreSQL control-plane entities

ranking_profiles
  id, name, scope, definition_json, version, status,
  created_by, approved_by, created_at, published_at

merchandising_rules
  id, name, scope_type, scope_id, conditions_json, action_json,
  priority, starts_at, ends_at, status, requires_approval,
  version, created_by, updated_by, approval_state

typesense_rule_sync
  id, merchandising_rule_id, typesense_resource_type,
  typesense_resource_id, source_rule_version, sync_status,
  synced_at, last_error

rule_audits
  id, rule_id, action, before_json, after_json, reason,
  actor_id, occurred_at

synonym_sets
  id, canonical_term, alternatives_json, scope, status,
  created_by, approved_by, version

protected_terms
  id, term, term_type, locale, status, created_by, updated_at

redirect_rules
  id, query_match_type, query_value, destination_type,
  destination_id, starts_at, ends_at, status

campaigns
  id, name, starts_at, ends_at, status, owner_id, metadata_json

campaign_members
  campaign_id, member_type, member_id

experiments
  id, hypothesis, surface, assignment_unit, variants_json,
  primary_metric, guardrails_json, starts_at, ends_at, status

golden_cases
  id, case_type, input_json, assertions_json, severity, enabled

attribute_dictionary
  id, attribute_type, canonical_value, display_name, aliases_json,
  status, owner_id

Use JSONB for flexible condition/action definitions, but validate every entity through typed application schemas. Do not rely on arbitrary JSON interpreted at runtime without versioning and validation.

8.3 Bootstrap state and seeded configuration

Database migrations create empty tables, but the platform must not launch with an empty control plane. Bootstrap data is a reviewed, versioned configuration bundle imported after schema migration and before Typesense traffic is enabled.

Entity	Initial state	How it is populated
`attribute_dictionary`	Required before indexing	Catalogue/product-approved canonical categories, brands, concerns, ingredients, formats, routine steps, safety values, and aliases.
`ranking_profiles`	At least one active profile	Seed a `search-default` profile with conservative lexical weights and behavioral ranking disabled.
`protected_terms`	Required baseline	Import brand, product-family, molecule, and clinical terms from catalogue/compliance review.
`synonym_sets`	Curated baseline only	Import existing vendor synonyms after normalization, deduplication, and owner approval.
`redirect_rules`	Existing navigational redirects	Import active vendor redirects that remain intentional.
`golden_cases`	Required before shadow mode	100-200 anonymized priority queries plus expected outcomes and safety assertions.
`merchandising_rules`	Empty unless an approved migration/campaign needs a rule	Do not copy every legacy vendor manipulation blindly; recreate only reviewed rules.
`campaigns`, `campaign_members`	Empty until used	Created by operators after the basic rule flow is proven.
`experiments`	Empty until first controlled rollout	Provider rollout/experiment configuration is created explicitly, not inferred.
`typesense_rule_sync`	Empty until a rule/synonym is compiled	Derived operational state; never hand-seeded.
`rule_audits`	Empty at creation	Append-only records begin with seed imports and subsequent operator actions.

Seed imports must be idempotent, include a source/version and an owner, produce an audit record, and support dry-run validation. Treat seed data as configuration under change control, not application code and not manually edited production SQL.

8.4 Merchandising rule format

{
  "scope": {
    "surface": "search",
    "pageType": null
  },
  "conditions": {
    "all": [
      { "field": "query.normalized", "operator": "contains", "value": "sunscreen" },
      { "field": "request.platform", "operator": "equals", "value": "web" },
      { "field": "time.now", "operator": "between", "value": ["2026-04-01T00:00:00Z", "2026-06-30T23:59:59Z"] }
    ]
  },
  "action": {
    "type": "pin",
    "productId": "product_123",
    "position": 1
  },
  "priority": 100
}

Supported Phase 1 actions:

Redirect query to page/destination.
Pin a product at a deterministic position.
Hide a product from a scoped context.
Boost or bury an eligible product.
Add or disable a synonym/protected term.

Later actions include content injection, curated recommendation replacement, user-segment rules, campaigns, and bulk imports.

8.5 Phase 1 event model and destinations

The shared event contract is versioned and common across web/mobile/server producers, but Phase 1 does not persist every event into Compass PostgreSQL. Each event is routed to the system that needs it:

Destination	Event subset	Purpose
Typesense native analytics	Search auto-capture plus verified `click`, `conversion`, and optional `visit`	Popular/no-hit queries, suggestions, bounded document counters, optional behavior logs.
PostHog	Search exposure, UI interaction, cart, order, refund/cancellation, feature-flag exposure	Product funnels, cross-surface revenue, cohorts, and experiments.
Compass PostgreSQL	Explain record, rule/profile/configuration application, provider comparison, index sync state	Operational diagnosis, safety, auditing, and control-plane evidence.
Vector/Medusa/Orderbox	Cart/order line attribution and commerce state	Authoritative commercial source of truth.

The portable envelope below is the internal contract from which PostHog and Typesense-specific payloads are constructed:

{
  "event_id": "uuid",
  "event_name": "search_executed",
  "schema_version": 1,
  "occurred_at": "2026-08-03T10:15:20.123Z",
  "anonymous_id": "hashed-session-or-device-id",
  "user_id": "optional-known-user-id",
  "session_id": "uuid",
  "platform": "web",
  "request_id": "uuid",
  "discovery_context": {
    "search_id": "uuid",
    "surface": "search_results",
    "ranking_profile_version": "search-default:3",
    "config_version": "2026-08-03T10:15:00Z"
  },
  "properties": {}
}

Key internal event properties:

Event	Required properties
`search_executed`	raw/normalized query, interpretation, filters, sort, result count, fallback, returned product IDs and positions, latency
`search_result_clicked`	`search_id`, product ID, position, selected variant, click target
`suggestion_shown`	prefix, suggestion ID, type, rank
`suggestion_clicked`	suggestion ID, type, rank, destination
`plp_viewed`	page ID, page type, product IDs and positions, active filters/sort
`filter_changed`	page/search ID, filter, value, action, resulting count
`widget_impression`	widget ID, strategy/version, anchor context, products and slots, fallback source
`add_to_cart`	product ID, variant ID, originating search/page/widget context
`purchase_completed`	order ID, line items, all valid discovery attribution contexts

Destination routing rules:

Search: Compass emits search_executed to PostHog and keeps its result/position/rule explain evidence. TypeSense auto-captures the actual engine search for popular/no-hit analytics; shadow/test requests use enable_analytics=false.
Client interaction: web/mobile sends click, suggestion, filter, sort, and visible-impression events to the first-party endpoint. It validates the signed discovery context, forwards the normalized event to PostHog, and forwards only verified click/visit events needed by TypeSense rules.
Cart/order: Vector/Medusa persists discovery attribution on the line item and emits server-side PostHog events. A worker may forward verified TypeSense conversion events for bounded counters only after cart/order validation.
Operational evidence: Compass stores short-lived explain/provider-comparison records, not raw clickstream, in PostgreSQL.

The server emits authoritative exposure/result and commerce events; clients emit UI interaction events. Validators detect duplicates/missing data, and PostHog business outcomes reconcile to Medusa/Orderbox before they are trusted for decision-making.

9. Discovery API

9.1 External endpoints

POST /v1/discovery/search
POST /v1/discovery/suggest
GET  /v1/discovery/pages/{pageSlug}
POST /v1/discovery/widgets/{widgetKey}

POST /v1/admin/search/preview
POST /v1/admin/query-inspector
POST /v1/admin/page-inspector

The client never calls Typesense directly. This prevents rule bypasses and gives one place to apply eligibility, attribution, rollout flags, and response shaping.

9.2 Search request

{
  "query": "vit c serum for oily skin",
  "surface": "search_results",
  "platform": "web",
  "filters": {
    "brandIds": ["brand_1"],
    "formats": ["serum"],
    "price": { "lte": 700 }
  },
  "sort": "featured",
  "page": 1,
  "pageSize": 24,
  "sessionId": "uuid",
  "userId": "optional",
  "location": { "pincode": "optional-later" }
}

9.3 Search response

{
  "searchId": "uuid",
  "query": {
    "raw": "vit c serum for oily skin",
    "normalized": "vitamin c serum for oily skin",
    "intent": "combination",
    "rewrites": [{ "from": "vit c", "to": "vitamin c", "reason": "approved_alias" }],
    "extracted": {
      "ingredients": ["vitamin_c"],
      "skinHairTypes": ["oily"],
      "productTypes": ["serum"]
    }
  },
  "results": [
    {
      "productId": "product_123",
      "displayVariantId": "variant_123",
      "title": "Vitamin C Brightening Serum",
      "brand": "Example Brand",
      "price": 599,
      "inStock": true,
      "badges": ["bestseller"],
      "position": 1
    }
  ],
  "facets": [{ "key": "formats", "values": [{ "value": "serum", "count": 42 }] }],
  "pagination": { "page": 1, "pageSize": 24, "total": 42 },
  "fallback": null,
  "metadata": {
    "rankingProfile": "search-default:3",
    "latencyMs": 75
  }
}

Do not expose score internals in the public API. Retain them in a server-side explain record keyed by searchId, with access controlled for operations and support.

10A. End-to-End Discovery Flows

Every discovery request uses the same boundary: Compass interprets and governs the request; TypeSense performs primary candidate retrieval, filters, grouping, facets, and ranking; Compass performs final defensive safety/external eligibility checks, response shaping, and attribution.

Web / mobile
  -> Compass: query understanding, policy, profile and rule selection
  -> TypeSense: retrieval, filtering, grouping, ranking, native curation
  -> Compass: final validation, context token, explanation and event evidence
  -> Web / mobile

TypeSense executes the real-time ranking. Compass does not implement a general product-ranking engine in application code.

A.1 Exact product and product-family search

Examples: Acne UV Gel, Cetaphil Gentle Cleanser, Minimalist 10% Niacinamide.

Compass normalizes whitespace/case and recognizes protected product-family terms.
Compass classifies high-confidence product intent and selects an exact-product lexical profile.
Compass sends TypeSense title/product-family/brand-weighted query, eligibility filters, grouping, and matching pins/hides.
TypeSense performs typo-tolerant retrieval and ranks canonical product families.
Compass validates eligibility, selects the display variant, records result positions, and returns signed result tokens.

Vector/hybrid retrieval is disabled or heavily downweighted for this flow. An exact product/brand match must not lose to a semantically similar product.

A.2 Brand search

Examples: Cetaphil, Minimalist, Sebamed.

Compass matches the normalized query against the governed brand dictionary.
A configured Compass redirect may route directly to the brand PLP.
Otherwise Compass selects the brand profile and queries TypeSense with a high brand weight or exact brand filter.
TypeSense returns grouped canonical products and brand-relevant facets.
Compass applies active rules, eligibility, and response headers/brand entry points.

Redirects remain Compass policy; TypeSense is not required when direct navigation is the intended result.

A.3 Category, ingredient, and concern search

Examples: face wash, sunscreen, niacinamide, retinol serum, pigmentation, hair fall, dark spots.

Compass maps known terms and aliases to canonical category, ingredient, concern, and product-type IDs.
It selects the corresponding TypeSense ranking profile and facet configuration.
TypeSense retrieves/ranks against product title, category, concern, ingredient, and product-type fields; it computes facets and filter counts.
Compass applies claim/Rx policy, merchandising, page entry points, and final result shaping.

For example, retinol serum resolves to the canonical retinol ingredient plus serum product type before TypeSense retrieves candidates. This is more reliable than text matching raw descriptions alone.

A.4 Hinglish and mixed-language search

Examples: muhase ki cream, balo ka jhadna, jhaiyan, oily skin ke liye sunscreen. Hinglish starts as a governed Compass alias dictionary, not as a vector-only feature:

1. Compass normalizes the mixed-language query and recognizes protected terms.
2. It matches approved phrase/token aliases, for example:
   muhase -> acne
   balo ka jhadna -> hair_fall
   jhaiyan -> pigmentation
3. Compass extracts multiple structured signals where present:
   oily skin ke liye sunscreen -> sunscreen + oily skin type.
4. Compass builds a normal structured TypeSense lexical request with filters/preferences.
5. TypeSense retrieves, filters, facets, groups, and ranks products.

Every approved alias is explainable, testable, versioned, and reviewable from the control plane. Multilingual semantic retrieval can augment low-confidence cases later, but does not replace the governed dictionary.

A.5 Typo-tolerant and attribute-rich search

Examples: sunscreem spf 50, niacinemad, spf 50 sunscreen for oily skin under 700, 10% niacinamide serum.

1. Compass recognizes protected terms and approved aliases before typo behavior.
2. Compass extracts high-confidence attributes:
   SPF 50 -> spf >= 50
   under 700 -> price <= 700
   oily skin -> skin type = oily
   10% niacinamide -> ingredient/strength.
3. Explicit high-confidence values become TypeSense filters; ambiguous language becomes a ranking preference rather than a hard filter.
4. TypeSense performs native typo-tolerant matching, field-weighted ranking, numeric/Boolean filtering, grouping, and facets.
5. Compass decides whether to display a correction, records the interpretation, and enters the fallback ladder only if quality remains poor.

TypeSense performs typo matching. Compass decides protected-term policy, structured extraction, display/rewrite behavior, and the query plan.

A.6 Autosuggest

Examples: typed prefixes such as sun, acne u, and naci.

Client debounces for 150-250 ms and cancels stale prefix requests.
Compass applies the same normalization, protected terms, aliases, and safety policy as full search.
Compass queries TypeSense for grouped product, brand, category, concern, ingredient, and approved popular-query candidates.
Compass applies pinned/blocked/curated suggestion rules and quality filters.
Client renders destination-aware groups; visible impressions and taps are attributed.

TypeSense popular_queries supplies candidates only. Compass validates quality and safety before a suggestion reaches users.

A.7 Browse and PLP flow

Examples: category, brand, concern, ingredient, and campaign pages.

Compass/Tetris resolves the page definition: inclusion criteria, profile, allowed sorts, facets, quick filters, content slots, and page rules.
Compass builds a TypeSense request with canonical inclusion and eligibility filters; q may be * for pure browse.
TypeSense retrieves, filters, ranks, groups, paginates, and returns facet counts.
Compass/Tetris adds CMS content, badges, consult modules, and signed discovery context.
Client records page exposure and interactions.

A.8 Semantic and hybrid search

Examples: cream for dark spots, something for oily acne-prone skin, moisturizer that does not feel sticky. Semantic retrieval is introduced only after lexical search, controlled Hinglish aliases, relevance fixtures, and baseline metrics are dependable.

Compass runs deterministic query understanding first.
It selects a hybrid profile only for natural-language/weak-lexical query groups.
Compass obtains a query embedding and TypeSense runs lexical plus vector retrieval/ranking.
TypeSense combines text relevance and vector similarity while preserving filters/grouping.
Compass applies rules/safety/exact-match protection and records hybrid usage.

Exact product, brand, ingredient, and approved Hinglish-alias searches remain lexical dominant. Prescription/medical queries never use semantic retrieval to create a promotional shelf.

A.9 Redirect, medical, and fallback flow

Compass checks active redirect rules before product retrieval.
It detects medical/Rx-sensitive or deliberately out-of-scope intent before generic fallback.
For a normal search, it runs the primary TypeSense query plan.
If results are empty or poor: typo-safe correction -> relax low-confidence modifiers -> hybrid fallback when enabled -> closest honest category/concern response.
For uncarried medical demand or prescription molecules, Compass routes to consultation/clinical policy rather than unrelated merchandising.
Compass records every fallback/redirect/consultation decision in the explain record and PostHog event.

A.10 Recommendation retrieval flow

Recommendations are not text search, but use the same ownership boundary:

Compass receives anchor product, widget type, surface, and allowed session/profile context.
It loads the widget strategy and candidate source: curated, catalogue relationships, co-purchase, model later.
TypeSense retrieves/ranks eligible product candidates by controlled attributes where applicable.
Compass applies routine-step, ingredient-conflict, Rx, stock, anchor-product, and widget-specific exclusions.
Compass applies widget rules/pins/fallback and Tetris renders the widget.

Generic semantic similarity is never the only recommendation rule. Routine compatibility and safety remain deterministic.

A.11 Unified decision tree

Incoming discovery request
  -> configured redirect?             -> Compass destination
  -> medical/Rx-sensitive intent?     -> Compass clinical/safety route
  -> exact product or brand?          -> lexical-dominant TypeSense profile
  -> known category/ingredient/concern? -> structured lexical TypeSense profile
  -> approved Hinglish alias?         -> Compass mapping + structured lexical profile
  -> explicit attributes?             -> Compass extraction + TypeSense filters/preferences
  -> natural-language/weak lexical?   -> hybrid TypeSense profile when enabled
  -> still poor?                      -> Compass fallback ladder / honest alternative

A.12 Responsibility summary

TypeSense executes	Compass owns
Text and typo-tolerant retrieval	Query interpretation and protected terms
Field weighting, filtering, facets, grouping, sorting	Intent, aliases/Hinglish governance, and structured extraction
Lexical/vector hybrid ranking	Ranking-profile/rule selection and policy
Native pins/hides/overrides	Scheduling, approval, audit, preview, and rollback
Native search analytics aggregation	Safety, Rx/consult routing, external eligibility, attribution, and explainability

10. Search Low-Level Design

10.1 Query understanding pipeline

raw query
  -> Unicode/whitespace/case normalization
  -> protected-term recognition
  -> approved synonym and alias normalization
  -> typo candidate generation with protected-term guard
  -> intent classification
  -> structured detail extraction
  -> query plan

Implement the initial pipeline deterministically using the attribute dictionary and approved rules. For example:

vit c maps to the approved vitamin c alias.
SPF 50 extracts an spf >= 50 candidate filter/preference.
under 500 extracts price.maximum = 500.
for oily skin extracts an approved skin-type signal.
Protected terms are identified before typo correction and never auto-rewritten into generic words.

Intent classification should begin with transparent signals rather than a black-box model:

Intent	Initial evidence
Product	Exact/near-exact title or product-family match
Brand	Brand dictionary or protected-term match
Category	Category/alias dictionary match
Concern	Concern dictionary and approved aliases
Ingredient	Ingredient dictionary and approved aliases
Combination	Multiple recognized entity types or extracted attributes

Log confidence and evidence. Ambiguous queries may retrieve multiple intent routes; do not force false certainty.

10.2 Candidate retrieval

Typesense collection fields should include separate searchable fields and facet fields. A representative collection configuration is:

{
  "name": "discovery_products_v1",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "title", "type": "string" },
    { "name": "brandName", "type": "string", "facet": true },
    { "name": "categoryNames", "type": "string[]", "facet": true },
    { "name": "concernNames", "type": "string[]", "facet": true },
    { "name": "ingredientNames", "type": "string[]", "facet": true },
    { "name": "productType", "type": "string", "facet": true },
    { "name": "formats", "type": "string[]", "facet": true },
    { "name": "skinHairTypes", "type": "string[]", "facet": true },
    { "name": "spf", "type": "int32", "facet": true, "optional": true },
    { "name": "searchableDescription", "type": "string" },
    { "name": "listable", "type": "bool", "facet": true },
    { "name": "purchasable", "type": "bool", "facet": true },
    { "name": "inStock", "type": "bool", "facet": true },
    { "name": "rxClassification", "type": "string", "facet": true },
    { "name": "priceMinimum", "type": "float", "facet": true },
    { "name": "rating", "type": "float", "optional": true },
    { "name": "launchedAtTimestamp", "type": "int64", "optional": true }
  ],
  "default_sorting_field": "popularityScore"
}

The exact Typesense schema should be prototyped against real data. The key point is to separate:

Full-text fields: title, brand, categories, concerns, ingredients, product type, description.
Facet/filter fields: stock, listability, pricing, formats, SPF, safety, brands, categories.
Sort fields: price, rating, newness, approved popularity scores.
Stable identity fields: product family ID, source version, canonical taxonomy IDs.

10.3 Eligibility policy

Eligibility runs before final ranking and must be used by search, browse, recommendations, and future sponsorship.

function isEligible(product: DiscoveryProduct, context: DiscoveryContext): EligibilityDecision {
  if (!product.listable) return { eligible: false, reason: "not_listable" };
  if (!product.purchasable) return { eligible: false, reason: "not_purchasable" };
  if (context.widget && product.rxClassification !== "none") {
    return { eligible: false, reason: "rx_excluded_from_widget" };
  }
  if (context.promotion && product.rxClassification !== "none") {
    return { eligible: false, reason: "rx_excluded_from_promotion" };
  }
  if (context.excludeOutOfStock && !product.inStock) {
    return { eligible: false, reason: "out_of_stock" };
  }
  return { eligible: true };
}

For efficiency, the broadest eligibility filters are pushed to Typesense filters. The application policy runs again over selected candidates as defense in depth and supplies clear exclusion reasons to the inspector.

10.4 Ranking pipeline

The exact layer order is a platform invariant:

Eligibility and safety
Relevance
Behavioral quality (initially disabled or tightly bounded)
Declared business boosts
Merchandising boosts and buries
Pins

Initial score design:

relevanceScore =
  1.00 * titleExactOrNearExact
  0.85 * brandMatch
  0.70 * categoryMatch
  0.70 * concernMatch
  0.70 * ingredientMatch
  0.55 * productTypeMatch
  0.20 * descriptionMatch
  0.30 * extractedAttributeFit

finalOrganicScore = relevanceScore
  + declaredBusinessBoosts
  + boundedBehavioralScore
  + merchandisingBoostOrBury

These numbers are placeholders, not production truth. Store them in a versioned RankingProfile, evaluate them against golden queries, and validate material changes through interleaving or A/B testing. Pins are applied after organic scoring. A pin must be validated before publishing:

Product exists and is currently eligible for the target scope.
Product does not conflict with an existing pin at the same position.
Safety restrictions are never overridden.
Expiry date and audit reason are present where required.

10.5 Fallback ladder

For zero or materially poor results:

Run the protected-term-safe typo correction query.
Run a relaxed query plan by dropping low-confidence modifiers.
Run a hybrid/semantic candidate source only when introduced.
Route to the nearest relevant category/concern page with an honest message.
Return an explicit zero-result response plus supported alternative actions.

Each attempt is recorded in the search event. Never silently substitute an unrelated shelf.

10.6 Hinglish and multilingual support

Start with a governed alias dictionary sourced from real query traffic. Examples include mapped concern terms such as muhase, balo ka jhadna, and jhaiyan. Store language/locale and confidence on aliases. Do not add broad transliteration or LLM rewriting until query-health data shows the coverage need and quality review capacity exists.

11. Autosuggest Design

Autosuggest reuses query normalization, protected terms, candidate filtering, and safety policy from search. It must not be less tolerant than the full search endpoint.

11.1 Suggest endpoint behavior

For each prefix, return grouped suggestions:

Products: product image, title, brand, price, destination PDP
Brands: brand title, destination brand PLP
Categories: category title, destination category PLP
Concerns: concern title, destination concern PLP
Ingredients: ingredient title, destination ingredient PLP
Popular searches: approved query, destination search/PLP

The response includes suggestionId, type, rank, source, and destination so impressions and taps are attributable.

11.2 Suggestion quality controls

Reject candidates that lead to no/poor results based on an offline quality job.
Deduplicate near-identical strings.
Preserve product family and brand intent when confidence is high.
Exclude Rx items from promotion-like suggestion positions; allow direct matching where policy permits.
Apply merchandising pins/blocks/curated suggestions only after safety and quality checks.
Before typing, surface privacy-safe recent searches, approved trending queries, and configured seasonal shortcuts.

12. Browse and PLP Design

Browse pages are query templates over the same product index, not separate product databases.

12.1 Page definition

type PageDefinition = {
  id: string;
  slug: string;
  pageType: "category" | "brand" | "concern" | "ingredient" | "campaign" | "combination";
  title: string;
  description?: string;
  inclusionQuery: StructuredProductFilter;
  rankingProfileId: string;
  filterConfigurationId: string;
  allowedSorts: string[];
  contentSlots: ContentSlotDefinition[];
  seo: { canonicalUrl: string; indexPolicy: "index" | "noindex"; redirectsFrom: string[] };
};

This supports standard categories, brand shelves, cross-category concern/ingredient pages, and campaigns without custom code per page.

12.2 Filters and sort

FilterConfiguration is data associated with page type/page/category. It controls:

Visible facets and their display order.
Pinned facet values and quick filters.
Search-inside-filter support for high-cardinality fields such as brands.
Sort options and default ranking profile.
Maximum allowed selections and zero-result recovery behavior.

Multiple values in a facet use OR; filters across facets use AND. Applied filters are returned as removable chips. A sort/filter change resets pagination to page 1 server-side.

12.3 SEO requirements

Use a stable page ID and stable canonical URL independent of display-name changes.
Store redirects when products, brands, or pages retire.
Define canonical/index/noindex policy for filter combinations before releasing filtered pages to crawlers.
Capture organic traffic and indexability metrics before and after migration.
Do not change URL structures as an incidental side effect of changing the search provider.

13. Recommendations Design

Recommendations start in Phase 3 because they depend on reliable catalogue relationships and attribution. They do not require a separate model-serving platform initially.

Widget request
  -> load strategy by widget + surface
  -> gather candidates (curated, relationship data, co-purchase, model later)
  -> universal eligibility/Rx filters
  -> widget-specific hard filters
  -> routine safety and complementarity checks
  -> ranking and business rules
  -> merchandise pins/hides/replace rules
  -> fallback chain
  -> emit impression context

13.2 Candidate sources in delivery order

Curated lists and catalogue relationships.
Product similarity based on controlled attributes.
Co-purchase signals after event/order attribution is validated.
Bounded model output where data coverage justifies it.

Never use a language model as the primary candidate generator. It may enrich sparse metadata under human review, but product safety and routine rules remain deterministic.

13.3 Routine intelligence

Create versioned relationship/rule tables:

product_relationships
  anchor_product_id, related_product_id, relationship_type, confidence, source

routine_step_rules
  anchor_step, allowed_complement_steps, prohibited_same_steps

ingredient_conflicts
  ingredient_a, ingredient_b, severity, rationale, applies_to

safety_cautions
  attribute_or_ingredient, caution_type, policy

Widget hard rules must ensure:

Anchor product is excluded.
Out-of-stock and Rx products are excluded where required.
Same-step/duplicate active-ingredient suggestions are excluded for routine widgets.
Known ingredient conflicts and declared cautions are respected.
Category mismatch constraints are enforced per widget.
Short, high-quality lists are preferable to padded weak lists.

13.4 Recommendation quality gate

Maintain reviewed fixture sets with approved and prohibited recommendations per anchor product. Every widget strategy change runs against them before rollout. Online click/revenue metrics complement, but do not replace, this review.

14. Merchandising Control Plane

14.1 Initial operator workflows

The initial console should focus on the highest-value, lowest-risk capabilities:

Search query inspection.
Protected-term and synonym management.
Query redirects.
Search pins, hides, boosts, and buries.
Ranking-profile assignment.
Preview and publish/revert.
Rule audit history.

The UI should lead with the affected query/page preview, not a generic JSON rule builder. Power-user bulk tools, campaign bundles, and cross-surface workflows come later.

14.2 Publishing workflow

Draft -> validate -> preview using live/safe snapshot -> approval if required
  -> publish version -> invalidate config cache -> active within minutes
  -> monitor impact -> expire or revert

Validation checks include syntax, target existence, eligibility, overlapping pin conflicts, schedule conflicts, Rx/promotion restrictions, and required audit reason.

14.3 Configuration caching

Store authoritative configurations in PostgreSQL.
Publish an immutable configuration version to Redis/cache on activation.
Discovery API reads the active version once per request or with a short TTL.
Attach config_version to every response/event.
Cache invalidation must be explicit and observable. A rule that saves but does not take effect is a production incident.

14.4 Compass source of truth and Typesense compilation

The merchandising panel writes only to Compass admin APIs and Compass PostgreSQL. It does not treat Typesense as the source of truth. Compass owns rule definition, scheduling, audit history, approval, preview, rollback, and conflict validation; Typesense is an optimized execution target for rules it can apply natively.

Merchandising panel
  -> Compass admin API
  -> PostgreSQL draft/audit/schedule/approval state
  -> preview against current Typesense data
  -> publish immutable Compass configuration version
  -> sync compatible rule to Typesense OR apply it as request-time parameters
  -> Compass validates final result eligibility and returns response

The panel displays TypeSense synchronization state for every compiled rule. A synced state is necessary for a native override to execute, but Compass remains the emergency authority: disabling a rule stops Compass from applying request-time behavior immediately and queues removal of the Typesense resource.

Rule/action	Source of truth	Serving execution
Exact query pin	Compass rule	TypeSense override or request-time `pinned_hits`.
Exact query hide	Compass rule	TypeSense override or request-time `hidden_hits`.
Approved synonym	Compass synonym set	Synchronize approved synonym to TypeSense.
Protected term	Compass	Compass query policy before correction/retrieval.
Redirect	Compass redirect rule	Compass returns destination without product retrieval.
Simple query/page boost or bury	Compass rule	Compass builds TypeSense ranking/evaluation parameters.
Complex segmented boost/bury	Compass rule	Compass over-fetches eligible candidates and performs a bounded final rerank.
Rx/listing/availability restriction	Compass eligibility policy	Push broad filters into TypeSense and validate again in Compass.
Schedule, campaign, approval, preview, audit	Compass	Never delegated to TypeSense.

Do not persist a campaign boost as a permanent product popularity/document value. It becomes stale, is hard to reverse, and can leak beyond its intended scope. Pins/hides map cleanly to native TypeSense controls; boosts/buries are query-context policy and must remain explainable in Compass.

14.5 Rule publication, synchronization, and runtime behavior

Publishing a rule follows this sequence:

Merchandiser creates a draft in the panel.
Compass validates target existence, eligibility, dates, conflicts, permissions, and Rx/promotion restrictions.
Compass previews current results with the proposed rule.
Required approver accepts the draft.
Compass activates an immutable configuration version and invalidates the active configuration cache.
A sync worker creates/updates the corresponding TypeSense synonym or override when supported.
The worker records TypeSense resource ID, rule version, status, and failures in typesense_rule_sync.
Compass records applied rule IDs/configuration version in search events and explain records.

For a simple pin, the resulting request flow is:

User searches "sunscreen"
  -> Compass loads matching active rules from its cached configuration version
  -> Compass applies safety/listability filters and sends native pin/override to TypeSense
  -> TypeSense retrieves/ranks and places the eligible pinned product
  -> Compass validates final eligibility, writes explain record, and returns response

For complex boosts/buries, Compass retrieves more candidates than the visible page size, applies only bounded adjustments to eligible/relevant candidates, and returns the visible page. This final rerank is deliberately narrow; Compass must not become a slow general-purpose search engine. Rule precedence remains fixed:

Eligibility and safety
Organic relevance and bounded behavioral/business signals
Merchandising boosts/buries
Pins
Final Compass safety validation

No TypeSense override, query parameter, panel action, experiment, or campaign can resurrect an item that fails eligibility. Publish validation rejects conflicting pins for the same scope/position, invalid targets, overlapping ambiguous rules, and unauthorized global or Rx-sensitive changes. The inspector shows both policy and engine evidence: Compass rule ID/version, TypeSense override/resource ID when used, score adjustment, selected position, excluded products and reasons, and active configuration/ranking profile versions.

15. Ranking Governance and Explainability

15.1 Ranking profile example

{
  "name": "search-default",
  "version": 3,
  "availability": { "outOfStock": "demote" },
  "textWeights": {
    "title": 10,
    "brand": 9,
    "category": 7,
    "concern": 7,
    "ingredient": 7,
    "productType": 6,
    "description": 2
  },
  "businessSignals": {
    "rating": { "enabled": true, "weight": 0.10 },
    "newness": { "enabled": true, "weight": 0.05 },
    "stockDepth": { "enabled": false, "weight": 0 }
  },
  "behavioralSignals": { "enabled": false, "maxInfluence": 0.10 },
  "allowedSorts": ["featured", "price_asc", "price_desc", "newest", "rating"]
}

Profiles are versioned, previewed against golden cases, and staged. Product owns assignments; creating/changing scoring logic requires engineering review and experiments for material changes.

15.2 Explain record

For every searchId, persist a short-lived structured record:

{
  "searchId": "uuid",
  "query": { "raw": "sunscreem", "normalized": "sunscreen", "intent": "category" },
  "queryPlan": { "filters": ["listable:true"], "fallbackStep": "typo_correction" },
  "rules": ["rule_123"],
  "results": [
    {
      "productId": "product_1",
      "position": 1,
      "eligibility": "eligible",
      "score": {
        "relevance": 0.93,
        "business": 0.02,
        "behavioral": 0,
        "merchandising": 0.1
      },
      "pin": null,
      "reasons": ["title_match", "concern_match", "merchandise_boost"]
    }
  ]
}

This powers the query inspector, support escalation, ranking review, and incident diagnosis. Its retention period should balance operational value and privacy; do not expose raw personal queries widely.

16. Catalogue Ingestion and Quality

16.1 Delta ingestion

Catalogue change
  -> SNS event -> SQS queue
  -> SQS queue
  -> indexer validates event/version
  -> fetch or receive canonical product payload
  -> transform into DiscoveryProduct
  -> validate attribute dictionary and safety fields
  -> upsert/delete Typesense document
  -> record index status and latency

Use an idempotency key composed of product ID and source version. Out-of-order events must not overwrite newer product state. Put invalid payloads into a dead-letter queue with clear error reason and owner routing.

16.2 Freshness objectives

Data type	Target propagation	Notes
Stock and listability	Near real time, target under 5 minutes	Important for user trust and eligibility.
Price	Near real time, target under 5 minutes	Search/PLP price must not conflict with PDP.
Core product content	Minutes to hours	Title, images, descriptions, attributes.
Full rebuild	Recovery only	Never the normal update mechanism.

16.3 Quality gates

Define coverage requirements by category. Example:

Sunscreen: SPF 100%, format >= 95%, skin type >= 90%
Serum: hero ingredient >= 95%, concern >= 90%, format >= 95%
Haircare: product type >= 100%, hair concern >= 90%

New or edited products that fail required coverage are flagged for catalogue review. Whether they are blocked from a particular facet or from all discovery surfaces should be policy-driven and visible.

16.4 Assisted enrichment

Enrichment proposals can be generated from names, descriptions, and ingredient lists, but they remain proposed until a catalogue operator approves them. Record source, confidence, reviewer, and timestamp. Never silently alter controlled attributes.

17. Analytics, Reporting, and Experiments

17.1 Phase 1 analytics architecture

Phase 1 does not require a new data warehouse. Use Typesense’s native analytics for search-engine feedback, PostHog for product and revenue analytics, Compass PostgreSQL for operational/search explainability, and Vector/Medusa/Orderbox for authoritative commerce facts.

Compass search -> Typesense native analytics
  -> popular queries, no-hit queries, verified document counters, optional logs

Compass/Tetris/Vector/Medusa server events -> PostHog server events
Web/mobile interaction SDK -> first-party event ingestion -> PostHog
  -> funnels, cross-surface paths, revenue, cohorts, feature flags and experiments

Compass PostgreSQL
  -> result positions, ranking/profile/rule versions, fallback, audit and explain records

Vector/Medusa/Orderbox
  -> authoritative cart, completed-order, refund/cancellation facts and line-item attribution

PostHog is sufficient for Phase 1 dashboards when server-side order events carry persisted discovery attribution. Typesense and PostHog must not be treated as the commerce system of record; reconciliation remains against Medusa/Orderbox.

17.2 Typesense native analytics

Enable Typesense analytics only on the self-hosted production/search cluster and configure four rule families:

Rule family	Typesense rule type	Phase 1 use	Guardrails
Successful query demand	`popular_queries`	Popular searches and query-suggestion candidates	Separate actual search submissions from typeahead; Typesense aggregates after a four-second pause.
Raw engine no-hits	`nohits_queries`	Candidate synonym, catalogue-gap, redirect, and consultation-routing review	This measures engine-level no hits, not whether Compass’s fallback ladder recovered the user.
Verified engagement	`counter` with `click`/`conversion` events	Small, controlled product popularity signal and fast relevance triage	Forward only validated events; keep disabled as a major ranking signal initially.
Behavioral history	`log`	Future pseudonymous session/user personalization and recent-behavior inspection	Access controlled, retention reviewed, never the finance/reporting store.

Typesense analytics supports search, click, conversion, and visit events. Each Compass request uses analytics_tag to distinguish the surface/platform, for example search_results:web, search_results:android, or concern_plp:web. Send a pseudonymous X-TYPESENSE-USER-ID derived from approved Sesh/Hodor identity policy rather than a raw customer identifier. Use filter_by and analytics_tag metadata only for lightweight search segmentation. Keep ranking profile/version, rule IDs, experiment assignment, fallback outcome, exact positions, and safety decisions in Compass because Typesense does not model the full discovery policy. Counter events are forwarded only after first-party token validation or authoritative cart/order handling. A suggested initial weighting is click +1, add-to-cart +3, purchase +10, but the resulting value is only a bounded tie-breaker within eligible/relevant results. It must not override relevance, safety, pins, or merchandise rules. Position bias, fraud, stale stock, and popularity feedback loops require evidence thresholds, rate caps, decay, holdout evaluation, and a kill switch before behavioral ranking is enabled.

17.3 PostHog analytics and attribution

PostHog is the Phase 1 cross-surface analytics layer. It receives validated client interaction events and server-side commerce events, and answers:

Search-to-PDP, search-to-cart, and search-to-order funnels.
Revenue by query, intent, platform, suggestion type, ranking profile, rule/campaign, fallback path, and experiment variant.
Cross-session conversion after Sesh/Hodor identity linking, under the approved privacy policy.
Comparison of search, PLP, widget, and direct PDP journeys.
Filter/sort/suggestion behavior, cohorts, feature flags, and experiments.

order_completed is emitted server-side by Vector, Medusa, or Orderbox and includes line-item revenue plus the discovery attribution persisted at cart-add time. This makes PostHog suitable for Phase 1 search-to-order revenue dashboards; browser-reported purchases are never authoritative.

17.4 Future warehouse plan

Introduce a data warehouse only after a concrete need exceeds PostHog, Typesense, Compass, and commerce reporting. Triggers include:

Finance-grade reconciliation of revenue, refunds, taxes, and order lines directly against commerce data.
Large/custom SQL joins across discovery, catalogue, inventory, CRM, marketing, and orders.
Long-term immutable raw-event retention or lower-cost high-volume analytical storage.
Offline training for position-debiased behavioral ranking/learning-to-rank.
Statistical analysis or experiment methods beyond PostHog.
PostHog query performance, retention, or cost becoming a product constraint.

When triggered, use S3 raw events, Athena or the existing company warehouse, and dbt/SQL transformations. The initial future models are:

fact_discovery_requests
fact_search_results
fact_discovery_interactions
fact_discovery_attribution
fact_merchandising_rule_exposure
fact_widget_exposure
dim_query_interpretation
dim_ranking_profile_version
dim_configuration_version

17.5 Future owned analytics stack

An owned analytics stack is a future option, not a replacement for TypeSense analytics or PostHog at launch. Build it only when the triggers above are accepted and an owner is assigned. The target responsibilities are:

First-party ingestion / server outbox
  -> durable raw immutable event storage
  -> schema validation and deduplication
  -> curated discovery and commerce models
  -> SQL/BI, finance reconciliation, offline experimentation, and model training

Recommended staged implementation:

Stage	Add	Outcome
1. Dual-write raw events	First-party endpoint and server outboxes additionally write partitioned raw events to S3 through SNS/SQS workers	Vendor-independent recovery/replay without changing PostHog dashboards.
2. Curated models	Glue/Athena or the company-standard warehouse plus dbt/SQL transformations	Reconciled search, interaction, cart, order, refund, rule, and experiment models.
3. BI and governance	Versioned metrics definitions, access controls, retention/deletion jobs, data-quality monitors, and BI dashboards	Product/finance reports are reproducible outside PostHog.
4. Learning data	Position-aware exposure/click training sets, debiasing, holdouts, and offline evaluation datasets	Safe behavioral ranking and future learning-to-rank development.

The owned stack remains downstream and additive during migration. Continue sending PostHog events until its dashboards and experiments have validated against the owned models. Do not make TypeSense analytics the raw event source: it is optimized for search-native aggregation and limited log retrieval, not immutable organization-wide event retention. Required future raw-event fields include the portable envelope, destination delivery status, producer/service version, schema version, deduplication key, event and receipt timestamps, discovery IDs/positions, configuration/profile/rule versions, and persisted cart/order line attribution. Raw sensitive query/health data requires separate access controls and retention/deletion policy.

17.6 Event ownership and source of truth

Use a hybrid event model. The browser/mobile app reports interaction only visible in the interface; backend services report the discovery response and business facts they authoritatively know. This avoids trusting clients with result positions, ranking versions, cart state, or completed orders.

Event class	Primary producer	Reason
Search executed, returned results/positions, query interpretation, ranking/rule/fallback details	Compass	Compass generated the exact result list and response.
PLP/widget payload served, strategy, fallback, slot positions	Compass/Tetris	The server assembled the content and knows its version.
Result/suggestion click, filter/sort change, visible-card/widget impression	Web/mobile SDK	Only the client knows what the user saw or interacted with.
Cart mutation and line-item attribution	Vector/Medusa	Cart state is a server-side commerce fact.
Purchase/order completion	Medusa/Vector/Orderbox	The order system is authoritative.

Analytics delivery is always asynchronous. No search, click navigation, cart, or checkout response waits for PostHog, S3, SNS, or a warehouse job.

17.7 Identity and discovery context

Each event contains the identities needed for reliable joining:

Identifier	Owner	Purpose
`anonymous_id`	Sesh/client	Stable browser or app-installation identity before sign-in.
`session_id`	Sesh	Groups activity within one visit/session.
`user_id`	Hodor	Optional authenticated customer identity.
`request_id`	Compass	Correlates one service request across logs and services.
`search_id`	Compass	Identifies one executed search and exact returned list.
`page_view_id`	Compass/Tetris	Identifies one rendered PLP result set.
`widget_impression_id`	Compass/Tetris	Identifies a recommendation payload and strategy.
`event_id`	Producer	Makes retries and deduplication safe.

When a visitor signs in, preserve the anonymous activity and record an identity-link event. Do not rewrite historical records destructively. Compass returns an opaque signed discovery_context_token for every result card. The token binds the source, product, variant, position, session, issue time, and expiry. Clients pass it unchanged to click and add-to-cart flows; they never construct attribution fields themselves.

{
  "version": 1,
  "source_type": "search",
  "source_id": "srch_01...",
  "product_id": "prd_100",
  "variant_id": "var_100_default",
  "position": 1,
  "session_id": "ses_01...",
  "issued_at": "2026-08-03T10:00:00Z",
  "expires_at": "2026-08-10T10:00:00Z"
}

The token is signed and validated server-side. It prevents clients from falsely assigning a product to a more favorable query or position, which protects attribution and any future behavioral ranking.

17.8 Shared web and mobile SDKs

Create a shared, versioned contract package and platform wrappers rather than scattering direct analytics calls through clients:

discovery-event-contracts
  event names, JSON schemas/types, schema versions, redaction rules

discovery-web-analytics
  JavaScript/TypeScript wrapper, batching, sendBeacon/keepalive delivery

discovery-mobile-analytics
  native/mobile wrapper, encrypted local queue, retry/flush behavior

Client code calls typed methods such as trackSearchResultClick and trackSuggestionClick, not arbitrary event strings. Both clients use the same event names and required properties. Web behavior:

Send navigation-adjacent events with navigator.sendBeacon, falling back to fetch with keepalive: true.
Queue and batch lower-priority interaction events.
Flush on visibilitychange and page lifecycle transitions.
Debounce autosuggest requests by 150-250 ms and cancel stale requests.
Emit viewport impressions only after a card/widget meets the agreed visibility threshold, for example 50% visible for 500 ms.

Mobile behavior:

Persist an encrypted local event queue.
Batch on normal connectivity and flush on foreground/interval events.
Use idempotent event_id values so retries are safe.
Never delay navigation or rendering for analytics delivery.

17.9 First-party client event ingestion

Web and mobile clients submit interaction events to a first-party endpoint, initially owned by Compass or the existing analytics edge:

POST /v1/events/discovery

Web/mobile SDK
  -> first-party event endpoint
  -> schema/token validation, rate limiting, server timestamp enrichment
  -> PostHog delivery
  -> optional Typesense verified click/visit event

Do not use direct browser/app-to-PostHog delivery as the only event path. A first-party endpoint provides validation, consistent schema enforcement, sensitive-data redaction, resilience against ad blockers, and safe forwarding to both PostHog and Typesense. PostHog remains the Phase 1 analytics layer; commerce reconciliation and operational attribution remain owned by server systems.

17.10 End-to-end search-to-purchase lifecycle

Sesh creates/resumes anonymous_id and session_id; Hodor supplies user_id after sign-in.
Web/mobile requests search from Compass with session context.
Compass creates search_id, records the full server-side search exposure, and returns results plus signed result tokens.
Client renders results. It emits visible impressions under viewport rules and sends click/filter/sort events through the SDK.
A product click navigates immediately; SDK delivery runs independently and token validation resolves source and position.
Add-to-cart sends the selected product/variant and context token to Vector/Medusa.
Vector/Medusa validates the token, persists primary discovery attribution on the cart line, and emits an authoritative add-to-cart event.
On payment/order completion, Medusa/Vector/Orderbox emits the authoritative purchase event using persisted line-item attribution.
PostHog joins discovery exposure/interactions with server-side cart/order events for Phase 1 dashboards and experiments; future warehouse models are introduced only when triggered.

Compass must record the complete result list and positions server-side. Client visibility events improve CTR analysis, but cannot replace the server exposure record. For cart attribution, persist one primary source: the most recent valid discovery context directly preceding the cart add. Preserve an optional bounded list of prior valid in-session contexts as assists. Direct PDP/cart actions without a valid token remain explicitly unattributed; do not invent an origin.

17.11 Durable server event delivery

Server events use the existing SNS/SQS transport. Search events can be published asynchronously with metrics/alerts for enqueue failure. Cart and order events require stronger delivery semantics through a transactional outbox:

Commerce transaction
  -> persist cart/order mutation
  -> persist outbox row in the same database transaction
  -> background publisher sends the row to SNS/SQS
  -> mark the outbox row delivered after successful publish

This prevents losing attribution because an order commits while a separate event publication fails. Consumers deduplicate by event_id; event processing is at-least-once and idempotent.

17.12 Impression and event rules

Event	Producer	Rule
Search response/result-list exposure	Compass	Record every returned list and all positions.
Visible product-card impression	Client	Record only after the agreed viewport threshold; batch and deduplicate.
Widget payload served	Compass/Tetris	Record strategy, candidates, fallback, slots, and context.
Visible widget impression	Client	Record when the widget becomes visible.
Suggestion shown	Client	Record when the dropdown is visible, with prefix/type/position.

Avoid one network request per visible card. Batch impression events and deduplicate within a bounded time window using session_id + source_id + product_id + position.

17.13 Event envelope and schema evolution

Every event uses a versioned envelope:

{
  "event_id": "uuid",
  "event_name": "search_result_clicked",
  "schema_version": 1,
  "occurred_at": "client-or-server timestamp",
  "received_at": "server timestamp",
  "producer": "web_sdk | ios_sdk | android_sdk | compass | vector | medusa",
  "anonymous_id": "optional",
  "session_id": "required where applicable",
  "user_id": "optional",
  "request_id": "optional",
  "properties": {}
}

Rules:

Event names are immutable after production release.
Add optional properties without changing the schema version.
Breaking changes require a new event name or schema version.
Validate client events at ingestion and server events before publication.
Store both event occurrence and server-receipt timestamps; accept bounded client clock skew.
Deduplicate by event_id.
Redact or access-control personal health and prescription query data.

17.14 Metrics and data-quality controls

Track at least:

Zero-result rate and fallback rescue rate.
Search to PDP click rate, add-to-cart rate, and purchase conversion.
Reformulation and dead-end exit rates.
Suggestion acceptance and downstream quality by suggestion type.
PLP click-through, conversion, filter use, and empty-grid rate.
Widget impressions, clicks, add-to-cart, revenue, fallback rate, and session-level effect.
Rule/campaign impact compared with baseline/control where possible.
Attribute coverage, index freshness, latency, errors, and event completeness.

Additionally alert on:

Result-click events without a valid source ID and position.
Cart lines missing valid attribution when a discovery token was supplied.
Purchase-event reconciliation differences against commerce orders.
Event duplicates, invalid/expired tokens, client/server delivery lag, and queue/DLQ backlog.
Compass search exposure counts that diverge from request logs.
PostHog delivery failures and abnormal event-volume changes by client platform/version.

17.15 Experiment assignment

Experiment assignment should be deterministic using a stable hash of anonymous/user ID, experiment ID, and assignment version. Persist exposure with the assigned variant and configuration version. Avoid assignment changes mid-session. Every experiment has:

Hypothesis and owner.
Surface and inclusion criteria.
Primary metric and guardrails.
Minimum duration/sample guidance.
Variant configuration references.
Stop/rollback conditions.
Final decision and explanation.

Use interleaving for ranking-only comparisons when traffic is limited. Use A/B tests for broader UX/configuration changes. Never make a decision from widget click-through alone when total session conversion may be unchanged or harmed.

18. Safety, Privacy, and Security

18.1 Rx handling

Rx policy belongs in the shared eligibility layer:

Rx products carry visible badges on applicable surfaces.
They are excluded from recommendation widgets, promotional boosts, bestseller logic, discount badges, and sponsored placements.
Exact search/category/concern discovery remains available according to regulatory policy.
Prescription molecule searches route toward consult/prescription flows rather than promotional shelves.
Rule validation, tests, and runtime policy all enforce this independently.

18.2 Claim-safe content

Content slot definitions include a claimSensitive flag and review status. Concern pages must have approved copy and a designated consultation module where a product grid could overstep clinical/compliance boundaries.

18.3 Personal data

Use pseudonymous session/device IDs for anonymous discovery behavior.
Do not place raw personal health data in the search index or client logs.
Protect inspector access with role-based authorization and audit queries of sensitive information.
Sensitive searches and Rx products do not appear in shared/recent-search surfaces without an explicit privacy policy.
Profiles are consent-governed, deletable, and isolated from third-party engine data where required.

18.4 Security controls

Service-to-service AWS IAM roles; no static production credentials.
Secrets in AWS Secrets Manager.
RDS encryption, S3 encryption, TLS in transit, private subnets for internal services.
WAF/rate limiting on public endpoints.
Admin role model: viewer, merchandiser, approver, product admin, engineer admin.
Audit all privileged config changes, previews, publishes, reverts, and inspector access.

19. Observability and Operational Readiness

19.1 Service level objectives

Initial targets should be validated against real UX expectations:

Metric	Initial target
Search API p95 latency	under 250 ms excluding client network
Autosuggest p95 latency	under 150 ms excluding client network
Search availability	99.9% monthly
Stock/price index freshness	under 5 minutes for 99% of updates
Event completeness	over 99%
Click attribution validity	over 99% of result clicks include search/page and position
Rx rule violations	zero
Config publish propagation	under 5 minutes, preferably seconds

19.2 Alerts

Alert on:

Search/suggest latency, error-rate, and timeout spikes.
Zero-result and fallback-rate anomalies by query/category.
Index queue age, DLQ size, index failures, and stale stock/price.
Event gaps and duplicate-event spikes.
Empty widget rate and serving-context integrity failures.
Rule publication failures/cache propagation gaps.
Safety-rule test failures and prohibited result exposure.
Capacity headroom before planned sales events.

Every alert must have an owner, severity, runbook, and escalation policy.

19.3 Test strategy

Test type	Purpose
Unit tests	Query normalization, eligibility, rule evaluation, scoring functions.
Contract tests	Catalogue event and Discovery API schema compatibility.
Golden relevance tests	Important search/PLP/widget cases and safety assertions.
Integration tests	Typesense indexing/filtering, PostgreSQL config loading, event emission.
End-to-end tests	Search through UI, analytics attribution, admin preview/publish/revert.
Load tests	Diwali-scale expected traffic, cache behavior, queue pressure.
Shadow tests	Compare candidate engines/profiles with production-like traffic.
Data quality tests	Attribute coverage, product count reconciliation, stale data detection.

Golden cases should include direct product/brand matches, common typos, protected terms, concern/ingredient queries, zero-result recoveries, filters, variant grouping, pins/hides, and Rx exclusions.

20. Netcore-to-Typesense Migration and Rollout

Compass remains the stable client-facing facade throughout migration. Web and mobile clients continue calling the same Compass APIs; Compass normalizes provider responses, applies shared safety/attribution policy, and selects the provider internally.

Web / mobile -> Compass public API -> provider router
  -> Netcore adapter
  -> Typesense adapter
  -> shadow comparison recorder
  -> one canonical Compass response contract

20.1 Provider interface and modes

Compass defines a provider-neutral internal contract for search, autosuggest, and browse. Netcore and Typesense implement that contract; provider-specific response shapes never reach clients.

DiscoveryProvider
  search(request) -> normalized provider response
  suggest(request) -> normalized provider response
  browse(request) -> normalized provider response

Provider-routing modes:

Mode	Customer response	Background behavior	Intended use
`netcore`	Netcore	None	Default at migration start and emergency global fallback.
`shadow`	Netcore	Typesense executes and comparison is recorded	Production parity validation with zero customer impact.
`typesense`	Typesense	Optional sampled Netcore comparison	Employees/internal QA and controlled tests.
`typesense_with_netcore_fallback`	Typesense	Netcore is used only after a Typesense failure or invalid response	Public canary and progressive rollout.

Compass owns request validation, identity/session context, query/search IDs, Rx/listing/availability policy, response normalization, discovery context tokens, exposure events, and provider-comparison records. This guarantees behavior that must be consistent across engines.

20.2 Dual indexing and compatibility inventory

Keep the existing Netcore feed unchanged while a Compass index worker builds and maintains the Typesense index:

Atlas / Storehouse / Medusa change
  -> existing Netcore feed
  -> Compass Typesense index worker
  -> canonical DiscoveryProduct transform
  -> bulk Typesense upsert/delete with source_version

Every provider result must map to shared canonical identifiers before comparison or client response:

provider item ID -> canonical_product_id -> product_family_id -> display_variant_id

Maintain an explicit migration inventory. A Netcore behavior is not removed until its Compass/Typesense equivalent passes staging and shadow validation.

Existing behavior	Target ownership/action
Synonyms and typo exceptions	Compass-governed terms; Typesense synonym/query configuration.
Protected brands, families, and molecules	Compass protected-term policy before correction.
Redirects	Compass redirect rules.
Pins, hides, boosts, buries	Compass rule model, with Typesense curation used only as a retrieval optimization.
Listing, availability, Rx policy	Shared Compass eligibility layer; never engine-specific.
Facets, filter values, category/brand identity	Canonical attribute dictionary and stable IDs.
Sorts/ranking	Versioned Compass ranking profiles and TypeSense query parameters.
Analytics	Typesense native feedback plus PostHog/Compass attribution contract.

Reconcile dual indexes continuously: active/listable count, in-stock count, Rx count, category/brand/ingredient coverage, canonical product-family coverage, price/stock mismatch rate, and source-to-index lag.

20.3 Shadow comparison

In shadow mode, Netcore serves the user while Compass invokes Typesense asynchronously with a bounded timeout. The Typesense request never delay the visible response and uses enable_analytics=false so it cannot double-count popular/no-hit query data. Store a comparison record with:

Raw and normalized query, filters, sort, platform, surface, and query interpretation.
Per-provider latency, errors, result count, product-family IDs/positions, display variants, facets, redirects, and fallback state.
Top-3/top-10/top-24 canonical product overlap.
Exact product and brand rank for golden/known queries.
Safety, listing, stock, price, filter/facet, and variant-duplication mismatches.
Active configuration/ranking profile versions.

Do not define success as identical rankings. The goal is Typesense non-inferiority against product expectations and safety requirements. Review high-volume, high-value, concern/ingredient, typo, Hinglish, Rx-sensitive, and high-difference query groups manually as well as automatically.

20.4 Sticky percentage rollout

Provider assignment is deterministic, server-side, and sticky. Use user_id when authenticated, otherwise anonymous_id, with session_id as the final fallback:

bucket = hash("discovery-provider-v1" + assignment_id) % 100

At a 25% rollout, buckets 0-24 receive typesense_with_netcore_fallback; remaining buckets receive Netcore. Preserve the assignment after sign-in and never switch a user between providers within a session. This protects pagination consistency, user experience, attribution, and experiment validity.

Stage	Public Typesense traffic	Minimum observation	Primary decision purpose
Shadow	0%	1-2 weeks	Data, result, safety, and latency parity.
Internal beta	0%	1 week	Employee/QA workflow validation.
Canary	1%	24-72 hours	Reliability, safety, freshness, and event integrity.
Early rollout	5%	3-7 days	Query-quality and funnel guardrails.
Controlled rollout	10%, then 25%	At least 1 week each	Conversion/revenue non-inferiority.
Majority	50%	Full business cycle; include campaign/sale if possible	Operational confidence at meaningful scale.
Full web	100%	2-4 week stabilization	Vendor fallback remains available.
Mobile	Restart at 1% per platform	Independent progression	SDK/network and platform parity.

At roughly 250,000 daily visitors, 1% is around 2,500 visits per day: enough for technical detection, not enough to claim small business-metric lift. Advance by decision gates, not elapsed time alone.

20.5 Feature flags and control precedence

Use separate controls for engine rollout and individual Typesense capabilities. Do not use one use_typesense flag for every behavior. Provider controls:

discovery.provider.search
discovery.provider.autosuggest
discovery.provider.browse
discovery.global_kill_switch

Capability controls:

discovery.typesense.query_normalization
discovery.typesense.typo_tolerance
discovery.typesense.synonyms
discovery.typesense.protected_terms
discovery.typesense.product_grouping
discovery.typesense.filters
discovery.typesense.redirects
discovery.typesense.fallback_ladder
discovery.typesense.merchandising_rules
discovery.typesense.native_analytics
discovery.typesense.popularity_tiebreaker
discovery.typesense.hinglish_aliases

Flags are for rollout, experimentation, and emergency control. Permanent product decisions such as synonyms, pins, redirects, and campaigns belong in the Compass merchandising configuration model. Flag evaluation order:

Emergency global kill switch
Safety/maintenance routing prerequisite
Internal QA/support allowlist or blocklist
Surface/platform restriction
Explicit experiment assignment
Sticky percentage rollout bucket
Default Netcore routing

Evaluate provider routing in Compass, not only in browser/mobile code. PostHog may provide experiment assignment, but safety-critical provider routing must continue to function if PostHog is unavailable. Log the selected provider, flag/configuration version, assignment reason, fallback outcome, request ID, and search ID on every request.

20.6 Backward compatibility

Maintain four compatibility guarantees during migration:

Client API: Compass endpoint paths, product-card fields, facets, sorting, pagination, and error semantics remain stable. New metadata is additive and optional.
Identity: Both providers map to the same canonical product/family/variant/brand/category identifiers.
Behavior: Existing Netcore rules are inventoried and reproduced or explicitly retired only after testing. Rx/safety behavior is centralized before public Typesense traffic.
Data: Netcore and Typesense index feeds run in parallel until decommission; all Typesense documents carry source product IDs, source versions, index timestamps, listability, purchasability, stock, and price.

20.7 Decision gates, fallback, and decommission

Hard-stop events immediately route the affected cohort to Netcore:

Verified Rx or safety-rule violation.
Invalid provider response, material price/availability breach, or index freshness breach.
TypeSense error/timeout rate above threshold or cluster health/quorum incident.
Attribution failure that makes the cohort unmeasurable.

Guardrail failures pause the rollout and trigger investigation:

p95/p99 latency, zero-result, reformulation, search-exit, click, cart, revenue/session, and facet-empty-state regressions versus Netcore control.
Event completeness, price/stock mismatch, index lag, or unresolved high-volume relevance differences.

Advance only when hard-stop metrics remain clean; guardrails are within tolerance; golden/high-volume review passes; Typesense is non-inferior on primary business outcomes; and discovery engineering, product, merchandising, catalogue/inventory, analytics, and clinical/compliance owners approve the review. Use interleaving only for ranking-only comparisons after candidate sets, grouping, filters, and safety behavior have reached parity. Do not interleave incompatible result sets or different redirect/fallback experiences. Netcore can be decommissioned only after 100% web traffic has completed the agreed stabilization period, mobile has completed or has an approved separate plan, safety mismatch is zero, search/business guardrails meet or exceed control, TypeSense capacity/recovery/snapshots are proven, vendor configurations are migrated/retired, and operations/support can diagnose through Compass rather than vendor tooling.

21. Delivery Plan

Phase 0: Discovery foundation and engine proof of concept

Deliverables:

Catalogue completeness audit and controlled attribute dictionary.
Canonical DiscoveryProduct contract.
100+ golden search/recommendation safety cases.
Baseline of current query, conversion, zero-result, latency, and reformulation metrics.
Typesense proof of concept against real products and queries.
Event schema and attribution design.
Architecture decision record and production SLOs.

Exit criteria:

Engine meets representative relevance and latency needs.
The team can ingest/update a representative product set.
The catalogue gaps blocking Phase 1 are assigned with owners.

Phase 1: Search, autosuggest foundation, and safety

Build:

Discovery API, Typesense collection/indexer, and catalogue delta ingestion.
Search result integration on one chosen surface.
Normalization, synonyms, protected terms, typo tolerance, and deterministic intent detection.
Eligibility, stock/listability, product-family grouping, basic facets, and ranking profiles.
Fallback ladder, redirects, pins/hides/boosts, and a narrow admin console.
Search/suggestion event backbone, query dashboard, inspector, alerts, regression suite, staged rollout, and shadow mode.
Rx and claim-safety rules.

Exit criteria:

Golden cases pass.
Safety violations are zero.
Event completeness and attribution targets are met.
Search quality matches or beats the current system under shadow/interleaving evaluation.
A merchandiser can correct a known query problem without deploying code.

Phase 2: Browse and SEO-safe PLPs

Build:

Category, brand, concern, ingredient, and campaign page definitions.
Configurable facets, quick filters, sorting, content slots, badges, and page-level rules.
Search-inside-filter for high-cardinality facets.
Stable URLs, redirects, canonical/index policy, and organic-traffic monitoring.
Catalogue quality gates and assisted enrichment workflow.
Page/filter health dashboards and page inspector.

Phase 3: Recommendations

Build:

Widget framework, strategies, placements, and measurement.
Catalogue relationship data, hard exclusion filters, routine conflict/complement rules, and fallback chains.
Curated and relationship-based widgets first, co-purchase candidates second.
Offline review fixtures and widget experiments.
Recently viewed/continue shopping and session intent adaptation.
Hinglish coverage expanded based on Phase 1 query data.

Phase 4: Full control plane and learning

Build:

Campaign bundles, bulk imports, approval workflows, and rule-health monitoring.
Ranking review workflow, bounded behavioral signals, profile-aware recommendations, and discovery profiles.
Experiment registry, automatic proposal queues, holdouts, and weekly discovery review.
Sponsorship only after explicit commercial, legal, relevance, and safety approval.

22. Cost and Scaling Strategy

22.1 Traffic planning assumptions

At roughly 250,000 daily website visitors, discovery is a meaningful production workload but not, by itself, a reason to introduce Kafka or OpenSearch. Use measured funnel data for final capacity planning. Until it is available, size the initial benchmark and load tests using these conservative assumptions:

Workload	Planning assumption	Approximate daily volume	Capacity implication
Site visitors	Known baseline	250,000 visits	Traffic is bursty; average daily rate is not a sizing target.
Executed searches	2-5 searches per visitor	500,000-1,250,000 searches	Average is about 6-15 requests/second across a day; test 10-20x average for peak sale-event traffic.
Autosuggest	3-8 requests per executed search after client debouncing	1.5M-10M requests	This is likely the highest request volume; debounce at 150-250 ms, cancel stale requests, cache short prefixes carefully, and target lower latency than full search.
Discovery interaction events	5-15 server/client events per visitor	1.25M-3.75M events	PostHog and first-party ingestion need capacity testing; events remain asynchronous and never block the customer response.
Catalogue deltas	Depends on catalogue and inventory activity, not visitor count	Usually far below query traffic	SNS/SQS with idempotent workers and DLQs is sufficient for product, price, stock, and listing updates.

Peak traffic must be determined from hourly production data, not daily averages. A sale, campaign, push notification, or SEO landing-page burst can reasonably produce 10-20 times average traffic. The Phase 1 load test should use the higher of the measured peak or this planning multiplier, with separate test mixes for search, autosuggest, filters, and product-click event ingestion.

22.2 Why SNS/SQS remains sufficient at this scale

SNS/SQS is the selected event backbone for Phases 0-3. It is appropriate because catalogue indexing is an asynchronous, idempotent work queue:

Each product, price, inventory, or listing update is processed independently using product_id + source_version as its idempotency key.
Ordering is required only per product, not globally. The indexer rejects stale versions, so normal SQS at-least-once delivery is safe.
Worker concurrency can scale with queue depth through the existing Kubernetes/KEDA deployment model.
SQS provides retries, visibility timeouts, buffering, and DLQ isolation already used by the platform.
The primary high-volume path is synchronous query serving to Typesense, not catalogue event consumption. Kafka would not improve search-request latency.

Kafka or MSK should be evaluated only if the platform later needs a shared, long-retention, replayable stream for many consumers or low-latency behavioral aggregation for personalization/ranking. It is not a prerequisite for serving this visitor volume.

22.3 Cost priorities

At the start, engineering time and operational risk cost more than a small difference in infrastructure pricing. Self-hosted Typesense is appropriate while:

Product schemas and relevance rules are evolving frequently.
Traffic is moderate.
The team is focused on building discovery capabilities rather than search-cluster operations.

Cost review should include total cost of ownership:

Typesense compute, persistent volumes, backups, and load-balancing price
+ compute and database cost
+ storage and analytics cost
+ engineering time for operations/tuning
+ incident/on-call cost
+ delayed delivery cost

22.4 Scaling paths

Component	Early stage	Growth stage
API	Compass deployment with existing KEDA scaling on CPU/RPS/latency	Split read-heavy query workloads and admin only if required; multi-region only with evidence.
Indexer	Single consumer group with SQS batching	Parallel workers partitioned by product ID; backpressure/DLQ monitoring.
PostgreSQL	RDS/Aurora with backups and read replica only if needed	Read replicas for inspector/admin analytics, partition audit/event metadata.
Redis	Small HA ElastiCache	Separate rate-limit/config/result-cache usage if contention emerges.
Typesense	Three self-hosted nodes with private Raft peering, persistent SSD/NVMe data and analytics volumes	Size CPU/RAM from benchmark data; move to five nodes only when two-node fault tolerance is justified; evaluate OpenSearch only against evidence.
Analytics	Typesense analytics, PostHog, Compass operational records, and authoritative commerce events	Add S3/Athena/dbt or the company warehouse only when the Section 17.4 triggers are met.

Do not prematurely introduce Kafka, a separate Kubernetes platform, a data lakehouse, feature stores, or dedicated vector databases. Introduce them only when the existing queues, PostHog/warehouse reporting, or engine capabilities genuinely prove insufficient.

23. Open Decisions and Owners

Decision	Suggested owner	When needed
Product family vs flavour grouping policy	Product + catalogue	Before indexing contract is final
Delivery-area handling: badge, demote, or hide	Product + operations	Before location-aware inventory launch
Rx claim/promotion boundaries	Clinical/compliance + product	Before Phase 1 public launch
Attribute coverage thresholds	Catalogue + product	Phase 0 completion
Search-first vs browse-first after Phase 1	Product + analytics	Phase 1 exit review
Sensitive recent-search behavior	Privacy/compliance + product	Before autosuggest launch
Sponsored autosuggest ban	Product + commercial + compliance	Before any sponsorship design
Search engine migration trigger thresholds	Engineering + finance + product	Before Phase 2 cost review

24. Immediate Next Steps

Confirm the Phase 1 surface: web search results is the recommended first integration.
Run the catalogue completeness audit and define the controlled attribute dictionary.
Collect 100-200 real, anonymized queries and build the first golden-query suite.
Create a Typesense proof of concept using the proposed DiscoveryProduct contract.
Define the event schema, Typesense analytics rules, and PostHog server/client ownership.
Build the eligibility/Rx policy module and test it independently before integrating search UI.
Establish the PostgreSQL configuration schema and a minimal query-rule preview workflow.
Shadow the existing vendor/system before any user traffic is shifted.

25. Definition of a Successful First Release

The first release is successful when it demonstrates all of the following in production or controlled rollout:

Users can find exact products, brands, categories, concerns, and ingredients reliably.
Common typos and approved aliases recover without damaging protected brands/molecules.
Non-listable, non-purchasable, and prohibited Rx contexts never leak into results.
Product variants are not duplicated in search.
A zero-result query follows a measured, honest fallback path.
Search clicks, add-to-carts, and purchases are attributable to the returned result list and position.
Product/merchandising can publish, preview, inspect, and revert a query-level correction without engineering.
The team can explain why a product appeared at its position.
The platform can be rolled back or degraded safely if the engine or index pipeline fails.

That foundation is enough to begin browse and recommendations confidently. It is deliberately more valuable than shipping advanced ML, broad personalization, or a large control plane before the core retrieval, safety, data, and measurement loops are dependable.

​Discovery Platform Architecture and Delivery Design

​1. Purpose

​2. Architecture Decisions

​3. Why This Stack

​3.1 Typesense for the first production version

​3.2 Why not Meilisearch by default

​3.3 Why not OpenSearch initially

​3.4 When to move to OpenSearch

​4. Guiding Principles

​5. High-Level Design

​5.1 Runtime request path

​6. Service Boundaries

​7. Infrastructure Design

​7.1 AWS resources

​7.2 Environments

​7.3 Availability and degraded operation

​8. Data Model

​7.4 Self-hosted Typesense operations

​8.1 Discovery product document

​8.2 PostgreSQL control-plane entities

​8.3 Bootstrap state and seeded configuration

​8.4 Merchandising rule format

​8.5 Phase 1 event model and destinations

​9. Discovery API

​9.1 External endpoints

​9.2 Search request

​9.3 Search response

​10A. End-to-End Discovery Flows

​A.1 Exact product and product-family search

​A.2 Brand search

​A.3 Category, ingredient, and concern search

​A.4 Hinglish and mixed-language search

​A.5 Typo-tolerant and attribute-rich search

​A.6 Autosuggest

​A.7 Browse and PLP flow

​A.8 Semantic and hybrid search

​A.9 Redirect, medical, and fallback flow

​A.10 Recommendation retrieval flow

​A.11 Unified decision tree

​A.12 Responsibility summary

​10. Search Low-Level Design

​10.1 Query understanding pipeline

​10.2 Candidate retrieval

​10.3 Eligibility policy

​10.4 Ranking pipeline

​10.5 Fallback ladder

​10.6 Hinglish and multilingual support

​11. Autosuggest Design

​11.1 Suggest endpoint behavior

​11.2 Suggestion quality controls

​12. Browse and PLP Design

​12.1 Page definition

​12.2 Filters and sort

​12.3 SEO requirements

​13. Recommendations Design

​13.1 Shared widget pipeline

​13.2 Candidate sources in delivery order

​13.3 Routine intelligence

​13.4 Recommendation quality gate

​14. Merchandising Control Plane

​14.1 Initial operator workflows

​14.2 Publishing workflow

​14.3 Configuration caching

​14.4 Compass source of truth and Typesense compilation

​14.5 Rule publication, synchronization, and runtime behavior

​15. Ranking Governance and Explainability

​15.1 Ranking profile example

​15.2 Explain record

​16. Catalogue Ingestion and Quality

​16.1 Delta ingestion

​16.2 Freshness objectives

​16.3 Quality gates

​16.4 Assisted enrichment

​17. Analytics, Reporting, and Experiments

​17.1 Phase 1 analytics architecture

​17.2 Typesense native analytics

​17.3 PostHog analytics and attribution

​17.4 Future warehouse plan

​17.5 Future owned analytics stack

​17.6 Event ownership and source of truth

Discovery Platform Architecture and Delivery Design

1. Purpose

2. Architecture Decisions

3. Why This Stack

3.1 Typesense for the first production version

3.2 Why not Meilisearch by default

3.3 Why not OpenSearch initially

3.4 When to move to OpenSearch

4. Guiding Principles

5. High-Level Design

5.1 Runtime request path

6. Service Boundaries

7. Infrastructure Design

7.1 AWS resources

7.2 Environments

7.3 Availability and degraded operation

8. Data Model

7.4 Self-hosted Typesense operations

8.1 Discovery product document

8.2 PostgreSQL control-plane entities

8.3 Bootstrap state and seeded configuration

8.4 Merchandising rule format

8.5 Phase 1 event model and destinations

9. Discovery API

9.1 External endpoints

9.2 Search request

9.3 Search response

10A. End-to-End Discovery Flows

A.1 Exact product and product-family search

A.2 Brand search

A.3 Category, ingredient, and concern search

A.4 Hinglish and mixed-language search

A.5 Typo-tolerant and attribute-rich search

A.6 Autosuggest

A.7 Browse and PLP flow

A.8 Semantic and hybrid search

A.9 Redirect, medical, and fallback flow

A.10 Recommendation retrieval flow

A.11 Unified decision tree

A.12 Responsibility summary

10. Search Low-Level Design

10.1 Query understanding pipeline

10.2 Candidate retrieval

10.3 Eligibility policy

10.4 Ranking pipeline

10.5 Fallback ladder

10.6 Hinglish and multilingual support

11. Autosuggest Design

11.1 Suggest endpoint behavior

11.2 Suggestion quality controls

12. Browse and PLP Design

12.1 Page definition

12.2 Filters and sort

12.3 SEO requirements

13. Recommendations Design

13.1 Shared widget pipeline

13.2 Candidate sources in delivery order

13.3 Routine intelligence

13.4 Recommendation quality gate

14. Merchandising Control Plane

14.1 Initial operator workflows

14.2 Publishing workflow

14.3 Configuration caching

14.4 Compass source of truth and Typesense compilation

14.5 Rule publication, synchronization, and runtime behavior

15. Ranking Governance and Explainability

15.1 Ranking profile example

15.2 Explain record

16. Catalogue Ingestion and Quality

16.1 Delta ingestion

16.2 Freshness objectives

16.3 Quality gates

16.4 Assisted enrichment

17. Analytics, Reporting, and Experiments

17.1 Phase 1 analytics architecture

17.2 Typesense native analytics

17.3 PostHog analytics and attribution

17.4 Future warehouse plan

17.5 Future owned analytics stack

17.6 Event ownership and source of truth