> ## Documentation Index
> Fetch the complete documentation index at: https://engineering.clinikally.work/llms.txt
> Use this file to discover all available pages before exploring further.

# Discovery Platform Architecture

> Target architecture and phased delivery plan for evolving Compass from vendor-backed discovery to an owned platform.

# Discovery Platform Architecture and Delivery Design

## 1. Purpose

This document turns `search-prd.txt` into an implementation design for an owned discovery platform. It covers the recommended technology stack, system boundaries, high-level and low-level designs, data contracts, feature delivery sequence, safety controls, operations, and decision criteria for evolving the search engine.

It extends the existing platform instead of creating a parallel discovery stack: Compass remains the customer-facing discovery service; Atlas remains the catalogue edge; Sesh provides anonymous/session context; Tetris composes CMS-backed content and widgets; Medusa, Storehouse, and Dermadesk remain sources for commerce, availability, and prescription/consultation context.

The complete architecture remains in this document. [Discovery platform low-level design](/architecture/discovery-platform-low-level-design) is an additive companion focused on executable query parsing, TypeSense request construction, ranking execution, event routing, and use-case examples.

The first production goal is deliberately narrow:

> A user can search for a product, brand, concern, or ingredient; receive safe and relevant results; have the interaction measured; and have a merchandiser correct a bad outcome without an engineering deployment.

The platform is not a search engine replacement alone. It is a product layer around candidate retrieval that owns catalogue quality, safety eligibility, ranking policy, merchandising, attribution, experimentation, and operational visibility.

## 2. Architecture Decisions

| Decision              | Recommendation                                                                          | Reason                                                                                                                                                |
| --------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| Initial search engine | Self-hosted open-source Typesense                                                       | Supports typo tolerance, fields, facets, filtering, grouping, autocomplete, and native search analytics while retaining operational/data control.     |
| Search ownership      | Extend Compass (FastAPI) as the application-owned Discovery API                         | Preserves its existing ownership of search, browse, autosuggest, and recommendations while keeping domain rules independent of the engine.            |
| System of record      | Existing commerce/catalogue services                                                    | Discovery stores an optimized read model, never the authoritative product record.                                                                     |
| Operational database  | A Compass-owned PostgreSQL schema/database, operated through existing platform patterns | Reliable transactional store for rules, configuration, audits, experiments, and schedules without adding a new service boundary.                      |
| Runtime cache         | Existing Redis platform                                                                 | Low-latency configuration cache, idempotency keys, rate limits, and short-lived result caching.                                                       |
| Compute               | Existing Compass FastAPI API and worker deployments on Kubernetes                       | Matches the established Helm, Argo CD, KEDA, Datadog, Prometheus, and Loki platform.                                                                  |
| Ingestion transport   | SNS/SQS queues                                                                          | Matches the established asynchronous transport, DLQ, and worker model used by Atlas, Storehouse, Gearbox, and other services.                         |
| Phase 1 analytics     | Typesense native analytics + PostHog + Compass PostgreSQL                               | Typesense owns search-native feedback; PostHog owns cross-surface funnels, revenue and experiments; Compass retains inspectable operational evidence. |
| Data warehouse        | Deferred behind concrete cross-system/reporting requirements                            | Do not block search launch on a new data platform when PostHog and authoritative commerce events meet Phase 1 needs.                                  |
| Deployment            | Existing Helm charts, Argo CD, and DevOps Terraform/traffic-split tooling               | Matches the documented production deployment and progressive delivery model.                                                                          |
| Observability         | Existing Datadog, Prometheus, Grafana, Loki, and PostHog                                | Covers metrics, tracing, dashboards, logs, and product analytics without creating duplicate tooling.                                                  |
| Semantic retrieval    | Deferred; add as a second candidate source only                                         | Lexical search, catalogue attributes, and measurement must be trustworthy first.                                                                      |
| Advanced engine       | OpenSearch evaluated later through shadow traffic                                       | Use only when concrete relevance, scale, or cost evidence justifies its higher complexity.                                                            |

## 3. Why This Stack

### 3.1 Typesense for the first production version

Typesense matches the Phase 1 requirements with low implementation and operational burden:

* Fielded full-text search with adjustable text-match weights.
* Typo tolerance, prefix matching, infix matching where appropriate, and search-as-you-type.
* Facets, numeric filters, Boolean filters, sorting, and filter counts.
* Grouping by canonical product ID to prevent variant duplicates.
* Synonym support and curated result overrides where they are useful.
* Fast response times without designing shard topology, analyzers, or a complex query DSL from day one.
* Native analytics rules for popular/no-hit queries, document counters, and pseudonymous behavioral logs.
* One self-contained server binary with a clear Kubernetes deployment model.

Typesense should be treated as a retrieval dependency, not as the discovery platform. Product policy must not be encoded only in its collection schema or its curation features.

### 3.2 Why not Meilisearch by default

Meilisearch is a strong choice for simple product search. It can be the right option for a lightweight MVP if there is no near-term need for detailed ranking governance or complex discovery controls.

This PRD, however, requires a path toward:

* Query-, page-, and context-specific merchandising rules.
* Explicit rank-layer precedence and per-result explanations.
* Product/variant family behavior.
* Rich per-page filter and sorting configuration.
* Controlled behavioral ranking and experiments.
* Hybrid lexical and semantic candidate retrieval.

Typesense gives a better early balance between simple operation and control over retrieval. The platform design below also makes a later move to either Meilisearch or OpenSearch possible.

### 3.3 Why not OpenSearch initially

OpenSearch is mature and highly capable. It becomes attractive when the business has proven needs for complex custom scoring, advanced analyzers, nested variant documents, very high throughput, or a mature search operations team.

It should not be the default starting point because it brings forward substantial work:

* Index mappings and analyzer strategy are expensive to change and require safe reindexing.
* Production reliability requires shard, replica, snapshot, storage, scaling, and upgrade decisions.
* Query DSL, scoring, relevance tuning, and performance diagnostics require specialist expertise.
* A resilient managed deployment has a materially higher baseline cost than a small managed Typesense deployment.

None of those costs eliminate the need to build the Discovery API, catalogue transformation, rule store, analytics, safety layer, and inspectors. Those are the durable capabilities in this PRD.

### 3.4 When to move to OpenSearch

Run an OpenSearch evaluation after Phase 2 or Phase 3 only when one or more evidence-based triggers is met:

* Typesense cannot express an important retrieval or ranking requirement without fragile workarounds.
* The catalogue or query volume makes Typesense materially less cost-effective.
* Strong hybrid retrieval or vector-search quality is a validated requirement.
* Explainability requires score details unavailable from the current engine.
* The team has committed operational ownership and has people experienced with OpenSearch.

Migration must use dual indexing, shadow queries, golden-query comparison, relevance review, and gradual traffic shifting. The public Discovery API does not change.

## 4. Guiding Principles

1. **Safety and eligibility are server-side invariants.** A client, rule, experiment, pin, model, or future sponsored placement cannot reintroduce an ineligible item.
2. **The catalogue is the quality bottleneck.** Attribute coverage and freshness are features, not data-cleanup work outside the platform.
3. **Rules are data, not deployments.** Routine merchandising changes are versioned configuration with previews and audit records.
4. **Every result is attributable.** The platform records the request, configuration version, returned products, positions, and downstream actions.
5. **Search is lexical first.** Exact names, brands, and product families must be dependable before semantic matching is introduced.
6. **Do not create a service per PRD module.** Begin with a cohesive Discovery API and a small set of worker processes. Split only at proven scaling or ownership boundaries.
7. **Configuration needs a rollback story.** All rule and profile changes must be previewable, schedulable, audited, and reversible.
8. **Defaults must work without personalization.** Personalization only reorders already eligible, relevant products and can be disabled per surface.
9. **Production launches are measured and staged.** Shadow mode, canaries, regression tests, and health signals are mandatory for material changes.

## 5. High-Level Design

```text theme={null}
                              +--------------------------+
                              | Atlas / Commerce         |
                              | product, variant, price, |
                              | stock, listing, taxonomy |
                              +------------+-------------+
                                            |
                              SNS/SQS product and inventory events
                                           |
                                           v
  +--------------------+        +--------------------------+        +-------------------+
  | Merchandising      |------->| Discovery configuration  |<-------| Product / Admin   |
  | console            |        | PostgreSQL + audit log   |        | operators         |
   +--------------------+        +------------+-------------+        +-------------------+
                                            |
                                            | cached configuration
                                           v
+-------------------+         +--------------------------+         +-------------------+
  | Web / mobile apps |-------->| Compass Discovery API    |-------->| Typesense         |
| search, PLP, PDP  |         | orchestration and policy |         | candidate index   |
+-------------------+         +-----+---------+----------+         +-------------------+
                                      |         |
                                      |         +--------------------+
                                      v                              v
                           +---------------------+          +---------------------+
                            | Analytics            |          | Inspector / explain |
                            | Typesense + PostHog |          | query/page debug UI |
                           +----------+----------+          +---------------------+
                                      |
                                      v
                            +---------------------+
                            | Future warehouse    |
                            | only when triggered |
                            +---------------------+

                    +------------------------------+
                    | Compass/Gearbox index workers |
                    | SNS/SQS -> transform          |
                    | -> Typesense upsert/delete    |
                    +------------------------------+
```

### 5.1 Runtime request path

```text theme={null}
Client request
  -> authenticate/session context
  -> request validation and rate limiting
  -> load active ranking profile and applicable rules
  -> normalize query and protect terms
  -> classify intent and extract filters/preferences
  -> check redirect rules
  -> retrieve lexical candidates from engine
  -> apply mandatory eligibility/safety policy
  -> group product variants and choose display variant
  -> score/re-rank using profile and approved boosts
  -> apply merchandiser boost/bury/hide rules
  -> apply deterministic pins
  -> execute fallback ladder if needed
  -> build facets, UI metadata, and search ID
  -> asynchronously persist exposure event
  -> return response
```

The Discovery API owns every stage except raw candidate retrieval. It can use engine-native capabilities for efficiency, but it must preserve enough information to explain the final result.

## 6. Service Boundaries

Begin with four deployable applications. Their code may live in one monorepo.

| Application              | Responsibility                                                                                                        | Initial scale model                                                                                    |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `Compass API`            | Search, autosuggest, browse retrieval, ranking orchestration, eligibility, rules, response generation, event emission | Existing FastAPI deployment, Helm release, and KEDA scaling model                                      |
| `Compass index worker`   | Consume catalogue/inventory changes, validate and transform products, index delta updates, execute full rebuilds      | Queue-driven worker deployment with SNS/SQS, retry policy, and DLQ                                     |
| `Compass admin API`      | Internal control-plane endpoints for rules, profiles, previews, audits, and approvals                                 | Begin within Compass; a separate UI can be introduced when operator workflows mature                   |
| `Gearbox/analytics jobs` | Typesense analytics review, PostHog dashboard support, demand-gap jobs, reconciliation and quality reports            | Existing scheduled/background job patterns; warehouse transforms only after a future adoption decision |

Do not make ranking, synonym management, personalization, recommendations, or experiments independent services until they have independent load, release cadence, data storage, or ownership requirements.

## 7. Infrastructure Design

### 7.1 AWS resources

```text theme={null}
Kubernetes
  Compass API deployment and Compass index-worker deployment
  Helm values for DEVELOPMENT, STAGING, and PRODUCTION
  KEDA ScaledObjects for API and queue-driven worker scaling

Platform data stores
  Compass PostgreSQL and Redis
  Self-hosted Typesense 3-node production cluster with persistent data and analytics volumes
  S3 for Typesense snapshots and index rebuild artifacts

Data and events
  SNS topic and SQS index queue + dead-letter queue
  Typesense native analytics and PostHog server/client event delivery

Operations
  AWS Secrets through the existing CSI configuration
  Datadog, Prometheus, Grafana, and Loki
  Argo CD progressive rollout and Terraform traffic-split tooling
```

### 7.2 Environments

Maintain at least `development`, `staging`, and `production`.

* **Development:** synthetic catalogue subset, seed configuration, no production personal data.
* **Staging:** production-like schemas and anonymized/sampled catalogue; validates migrations, index builds, and golden-query tests.
* **Production:** separate database, cache, queues, engine collections, credentials, and dashboards.

Every configuration entity contains environment-local IDs; never promote a production rule by copying database rows manually. Use versioned export/import or an approved configuration promotion flow.

### 7.3 Availability and degraded operation

The `discovery-api` must return a usable response when a downstream dependency is degraded:

| Dependency failure         | Expected behavior                                                                                                    |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Typesense unavailable      | Return cached popular search/category fallback where safe; route to browse/category experience; emit incident event. |
| Redis unavailable          | Read configuration from PostgreSQL at a protected rate; disable non-essential result caching.                        |
| PostgreSQL unavailable     | Use recently loaded, versioned read-only configuration cache; block admin mutations; alert.                          |
| Event pipeline unavailable | Queue events locally/durably where possible; never block search responses on analytics.                              |
| Catalogue feed delayed     | Continue serving last valid index, alert on freshness SLO violation.                                                 |

No degraded response may violate Rx, listing, or price/availability safety requirements. Cached responses should be limited to contexts where eligibility remains valid.

## 8. Data Model

### 7.4 Self-hosted Typesense operations

Production runs a three-node Typesense cluster. Typesense uses Raft consensus and replicates the full dataset to every node; three nodes tolerate one node failure. The Raft peering network remains private, while Compass connects through the internal load-balanced Typesense service and configures individual nodes as fallbacks.

Each node requires persistent SSD/NVMe-backed volumes for both the Typesense data directory and analytics directory. The cluster configuration enables native analytics explicitly:

```text theme={null}
--enable-search-analytics=true
--analytics-dir=/var/lib/typesense-analytics
--analytics-flush-interval=60
```

The 60-second flush interval is the documented minimum. It gives sufficiently fresh popular-query/no-hit data for operational review without treating analytics as a synchronous request dependency. Benchmark this configuration with production-like search, filter, and indexing traffic because native analytics consumes cluster resources.

Operational requirements:

* Use a predefined schema and leave display-only fields unindexed to reduce memory use.
* Plan keyword-search RAM at roughly 2-3 times the combined size of indexed/searchable/filterable/sortable values, then validate with the real catalogue.
* Use collection aliases and versioned collections for incompatible schema changes.
* Snapshot through Typesense's snapshot API and copy the resulting snapshot to S3. Never copy a live data directory directly.
* Monitor `/health`, `/metrics.json`, and `/stats.json`; alert before RAM reaches 85% or sustained CPU reaches 90%.
* Use bulk imports for indexer batches. Treat `503 Not Ready/Lagging` as backpressure, retry with jitter, and keep concurrent bulk imports at or below `vCPU count - 2`.
* Keep analytics collections, analytics directories, API keys, and snapshots isolated per environment.
* Set `enable_analytics=false` on tests and vendor-shadow queries so they do not contaminate production query counts/suggestions.

### 8.1 Discovery product document

Index at the canonical product-family level, not one document per SKU. This satisfies the PRD requirement to avoid near-duplicate variants while preserving variant choices for the PDP/cart.

```ts theme={null}
type DiscoveryProduct = {
  id: string;                       // canonical product/family ID
  sourceProductId: string;
  sourceVersion: string;
  title: string;
  normalizedTitle: string;
  brandId: string;
  brandName: string;
  categoryIds: string[];
  categoryNames: string[];
  concernIds: string[];
  concernNames: string[];
  ingredientIds: string[];
  ingredientNames: string[];
  productType: string;
  routineStep?: string;
  formats: string[];
  skinHairTypes: string[];
  spf?: number;
  ingredientStrengths: Array<{ ingredientId: string; percentage?: number }>;
  searchableDescription: string;
  tags: string[];

  listable: boolean;
  purchasable: boolean;
  inStock: boolean;
  stockStatus: "in_stock" | "low_stock" | "out_of_stock";
  rxClassification: "none" | "prescription" | "consult_required";
  pregnancyBreastfeedingCautions: string[];

  price: { currency: string; minimum: number; maximum: number };
  rating?: number;
  reviewCount?: number;
  launchedAt?: string;
  commercialFlags: string[];

  displayVariants: Array<{
    variantId: string;
    label: string;
    price: number;
    inStock: boolean;
    attributes: Record<string, string | number | boolean>;
  }>;
  defaultDisplayVariantId: string;
  imageUrl: string;
  productUrl: string;

  updatedAt: string;
};
```

Required principles:

* `listable`, `purchasable`, `inStock`, and `rxClassification` are distinct values.
* Do not infer search facets from ungoverned descriptions at query time.
* Attribute IDs are stable canonical IDs; display names can change without breaking rules or URLs.
* Variants are included to select a sensible display option, but the top-level product remains the search/browse unit.
* Product family, pack, and flavour policies must be explicit. The current open decision recommends grouping size variants but not distinct flavours.

### 8.2 PostgreSQL control-plane entities

```text theme={null}
ranking_profiles
  id, name, scope, definition_json, version, status,
  created_by, approved_by, created_at, published_at

merchandising_rules
  id, name, scope_type, scope_id, conditions_json, action_json,
  priority, starts_at, ends_at, status, requires_approval,
  version, created_by, updated_by, approval_state

typesense_rule_sync
  id, merchandising_rule_id, typesense_resource_type,
  typesense_resource_id, source_rule_version, sync_status,
  synced_at, last_error

rule_audits
  id, rule_id, action, before_json, after_json, reason,
  actor_id, occurred_at

synonym_sets
  id, canonical_term, alternatives_json, scope, status,
  created_by, approved_by, version

protected_terms
  id, term, term_type, locale, status, created_by, updated_at

redirect_rules
  id, query_match_type, query_value, destination_type,
  destination_id, starts_at, ends_at, status

campaigns
  id, name, starts_at, ends_at, status, owner_id, metadata_json

campaign_members
  campaign_id, member_type, member_id

experiments
  id, hypothesis, surface, assignment_unit, variants_json,
  primary_metric, guardrails_json, starts_at, ends_at, status

golden_cases
  id, case_type, input_json, assertions_json, severity, enabled

attribute_dictionary
  id, attribute_type, canonical_value, display_name, aliases_json,
  status, owner_id
```

Use `JSONB` for flexible condition/action definitions, but validate every entity through typed application schemas. Do not rely on arbitrary JSON interpreted at runtime without versioning and validation.

### 8.3 Bootstrap state and seeded configuration

Database migrations create empty tables, but the platform must not launch with an empty control plane. Bootstrap data is a reviewed, versioned configuration bundle imported after schema migration and before Typesense traffic is enabled.

| Entity                          | Initial state                                            | How it is populated                                                                                                                 |
| ------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `attribute_dictionary`          | Required before indexing                                 | Catalogue/product-approved canonical categories, brands, concerns, ingredients, formats, routine steps, safety values, and aliases. |
| `ranking_profiles`              | At least one active profile                              | Seed a `search-default` profile with conservative lexical weights and behavioral ranking disabled.                                  |
| `protected_terms`               | Required baseline                                        | Import brand, product-family, molecule, and clinical terms from catalogue/compliance review.                                        |
| `synonym_sets`                  | Curated baseline only                                    | Import existing vendor synonyms after normalization, deduplication, and owner approval.                                             |
| `redirect_rules`                | Existing navigational redirects                          | Import active vendor redirects that remain intentional.                                                                             |
| `golden_cases`                  | Required before shadow mode                              | 100-200 anonymized priority queries plus expected outcomes and safety assertions.                                                   |
| `merchandising_rules`           | Empty unless an approved migration/campaign needs a rule | Do not copy every legacy vendor manipulation blindly; recreate only reviewed rules.                                                 |
| `campaigns`, `campaign_members` | Empty until used                                         | Created by operators after the basic rule flow is proven.                                                                           |
| `experiments`                   | Empty until first controlled rollout                     | Provider rollout/experiment configuration is created explicitly, not inferred.                                                      |
| `typesense_rule_sync`           | Empty until a rule/synonym is compiled                   | Derived operational state; never hand-seeded.                                                                                       |
| `rule_audits`                   | Empty at creation                                        | Append-only records begin with seed imports and subsequent operator actions.                                                        |

Seed imports must be idempotent, include a source/version and an owner, produce an audit record, and support dry-run validation. Treat seed data as configuration under change control, not application code and not manually edited production SQL.

### 8.4 Merchandising rule format

```json theme={null}
{
  "scope": {
    "surface": "search",
    "pageType": null
  },
  "conditions": {
    "all": [
      { "field": "query.normalized", "operator": "contains", "value": "sunscreen" },
      { "field": "request.platform", "operator": "equals", "value": "web" },
      { "field": "time.now", "operator": "between", "value": ["2026-04-01T00:00:00Z", "2026-06-30T23:59:59Z"] }
    ]
  },
  "action": {
    "type": "pin",
    "productId": "product_123",
    "position": 1
  },
  "priority": 100
}
```

Supported Phase 1 actions:

* Redirect query to page/destination.
* Pin a product at a deterministic position.
* Hide a product from a scoped context.
* Boost or bury an eligible product.
* Add or disable a synonym/protected term.

Later actions include content injection, curated recommendation replacement, user-segment rules, campaigns, and bulk imports.

### 8.5 Phase 1 event model and destinations

The shared event contract is versioned and common across web/mobile/server producers, but Phase 1 does not persist every event into Compass PostgreSQL. Each event is routed to the system that needs it:

| Destination                | Event subset                                                                                  | Purpose                                                                                 |
| -------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Typesense native analytics | Search auto-capture plus verified `click`, `conversion`, and optional `visit`                 | Popular/no-hit queries, suggestions, bounded document counters, optional behavior logs. |
| PostHog                    | Search exposure, UI interaction, cart, order, refund/cancellation, feature-flag exposure      | Product funnels, cross-surface revenue, cohorts, and experiments.                       |
| Compass PostgreSQL         | Explain record, rule/profile/configuration application, provider comparison, index sync state | Operational diagnosis, safety, auditing, and control-plane evidence.                    |
| Vector/Medusa/Orderbox     | Cart/order line attribution and commerce state                                                | Authoritative commercial source of truth.                                               |

The portable envelope below is the internal contract from which PostHog and Typesense-specific payloads are constructed:

```json theme={null}
{
  "event_id": "uuid",
  "event_name": "search_executed",
  "schema_version": 1,
  "occurred_at": "2026-08-03T10:15:20.123Z",
  "anonymous_id": "hashed-session-or-device-id",
  "user_id": "optional-known-user-id",
  "session_id": "uuid",
  "platform": "web",
  "request_id": "uuid",
  "discovery_context": {
    "search_id": "uuid",
    "surface": "search_results",
    "ranking_profile_version": "search-default:3",
    "config_version": "2026-08-03T10:15:00Z"
  },
  "properties": {}
}
```

Key internal event properties:

| Event                   | Required properties                                                                                                      |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `search_executed`       | raw/normalized query, interpretation, filters, sort, result count, fallback, returned product IDs and positions, latency |
| `search_result_clicked` | `search_id`, product ID, position, selected variant, click target                                                        |
| `suggestion_shown`      | prefix, suggestion ID, type, rank                                                                                        |
| `suggestion_clicked`    | suggestion ID, type, rank, destination                                                                                   |
| `plp_viewed`            | page ID, page type, product IDs and positions, active filters/sort                                                       |
| `filter_changed`        | page/search ID, filter, value, action, resulting count                                                                   |
| `widget_impression`     | widget ID, strategy/version, anchor context, products and slots, fallback source                                         |
| `add_to_cart`           | product ID, variant ID, originating search/page/widget context                                                           |
| `purchase_completed`    | order ID, line items, all valid discovery attribution contexts                                                           |

Destination routing rules:

* **Search:** Compass emits `search_executed` to PostHog and keeps its result/position/rule explain evidence. TypeSense auto-captures the actual engine search for popular/no-hit analytics; shadow/test requests use `enable_analytics=false`.
* **Client interaction:** web/mobile sends click, suggestion, filter, sort, and visible-impression events to the first-party endpoint. It validates the signed discovery context, forwards the normalized event to PostHog, and forwards only verified click/visit events needed by TypeSense rules.
* **Cart/order:** Vector/Medusa persists discovery attribution on the line item and emits server-side PostHog events. A worker may forward verified TypeSense `conversion` events for bounded counters only after cart/order validation.
* **Operational evidence:** Compass stores short-lived explain/provider-comparison records, not raw clickstream, in PostgreSQL.

The server emits authoritative exposure/result and commerce events; clients emit UI interaction events. Validators detect duplicates/missing data, and PostHog business outcomes reconcile to Medusa/Orderbox before they are trusted for decision-making.

## 9. Discovery API

### 9.1 External endpoints

```text theme={null}
POST /v1/discovery/search
POST /v1/discovery/suggest
GET  /v1/discovery/pages/{pageSlug}
POST /v1/discovery/widgets/{widgetKey}

POST /v1/admin/search/preview
POST /v1/admin/query-inspector
POST /v1/admin/page-inspector
```

The client never calls Typesense directly. This prevents rule bypasses and gives one place to apply eligibility, attribution, rollout flags, and response shaping.

### 9.2 Search request

```json theme={null}
{
  "query": "vit c serum for oily skin",
  "surface": "search_results",
  "platform": "web",
  "filters": {
    "brandIds": ["brand_1"],
    "formats": ["serum"],
    "price": { "lte": 700 }
  },
  "sort": "featured",
  "page": 1,
  "pageSize": 24,
  "sessionId": "uuid",
  "userId": "optional",
  "location": { "pincode": "optional-later" }
}
```

### 9.3 Search response

```json theme={null}
{
  "searchId": "uuid",
  "query": {
    "raw": "vit c serum for oily skin",
    "normalized": "vitamin c serum for oily skin",
    "intent": "combination",
    "rewrites": [{ "from": "vit c", "to": "vitamin c", "reason": "approved_alias" }],
    "extracted": {
      "ingredients": ["vitamin_c"],
      "skinHairTypes": ["oily"],
      "productTypes": ["serum"]
    }
  },
  "results": [
    {
      "productId": "product_123",
      "displayVariantId": "variant_123",
      "title": "Vitamin C Brightening Serum",
      "brand": "Example Brand",
      "price": 599,
      "inStock": true,
      "badges": ["bestseller"],
      "position": 1
    }
  ],
  "facets": [{ "key": "formats", "values": [{ "value": "serum", "count": 42 }] }],
  "pagination": { "page": 1, "pageSize": 24, "total": 42 },
  "fallback": null,
  "metadata": {
    "rankingProfile": "search-default:3",
    "latencyMs": 75
  }
}
```

Do not expose score internals in the public API. Retain them in a server-side explain record keyed by `searchId`, with access controlled for operations and support.

## 10A. End-to-End Discovery Flows

Every discovery request uses the same boundary: Compass interprets and governs the request; TypeSense performs primary candidate retrieval, filters, grouping, facets, and ranking; Compass performs final defensive safety/external eligibility checks, response shaping, and attribution.

```text theme={null}
Web / mobile
  -> Compass: query understanding, policy, profile and rule selection
  -> TypeSense: retrieval, filtering, grouping, ranking, native curation
  -> Compass: final validation, context token, explanation and event evidence
  -> Web / mobile
```

TypeSense executes the real-time ranking. Compass does not implement a general product-ranking engine in application code.

### A.1 Exact product and product-family search

Examples: `Acne UV Gel`, `Cetaphil Gentle Cleanser`, `Minimalist 10% Niacinamide`.

```text theme={null}
1. Compass normalizes whitespace/case and recognizes protected product-family terms.
2. Compass classifies high-confidence product intent and selects an exact-product lexical profile.
3. Compass sends TypeSense title/product-family/brand-weighted query, eligibility filters, grouping, and matching pins/hides.
4. TypeSense performs typo-tolerant retrieval and ranks canonical product families.
5. Compass validates eligibility, selects the display variant, records result positions, and returns signed result tokens.
```

Vector/hybrid retrieval is disabled or heavily downweighted for this flow. An exact product/brand match must not lose to a semantically similar product.

### A.2 Brand search

Examples: `Cetaphil`, `Minimalist`, `Sebamed`.

```text theme={null}
1. Compass matches the normalized query against the governed brand dictionary.
2. A configured Compass redirect may route directly to the brand PLP.
3. Otherwise Compass selects the brand profile and queries TypeSense with a high brand weight or exact brand filter.
4. TypeSense returns grouped canonical products and brand-relevant facets.
5. Compass applies active rules, eligibility, and response headers/brand entry points.
```

Redirects remain Compass policy; TypeSense is not required when direct navigation is the intended result.

### A.3 Category, ingredient, and concern search

Examples: `face wash`, `sunscreen`, `niacinamide`, `retinol serum`, `pigmentation`, `hair fall`, `dark spots`.

```text theme={null}
1. Compass maps known terms and aliases to canonical category, ingredient, concern, and product-type IDs.
2. It selects the corresponding TypeSense ranking profile and facet configuration.
3. TypeSense retrieves/ranks against product title, category, concern, ingredient, and product-type fields; it computes facets and filter counts.
4. Compass applies claim/Rx policy, merchandising, page entry points, and final result shaping.
```

For example, `retinol serum` resolves to the canonical retinol ingredient plus serum product type before TypeSense retrieves candidates. This is more reliable than text matching raw descriptions alone.

### A.4 Hinglish and mixed-language search

Examples: `muhase ki cream`, `balo ka jhadna`, `jhaiyan`, `oily skin ke liye sunscreen`.

Hinglish starts as a governed Compass alias dictionary, not as a vector-only feature:

```text theme={null}
1. Compass normalizes the mixed-language query and recognizes protected terms.
2. It matches approved phrase/token aliases, for example:
   muhase -> acne
   balo ka jhadna -> hair_fall
   jhaiyan -> pigmentation
3. Compass extracts multiple structured signals where present:
   oily skin ke liye sunscreen -> sunscreen + oily skin type.
4. Compass builds a normal structured TypeSense lexical request with filters/preferences.
5. TypeSense retrieves, filters, facets, groups, and ranks products.
```

Every approved alias is explainable, testable, versioned, and reviewable from the control plane. Multilingual semantic retrieval can augment low-confidence cases later, but does not replace the governed dictionary.

### A.5 Typo-tolerant and attribute-rich search

Examples: `sunscreem spf 50`, `niacinemad`, `spf 50 sunscreen for oily skin under 700`, `10% niacinamide serum`.

```text theme={null}
1. Compass recognizes protected terms and approved aliases before typo behavior.
2. Compass extracts high-confidence attributes:
   SPF 50 -> spf >= 50
   under 700 -> price <= 700
   oily skin -> skin type = oily
   10% niacinamide -> ingredient/strength.
3. Explicit high-confidence values become TypeSense filters; ambiguous language becomes a ranking preference rather than a hard filter.
4. TypeSense performs native typo-tolerant matching, field-weighted ranking, numeric/Boolean filtering, grouping, and facets.
5. Compass decides whether to display a correction, records the interpretation, and enters the fallback ladder only if quality remains poor.
```

TypeSense performs typo matching. Compass decides protected-term policy, structured extraction, display/rewrite behavior, and the query plan.

### A.6 Autosuggest

Examples: typed prefixes such as `sun`, `acne u`, and `naci`.

```text theme={null}
1. Client debounces for 150-250 ms and cancels stale prefix requests.
2. Compass applies the same normalization, protected terms, aliases, and safety policy as full search.
3. Compass queries TypeSense for grouped product, brand, category, concern, ingredient, and approved popular-query candidates.
4. Compass applies pinned/blocked/curated suggestion rules and quality filters.
5. Client renders destination-aware groups; visible impressions and taps are attributed.
```

TypeSense `popular_queries` supplies candidates only. Compass validates quality and safety before a suggestion reaches users.

### A.7 Browse and PLP flow

Examples: category, brand, concern, ingredient, and campaign pages.

```text theme={null}
1. Compass/Tetris resolves the page definition: inclusion criteria, profile, allowed sorts, facets, quick filters, content slots, and page rules.
2. Compass builds a TypeSense request with canonical inclusion and eligibility filters; q may be * for pure browse.
3. TypeSense retrieves, filters, ranks, groups, paginates, and returns facet counts.
4. Compass/Tetris adds CMS content, badges, consult modules, and signed discovery context.
5. Client records page exposure and interactions.
```

### A.8 Semantic and hybrid search

Examples: `cream for dark spots`, `something for oily acne-prone skin`, `moisturizer that does not feel sticky`.

Semantic retrieval is introduced only after lexical search, controlled Hinglish aliases, relevance fixtures, and baseline metrics are dependable.

```text theme={null}
1. Compass runs deterministic query understanding first.
2. It selects a hybrid profile only for natural-language/weak-lexical query groups.
3. Compass obtains a query embedding and TypeSense runs lexical plus vector retrieval/ranking.
4. TypeSense combines text relevance and vector similarity while preserving filters/grouping.
5. Compass applies rules/safety/exact-match protection and records hybrid usage.
```

Exact product, brand, ingredient, and approved Hinglish-alias searches remain lexical dominant. Prescription/medical queries never use semantic retrieval to create a promotional shelf.

### A.9 Redirect, medical, and fallback flow

```text theme={null}
1. Compass checks active redirect rules before product retrieval.
2. It detects medical/Rx-sensitive or deliberately out-of-scope intent before generic fallback.
3. For a normal search, it runs the primary TypeSense query plan.
4. If results are empty or poor: typo-safe correction -> relax low-confidence modifiers -> hybrid fallback when enabled -> closest honest category/concern response.
5. For uncarried medical demand or prescription molecules, Compass routes to consultation/clinical policy rather than unrelated merchandising.
6. Compass records every fallback/redirect/consultation decision in the explain record and PostHog event.
```

### A.10 Recommendation retrieval flow

Recommendations are not text search, but use the same ownership boundary:

```text theme={null}
1. Compass receives anchor product, widget type, surface, and allowed session/profile context.
2. It loads the widget strategy and candidate source: curated, catalogue relationships, co-purchase, model later.
3. TypeSense retrieves/ranks eligible product candidates by controlled attributes where applicable.
4. Compass applies routine-step, ingredient-conflict, Rx, stock, anchor-product, and widget-specific exclusions.
5. Compass applies widget rules/pins/fallback and Tetris renders the widget.
```

Generic semantic similarity is never the only recommendation rule. Routine compatibility and safety remain deterministic.

### A.11 Unified decision tree

```text theme={null}
Incoming discovery request
  -> configured redirect?             -> Compass destination
  -> medical/Rx-sensitive intent?     -> Compass clinical/safety route
  -> exact product or brand?          -> lexical-dominant TypeSense profile
  -> known category/ingredient/concern? -> structured lexical TypeSense profile
  -> approved Hinglish alias?         -> Compass mapping + structured lexical profile
  -> explicit attributes?             -> Compass extraction + TypeSense filters/preferences
  -> natural-language/weak lexical?   -> hybrid TypeSense profile when enabled
  -> still poor?                      -> Compass fallback ladder / honest alternative
```

### A.12 Responsibility summary

| TypeSense executes                                    | Compass owns                                                                      |
| ----------------------------------------------------- | --------------------------------------------------------------------------------- |
| Text and typo-tolerant retrieval                      | Query interpretation and protected terms                                          |
| Field weighting, filtering, facets, grouping, sorting | Intent, aliases/Hinglish governance, and structured extraction                    |
| Lexical/vector hybrid ranking                         | Ranking-profile/rule selection and policy                                         |
| Native pins/hides/overrides                           | Scheduling, approval, audit, preview, and rollback                                |
| Native search analytics aggregation                   | Safety, Rx/consult routing, external eligibility, attribution, and explainability |

## 10. Search Low-Level Design

### 10.1 Query understanding pipeline

```text theme={null}
raw query
  -> Unicode/whitespace/case normalization
  -> protected-term recognition
  -> approved synonym and alias normalization
  -> typo candidate generation with protected-term guard
  -> intent classification
  -> structured detail extraction
  -> query plan
```

Implement the initial pipeline deterministically using the attribute dictionary and approved rules. For example:

* `vit c` maps to the approved `vitamin c` alias.
* `SPF 50` extracts an `spf >= 50` candidate filter/preference.
* `under 500` extracts `price.maximum = 500`.
* `for oily skin` extracts an approved skin-type signal.
* Protected terms are identified before typo correction and never auto-rewritten into generic words.

Intent classification should begin with transparent signals rather than a black-box model:

| Intent      | Initial evidence                                         |
| ----------- | -------------------------------------------------------- |
| Product     | Exact/near-exact title or product-family match           |
| Brand       | Brand dictionary or protected-term match                 |
| Category    | Category/alias dictionary match                          |
| Concern     | Concern dictionary and approved aliases                  |
| Ingredient  | Ingredient dictionary and approved aliases               |
| Combination | Multiple recognized entity types or extracted attributes |

Log confidence and evidence. Ambiguous queries may retrieve multiple intent routes; do not force false certainty.

### 10.2 Candidate retrieval

Typesense collection fields should include separate searchable fields and facet fields. A representative collection configuration is:

```json theme={null}
{
  "name": "discovery_products_v1",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "title", "type": "string" },
    { "name": "brandName", "type": "string", "facet": true },
    { "name": "categoryNames", "type": "string[]", "facet": true },
    { "name": "concernNames", "type": "string[]", "facet": true },
    { "name": "ingredientNames", "type": "string[]", "facet": true },
    { "name": "productType", "type": "string", "facet": true },
    { "name": "formats", "type": "string[]", "facet": true },
    { "name": "skinHairTypes", "type": "string[]", "facet": true },
    { "name": "spf", "type": "int32", "facet": true, "optional": true },
    { "name": "searchableDescription", "type": "string" },
    { "name": "listable", "type": "bool", "facet": true },
    { "name": "purchasable", "type": "bool", "facet": true },
    { "name": "inStock", "type": "bool", "facet": true },
    { "name": "rxClassification", "type": "string", "facet": true },
    { "name": "priceMinimum", "type": "float", "facet": true },
    { "name": "rating", "type": "float", "optional": true },
    { "name": "launchedAtTimestamp", "type": "int64", "optional": true }
  ],
  "default_sorting_field": "popularityScore"
}
```

The exact Typesense schema should be prototyped against real data. The key point is to separate:

* Full-text fields: title, brand, categories, concerns, ingredients, product type, description.
* Facet/filter fields: stock, listability, pricing, formats, SPF, safety, brands, categories.
* Sort fields: price, rating, newness, approved popularity scores.
* Stable identity fields: product family ID, source version, canonical taxonomy IDs.

### 10.3 Eligibility policy

Eligibility runs before final ranking and must be used by search, browse, recommendations, and future sponsorship.

```ts theme={null}
function isEligible(product: DiscoveryProduct, context: DiscoveryContext): EligibilityDecision {
  if (!product.listable) return { eligible: false, reason: "not_listable" };
  if (!product.purchasable) return { eligible: false, reason: "not_purchasable" };
  if (context.widget && product.rxClassification !== "none") {
    return { eligible: false, reason: "rx_excluded_from_widget" };
  }
  if (context.promotion && product.rxClassification !== "none") {
    return { eligible: false, reason: "rx_excluded_from_promotion" };
  }
  if (context.excludeOutOfStock && !product.inStock) {
    return { eligible: false, reason: "out_of_stock" };
  }
  return { eligible: true };
}
```

For efficiency, the broadest eligibility filters are pushed to Typesense filters. The application policy runs again over selected candidates as defense in depth and supplies clear exclusion reasons to the inspector.

### 10.4 Ranking pipeline

The exact layer order is a platform invariant:

```text theme={null}
1. Eligibility and safety
2. Relevance
3. Behavioral quality (initially disabled or tightly bounded)
4. Declared business boosts
5. Merchandising boosts and buries
6. Pins
```

Initial score design:

```text theme={null}
relevanceScore =
  1.00 * titleExactOrNearExact
  0.85 * brandMatch
  0.70 * categoryMatch
  0.70 * concernMatch
  0.70 * ingredientMatch
  0.55 * productTypeMatch
  0.20 * descriptionMatch
  0.30 * extractedAttributeFit

finalOrganicScore = relevanceScore
  + declaredBusinessBoosts
  + boundedBehavioralScore
  + merchandisingBoostOrBury
```

These numbers are placeholders, not production truth. Store them in a versioned `RankingProfile`, evaluate them against golden queries, and validate material changes through interleaving or A/B testing.

Pins are applied after organic scoring. A pin must be validated before publishing:

* Product exists and is currently eligible for the target scope.
* Product does not conflict with an existing pin at the same position.
* Safety restrictions are never overridden.
* Expiry date and audit reason are present where required.

### 10.5 Fallback ladder

For zero or materially poor results:

1. Run the protected-term-safe typo correction query.
2. Run a relaxed query plan by dropping low-confidence modifiers.
3. Run a hybrid/semantic candidate source only when introduced.
4. Route to the nearest relevant category/concern page with an honest message.
5. Return an explicit zero-result response plus supported alternative actions.

Each attempt is recorded in the search event. Never silently substitute an unrelated shelf.

### 10.6 Hinglish and multilingual support

Start with a governed alias dictionary sourced from real query traffic. Examples include mapped concern terms such as `muhase`, `balo ka jhadna`, and `jhaiyan`. Store language/locale and confidence on aliases. Do not add broad transliteration or LLM rewriting until query-health data shows the coverage need and quality review capacity exists.

## 11. Autosuggest Design

Autosuggest reuses query normalization, protected terms, candidate filtering, and safety policy from search. It must not be less tolerant than the full search endpoint.

### 11.1 Suggest endpoint behavior

For each prefix, return grouped suggestions:

```text theme={null}
Products: product image, title, brand, price, destination PDP
Brands: brand title, destination brand PLP
Categories: category title, destination category PLP
Concerns: concern title, destination concern PLP
Ingredients: ingredient title, destination ingredient PLP
Popular searches: approved query, destination search/PLP
```

The response includes `suggestionId`, type, rank, source, and destination so impressions and taps are attributable.

### 11.2 Suggestion quality controls

* Reject candidates that lead to no/poor results based on an offline quality job.
* Deduplicate near-identical strings.
* Preserve product family and brand intent when confidence is high.
* Exclude Rx items from promotion-like suggestion positions; allow direct matching where policy permits.
* Apply merchandising pins/blocks/curated suggestions only after safety and quality checks.
* Before typing, surface privacy-safe recent searches, approved trending queries, and configured seasonal shortcuts.

## 12. Browse and PLP Design

Browse pages are query templates over the same product index, not separate product databases.

### 12.1 Page definition

```ts theme={null}
type PageDefinition = {
  id: string;
  slug: string;
  pageType: "category" | "brand" | "concern" | "ingredient" | "campaign" | "combination";
  title: string;
  description?: string;
  inclusionQuery: StructuredProductFilter;
  rankingProfileId: string;
  filterConfigurationId: string;
  allowedSorts: string[];
  contentSlots: ContentSlotDefinition[];
  seo: { canonicalUrl: string; indexPolicy: "index" | "noindex"; redirectsFrom: string[] };
};
```

This supports standard categories, brand shelves, cross-category concern/ingredient pages, and campaigns without custom code per page.

### 12.2 Filters and sort

`FilterConfiguration` is data associated with page type/page/category. It controls:

* Visible facets and their display order.
* Pinned facet values and quick filters.
* Search-inside-filter support for high-cardinality fields such as brands.
* Sort options and default ranking profile.
* Maximum allowed selections and zero-result recovery behavior.

Multiple values in a facet use OR; filters across facets use AND. Applied filters are returned as removable chips. A sort/filter change resets pagination to page 1 server-side.

### 12.3 SEO requirements

* Use a stable page ID and stable canonical URL independent of display-name changes.
* Store redirects when products, brands, or pages retire.
* Define canonical/index/noindex policy for filter combinations before releasing filtered pages to crawlers.
* Capture organic traffic and indexability metrics before and after migration.
* Do not change URL structures as an incidental side effect of changing the search provider.

## 13. Recommendations Design

Recommendations start in Phase 3 because they depend on reliable catalogue relationships and attribution. They do not require a separate model-serving platform initially.

### 13.1 Shared widget pipeline

```text theme={null}
Widget request
  -> load strategy by widget + surface
  -> gather candidates (curated, relationship data, co-purchase, model later)
  -> universal eligibility/Rx filters
  -> widget-specific hard filters
  -> routine safety and complementarity checks
  -> ranking and business rules
  -> merchandise pins/hides/replace rules
  -> fallback chain
  -> emit impression context
```

### 13.2 Candidate sources in delivery order

1. Curated lists and catalogue relationships.
2. Product similarity based on controlled attributes.
3. Co-purchase signals after event/order attribution is validated.
4. Bounded model output where data coverage justifies it.

Never use a language model as the primary candidate generator. It may enrich sparse metadata under human review, but product safety and routine rules remain deterministic.

### 13.3 Routine intelligence

Create versioned relationship/rule tables:

```text theme={null}
product_relationships
  anchor_product_id, related_product_id, relationship_type, confidence, source

routine_step_rules
  anchor_step, allowed_complement_steps, prohibited_same_steps

ingredient_conflicts
  ingredient_a, ingredient_b, severity, rationale, applies_to

safety_cautions
  attribute_or_ingredient, caution_type, policy
```

Widget hard rules must ensure:

* Anchor product is excluded.
* Out-of-stock and Rx products are excluded where required.
* Same-step/duplicate active-ingredient suggestions are excluded for routine widgets.
* Known ingredient conflicts and declared cautions are respected.
* Category mismatch constraints are enforced per widget.
* Short, high-quality lists are preferable to padded weak lists.

### 13.4 Recommendation quality gate

Maintain reviewed fixture sets with approved and prohibited recommendations per anchor product. Every widget strategy change runs against them before rollout. Online click/revenue metrics complement, but do not replace, this review.

## 14. Merchandising Control Plane

### 14.1 Initial operator workflows

The initial console should focus on the highest-value, lowest-risk capabilities:

1. Search query inspection.
2. Protected-term and synonym management.
3. Query redirects.
4. Search pins, hides, boosts, and buries.
5. Ranking-profile assignment.
6. Preview and publish/revert.
7. Rule audit history.

The UI should lead with the affected query/page preview, not a generic JSON rule builder. Power-user bulk tools, campaign bundles, and cross-surface workflows come later.

### 14.2 Publishing workflow

```text theme={null}
Draft -> validate -> preview using live/safe snapshot -> approval if required
  -> publish version -> invalidate config cache -> active within minutes
  -> monitor impact -> expire or revert
```

Validation checks include syntax, target existence, eligibility, overlapping pin conflicts, schedule conflicts, Rx/promotion restrictions, and required audit reason.

### 14.3 Configuration caching

* Store authoritative configurations in PostgreSQL.
* Publish an immutable configuration version to Redis/cache on activation.
* Discovery API reads the active version once per request or with a short TTL.
* Attach `config_version` to every response/event.
* Cache invalidation must be explicit and observable. A rule that saves but does not take effect is a production incident.

### 14.4 Compass source of truth and Typesense compilation

The merchandising panel writes only to Compass admin APIs and Compass PostgreSQL. It does not treat Typesense as the source of truth. Compass owns rule definition, scheduling, audit history, approval, preview, rollback, and conflict validation; Typesense is an optimized execution target for rules it can apply natively.

```text theme={null}
Merchandising panel
  -> Compass admin API
  -> PostgreSQL draft/audit/schedule/approval state
  -> preview against current Typesense data
  -> publish immutable Compass configuration version
  -> sync compatible rule to Typesense OR apply it as request-time parameters
  -> Compass validates final result eligibility and returns response
```

The panel displays TypeSense synchronization state for every compiled rule. A `synced` state is necessary for a native override to execute, but Compass remains the emergency authority: disabling a rule stops Compass from applying request-time behavior immediately and queues removal of the Typesense resource.

| Rule/action                                  | Source of truth            | Serving execution                                                             |
| -------------------------------------------- | -------------------------- | ----------------------------------------------------------------------------- |
| Exact query pin                              | Compass rule               | TypeSense override or request-time `pinned_hits`.                             |
| Exact query hide                             | Compass rule               | TypeSense override or request-time `hidden_hits`.                             |
| Approved synonym                             | Compass synonym set        | Synchronize approved synonym to TypeSense.                                    |
| Protected term                               | Compass                    | Compass query policy before correction/retrieval.                             |
| Redirect                                     | Compass redirect rule      | Compass returns destination without product retrieval.                        |
| Simple query/page boost or bury              | Compass rule               | Compass builds TypeSense ranking/evaluation parameters.                       |
| Complex segmented boost/bury                 | Compass rule               | Compass over-fetches eligible candidates and performs a bounded final rerank. |
| Rx/listing/availability restriction          | Compass eligibility policy | Push broad filters into TypeSense and validate again in Compass.              |
| Schedule, campaign, approval, preview, audit | Compass                    | Never delegated to TypeSense.                                                 |

Do not persist a campaign boost as a permanent product popularity/document value. It becomes stale, is hard to reverse, and can leak beyond its intended scope. Pins/hides map cleanly to native TypeSense controls; boosts/buries are query-context policy and must remain explainable in Compass.

### 14.5 Rule publication, synchronization, and runtime behavior

Publishing a rule follows this sequence:

```text theme={null}
1. Merchandiser creates a draft in the panel.
2. Compass validates target existence, eligibility, dates, conflicts, permissions, and Rx/promotion restrictions.
3. Compass previews current results with the proposed rule.
4. Required approver accepts the draft.
5. Compass activates an immutable configuration version and invalidates the active configuration cache.
6. A sync worker creates/updates the corresponding TypeSense synonym or override when supported.
7. The worker records TypeSense resource ID, rule version, status, and failures in typesense_rule_sync.
8. Compass records applied rule IDs/configuration version in search events and explain records.
```

For a simple pin, the resulting request flow is:

```text theme={null}
User searches "sunscreen"
  -> Compass loads matching active rules from its cached configuration version
  -> Compass applies safety/listability filters and sends native pin/override to TypeSense
  -> TypeSense retrieves/ranks and places the eligible pinned product
  -> Compass validates final eligibility, writes explain record, and returns response
```

For complex boosts/buries, Compass retrieves more candidates than the visible page size, applies only bounded adjustments to eligible/relevant candidates, and returns the visible page. This final rerank is deliberately narrow; Compass must not become a slow general-purpose search engine.

Rule precedence remains fixed:

```text theme={null}
1. Eligibility and safety
2. Organic relevance and bounded behavioral/business signals
3. Merchandising boosts/buries
4. Pins
5. Final Compass safety validation
```

No TypeSense override, query parameter, panel action, experiment, or campaign can resurrect an item that fails eligibility. Publish validation rejects conflicting pins for the same scope/position, invalid targets, overlapping ambiguous rules, and unauthorized global or Rx-sensitive changes.

The inspector shows both policy and engine evidence: Compass rule ID/version, TypeSense override/resource ID when used, score adjustment, selected position, excluded products and reasons, and active configuration/ranking profile versions.

## 15. Ranking Governance and Explainability

### 15.1 Ranking profile example

```json theme={null}
{
  "name": "search-default",
  "version": 3,
  "availability": { "outOfStock": "demote" },
  "textWeights": {
    "title": 10,
    "brand": 9,
    "category": 7,
    "concern": 7,
    "ingredient": 7,
    "productType": 6,
    "description": 2
  },
  "businessSignals": {
    "rating": { "enabled": true, "weight": 0.10 },
    "newness": { "enabled": true, "weight": 0.05 },
    "stockDepth": { "enabled": false, "weight": 0 }
  },
  "behavioralSignals": { "enabled": false, "maxInfluence": 0.10 },
  "allowedSorts": ["featured", "price_asc", "price_desc", "newest", "rating"]
}
```

Profiles are versioned, previewed against golden cases, and staged. Product owns assignments; creating/changing scoring logic requires engineering review and experiments for material changes.

### 15.2 Explain record

For every `searchId`, persist a short-lived structured record:

```json theme={null}
{
  "searchId": "uuid",
  "query": { "raw": "sunscreem", "normalized": "sunscreen", "intent": "category" },
  "queryPlan": { "filters": ["listable:true"], "fallbackStep": "typo_correction" },
  "rules": ["rule_123"],
  "results": [
    {
      "productId": "product_1",
      "position": 1,
      "eligibility": "eligible",
      "score": {
        "relevance": 0.93,
        "business": 0.02,
        "behavioral": 0,
        "merchandising": 0.1
      },
      "pin": null,
      "reasons": ["title_match", "concern_match", "merchandise_boost"]
    }
  ]
}
```

This powers the query inspector, support escalation, ranking review, and incident diagnosis. Its retention period should balance operational value and privacy; do not expose raw personal queries widely.

## 16. Catalogue Ingestion and Quality

### 16.1 Delta ingestion

```text theme={null}
Catalogue change
  -> SNS event -> SQS queue
  -> SQS queue
  -> indexer validates event/version
  -> fetch or receive canonical product payload
  -> transform into DiscoveryProduct
  -> validate attribute dictionary and safety fields
  -> upsert/delete Typesense document
  -> record index status and latency
```

Use an idempotency key composed of product ID and source version. Out-of-order events must not overwrite newer product state. Put invalid payloads into a dead-letter queue with clear error reason and owner routing.

### 16.2 Freshness objectives

| Data type             | Target propagation                     | Notes                                        |
| --------------------- | -------------------------------------- | -------------------------------------------- |
| Stock and listability | Near real time, target under 5 minutes | Important for user trust and eligibility.    |
| Price                 | Near real time, target under 5 minutes | Search/PLP price must not conflict with PDP. |
| Core product content  | Minutes to hours                       | Title, images, descriptions, attributes.     |
| Full rebuild          | Recovery only                          | Never the normal update mechanism.           |

### 16.3 Quality gates

Define coverage requirements by category. Example:

```text theme={null}
Sunscreen: SPF 100%, format >= 95%, skin type >= 90%
Serum: hero ingredient >= 95%, concern >= 90%, format >= 95%
Haircare: product type >= 100%, hair concern >= 90%
```

New or edited products that fail required coverage are flagged for catalogue review. Whether they are blocked from a particular facet or from all discovery surfaces should be policy-driven and visible.

### 16.4 Assisted enrichment

Enrichment proposals can be generated from names, descriptions, and ingredient lists, but they remain `proposed` until a catalogue operator approves them. Record source, confidence, reviewer, and timestamp. Never silently alter controlled attributes.

## 17. Analytics, Reporting, and Experiments

### 17.1 Phase 1 analytics architecture

Phase 1 does not require a new data warehouse. Use Typesense's native analytics for search-engine feedback, PostHog for product and revenue analytics, Compass PostgreSQL for operational/search explainability, and Vector/Medusa/Orderbox for authoritative commerce facts.

```text theme={null}
Compass search -> Typesense native analytics
  -> popular queries, no-hit queries, verified document counters, optional logs

Compass/Tetris/Vector/Medusa server events -> PostHog server events
Web/mobile interaction SDK -> first-party event ingestion -> PostHog
  -> funnels, cross-surface paths, revenue, cohorts, feature flags and experiments

Compass PostgreSQL
  -> result positions, ranking/profile/rule versions, fallback, audit and explain records

Vector/Medusa/Orderbox
  -> authoritative cart, completed-order, refund/cancellation facts and line-item attribution
```

PostHog is sufficient for Phase 1 dashboards when server-side order events carry persisted discovery attribution. Typesense and PostHog must not be treated as the commerce system of record; reconciliation remains against Medusa/Orderbox.

### 17.2 Typesense native analytics

Enable Typesense analytics only on the self-hosted production/search cluster and configure four rule families:

| Rule family             | Typesense rule type                        | Phase 1 use                                                                     | Guardrails                                                                                         |
| ----------------------- | ------------------------------------------ | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Successful query demand | `popular_queries`                          | Popular searches and query-suggestion candidates                                | Separate actual search submissions from typeahead; Typesense aggregates after a four-second pause. |
| Raw engine no-hits      | `nohits_queries`                           | Candidate synonym, catalogue-gap, redirect, and consultation-routing review     | This measures engine-level no hits, not whether Compass's fallback ladder recovered the user.      |
| Verified engagement     | `counter` with `click`/`conversion` events | Small, controlled product popularity signal and fast relevance triage           | Forward only validated events; keep disabled as a major ranking signal initially.                  |
| Behavioral history      | `log`                                      | Future pseudonymous session/user personalization and recent-behavior inspection | Access controlled, retention reviewed, never the finance/reporting store.                          |

Typesense analytics supports `search`, `click`, `conversion`, and `visit` events. Each Compass request uses `analytics_tag` to distinguish the surface/platform, for example `search_results:web`, `search_results:android`, or `concern_plp:web`. Send a pseudonymous `X-TYPESENSE-USER-ID` derived from approved Sesh/Hodor identity policy rather than a raw customer identifier.

Use `filter_by` and `analytics_tag` metadata only for lightweight search segmentation. Keep ranking profile/version, rule IDs, experiment assignment, fallback outcome, exact positions, and safety decisions in Compass because Typesense does not model the full discovery policy.

Counter events are forwarded only after first-party token validation or authoritative cart/order handling. A suggested initial weighting is click `+1`, add-to-cart `+3`, purchase `+10`, but the resulting value is only a bounded tie-breaker within eligible/relevant results. It must not override relevance, safety, pins, or merchandise rules. Position bias, fraud, stale stock, and popularity feedback loops require evidence thresholds, rate caps, decay, holdout evaluation, and a kill switch before behavioral ranking is enabled.

### 17.3 PostHog analytics and attribution

PostHog is the Phase 1 cross-surface analytics layer. It receives validated client interaction events and server-side commerce events, and answers:

* Search-to-PDP, search-to-cart, and search-to-order funnels.
* Revenue by query, intent, platform, suggestion type, ranking profile, rule/campaign, fallback path, and experiment variant.
* Cross-session conversion after Sesh/Hodor identity linking, under the approved privacy policy.
* Comparison of search, PLP, widget, and direct PDP journeys.
* Filter/sort/suggestion behavior, cohorts, feature flags, and experiments.

`order_completed` is emitted server-side by Vector, Medusa, or Orderbox and includes line-item revenue plus the discovery attribution persisted at cart-add time. This makes PostHog suitable for Phase 1 search-to-order revenue dashboards; browser-reported purchases are never authoritative.

### 17.4 Future warehouse plan

Introduce a data warehouse only after a concrete need exceeds PostHog, Typesense, Compass, and commerce reporting. Triggers include:

* Finance-grade reconciliation of revenue, refunds, taxes, and order lines directly against commerce data.
* Large/custom SQL joins across discovery, catalogue, inventory, CRM, marketing, and orders.
* Long-term immutable raw-event retention or lower-cost high-volume analytical storage.
* Offline training for position-debiased behavioral ranking/learning-to-rank.
* Statistical analysis or experiment methods beyond PostHog.
* PostHog query performance, retention, or cost becoming a product constraint.

When triggered, use S3 raw events, Athena or the existing company warehouse, and dbt/SQL transformations. The initial future models are:

* `fact_discovery_requests`
* `fact_search_results`
* `fact_discovery_interactions`
* `fact_discovery_attribution`
* `fact_merchandising_rule_exposure`
* `fact_widget_exposure`
* `dim_query_interpretation`
* `dim_ranking_profile_version`
* `dim_configuration_version`

### 17.5 Future owned analytics stack

An owned analytics stack is a future option, not a replacement for TypeSense analytics or PostHog at launch. Build it only when the triggers above are accepted and an owner is assigned. The target responsibilities are:

```text theme={null}
First-party ingestion / server outbox
  -> durable raw immutable event storage
  -> schema validation and deduplication
  -> curated discovery and commerce models
  -> SQL/BI, finance reconciliation, offline experimentation, and model training
```

Recommended staged implementation:

| Stage                    | Add                                                                                                               | Outcome                                                                           |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| 1. Dual-write raw events | First-party endpoint and server outboxes additionally write partitioned raw events to S3 through SNS/SQS workers  | Vendor-independent recovery/replay without changing PostHog dashboards.           |
| 2. Curated models        | Glue/Athena or the company-standard warehouse plus dbt/SQL transformations                                        | Reconciled search, interaction, cart, order, refund, rule, and experiment models. |
| 3. BI and governance     | Versioned metrics definitions, access controls, retention/deletion jobs, data-quality monitors, and BI dashboards | Product/finance reports are reproducible outside PostHog.                         |
| 4. Learning data         | Position-aware exposure/click training sets, debiasing, holdouts, and offline evaluation datasets                 | Safe behavioral ranking and future learning-to-rank development.                  |

The owned stack remains downstream and additive during migration. Continue sending PostHog events until its dashboards and experiments have validated against the owned models. Do not make TypeSense analytics the raw event source: it is optimized for search-native aggregation and limited log retrieval, not immutable organization-wide event retention.

Required future raw-event fields include the portable envelope, destination delivery status, producer/service version, schema version, deduplication key, event and receipt timestamps, discovery IDs/positions, configuration/profile/rule versions, and persisted cart/order line attribution. Raw sensitive query/health data requires separate access controls and retention/deletion policy.

### 17.6 Event ownership and source of truth

Use a hybrid event model. The browser/mobile app reports interaction only visible in the interface; backend services report the discovery response and business facts they authoritatively know. This avoids trusting clients with result positions, ranking versions, cart state, or completed orders.

| Event class                                                                                      | Primary producer       | Reason                                                      |
| ------------------------------------------------------------------------------------------------ | ---------------------- | ----------------------------------------------------------- |
| Search executed, returned results/positions, query interpretation, ranking/rule/fallback details | Compass                | Compass generated the exact result list and response.       |
| PLP/widget payload served, strategy, fallback, slot positions                                    | Compass/Tetris         | The server assembled the content and knows its version.     |
| Result/suggestion click, filter/sort change, visible-card/widget impression                      | Web/mobile SDK         | Only the client knows what the user saw or interacted with. |
| Cart mutation and line-item attribution                                                          | Vector/Medusa          | Cart state is a server-side commerce fact.                  |
| Purchase/order completion                                                                        | Medusa/Vector/Orderbox | The order system is authoritative.                          |

Analytics delivery is always asynchronous. No search, click navigation, cart, or checkout response waits for PostHog, S3, SNS, or a warehouse job.

### 17.7 Identity and discovery context

Each event contains the identities needed for reliable joining:

| Identifier             | Owner          | Purpose                                                     |
| ---------------------- | -------------- | ----------------------------------------------------------- |
| `anonymous_id`         | Sesh/client    | Stable browser or app-installation identity before sign-in. |
| `session_id`           | Sesh           | Groups activity within one visit/session.                   |
| `user_id`              | Hodor          | Optional authenticated customer identity.                   |
| `request_id`           | Compass        | Correlates one service request across logs and services.    |
| `search_id`            | Compass        | Identifies one executed search and exact returned list.     |
| `page_view_id`         | Compass/Tetris | Identifies one rendered PLP result set.                     |
| `widget_impression_id` | Compass/Tetris | Identifies a recommendation payload and strategy.           |
| `event_id`             | Producer       | Makes retries and deduplication safe.                       |

When a visitor signs in, preserve the anonymous activity and record an identity-link event. Do not rewrite historical records destructively.

Compass returns an opaque signed `discovery_context_token` for every result card. The token binds the source, product, variant, position, session, issue time, and expiry. Clients pass it unchanged to click and add-to-cart flows; they never construct attribution fields themselves.

```json theme={null}
{
  "version": 1,
  "source_type": "search",
  "source_id": "srch_01...",
  "product_id": "prd_100",
  "variant_id": "var_100_default",
  "position": 1,
  "session_id": "ses_01...",
  "issued_at": "2026-08-03T10:00:00Z",
  "expires_at": "2026-08-10T10:00:00Z"
}
```

The token is signed and validated server-side. It prevents clients from falsely assigning a product to a more favorable query or position, which protects attribution and any future behavioral ranking.

### 17.8 Shared web and mobile SDKs

Create a shared, versioned contract package and platform wrappers rather than scattering direct analytics calls through clients:

```text theme={null}
discovery-event-contracts
  event names, JSON schemas/types, schema versions, redaction rules

discovery-web-analytics
  JavaScript/TypeScript wrapper, batching, sendBeacon/keepalive delivery

discovery-mobile-analytics
  native/mobile wrapper, encrypted local queue, retry/flush behavior
```

Client code calls typed methods such as `trackSearchResultClick` and `trackSuggestionClick`, not arbitrary event strings. Both clients use the same event names and required properties.

Web behavior:

* Send navigation-adjacent events with `navigator.sendBeacon`, falling back to `fetch` with `keepalive: true`.
* Queue and batch lower-priority interaction events.
* Flush on `visibilitychange` and page lifecycle transitions.
* Debounce autosuggest requests by 150-250 ms and cancel stale requests.
* Emit viewport impressions only after a card/widget meets the agreed visibility threshold, for example 50% visible for 500 ms.

Mobile behavior:

* Persist an encrypted local event queue.
* Batch on normal connectivity and flush on foreground/interval events.
* Use idempotent `event_id` values so retries are safe.
* Never delay navigation or rendering for analytics delivery.

### 17.9 First-party client event ingestion

Web and mobile clients submit interaction events to a first-party endpoint, initially owned by Compass or the existing analytics edge:

```text theme={null}
POST /v1/events/discovery
```

```text theme={null}
Web/mobile SDK
  -> first-party event endpoint
  -> schema/token validation, rate limiting, server timestamp enrichment
  -> PostHog delivery
  -> optional Typesense verified click/visit event
```

Do not use direct browser/app-to-PostHog delivery as the only event path. A first-party endpoint provides validation, consistent schema enforcement, sensitive-data redaction, resilience against ad blockers, and safe forwarding to both PostHog and Typesense. PostHog remains the Phase 1 analytics layer; commerce reconciliation and operational attribution remain owned by server systems.

### 17.10 End-to-end search-to-purchase lifecycle

```text theme={null}
1. Sesh creates/resumes anonymous_id and session_id; Hodor supplies user_id after sign-in.
2. Web/mobile requests search from Compass with session context.
3. Compass creates search_id, records the full server-side search exposure, and returns results plus signed result tokens.
4. Client renders results. It emits visible impressions under viewport rules and sends click/filter/sort events through the SDK.
5. A product click navigates immediately; SDK delivery runs independently and token validation resolves source and position.
6. Add-to-cart sends the selected product/variant and context token to Vector/Medusa.
7. Vector/Medusa validates the token, persists primary discovery attribution on the cart line, and emits an authoritative add-to-cart event.
8. On payment/order completion, Medusa/Vector/Orderbox emits the authoritative purchase event using persisted line-item attribution.
9. PostHog joins discovery exposure/interactions with server-side cart/order events for Phase 1 dashboards and experiments; future warehouse models are introduced only when triggered.
```

Compass must record the complete result list and positions server-side. Client visibility events improve CTR analysis, but cannot replace the server exposure record.

For cart attribution, persist one primary source: the most recent valid discovery context directly preceding the cart add. Preserve an optional bounded list of prior valid in-session contexts as assists. Direct PDP/cart actions without a valid token remain explicitly unattributed; do not invent an origin.

### 17.11 Durable server event delivery

Server events use the existing SNS/SQS transport. Search events can be published asynchronously with metrics/alerts for enqueue failure. Cart and order events require stronger delivery semantics through a transactional outbox:

```text theme={null}
Commerce transaction
  -> persist cart/order mutation
  -> persist outbox row in the same database transaction
  -> background publisher sends the row to SNS/SQS
  -> mark the outbox row delivered after successful publish
```

This prevents losing attribution because an order commits while a separate event publication fails. Consumers deduplicate by `event_id`; event processing is at-least-once and idempotent.

### 17.12 Impression and event rules

| Event                                | Producer       | Rule                                                                    |
| ------------------------------------ | -------------- | ----------------------------------------------------------------------- |
| Search response/result-list exposure | Compass        | Record every returned list and all positions.                           |
| Visible product-card impression      | Client         | Record only after the agreed viewport threshold; batch and deduplicate. |
| Widget payload served                | Compass/Tetris | Record strategy, candidates, fallback, slots, and context.              |
| Visible widget impression            | Client         | Record when the widget becomes visible.                                 |
| Suggestion shown                     | Client         | Record when the dropdown is visible, with prefix/type/position.         |

Avoid one network request per visible card. Batch impression events and deduplicate within a bounded time window using `session_id + source_id + product_id + position`.

### 17.13 Event envelope and schema evolution

Every event uses a versioned envelope:

```json theme={null}
{
  "event_id": "uuid",
  "event_name": "search_result_clicked",
  "schema_version": 1,
  "occurred_at": "client-or-server timestamp",
  "received_at": "server timestamp",
  "producer": "web_sdk | ios_sdk | android_sdk | compass | vector | medusa",
  "anonymous_id": "optional",
  "session_id": "required where applicable",
  "user_id": "optional",
  "request_id": "optional",
  "properties": {}
}
```

Rules:

* Event names are immutable after production release.
* Add optional properties without changing the schema version.
* Breaking changes require a new event name or schema version.
* Validate client events at ingestion and server events before publication.
* Store both event occurrence and server-receipt timestamps; accept bounded client clock skew.
* Deduplicate by `event_id`.
* Redact or access-control personal health and prescription query data.

### 17.14 Metrics and data-quality controls

Track at least:

* Zero-result rate and fallback rescue rate.
* Search to PDP click rate, add-to-cart rate, and purchase conversion.
* Reformulation and dead-end exit rates.
* Suggestion acceptance and downstream quality by suggestion type.
* PLP click-through, conversion, filter use, and empty-grid rate.
* Widget impressions, clicks, add-to-cart, revenue, fallback rate, and session-level effect.
* Rule/campaign impact compared with baseline/control where possible.
* Attribute coverage, index freshness, latency, errors, and event completeness.

Additionally alert on:

* Result-click events without a valid source ID and position.
* Cart lines missing valid attribution when a discovery token was supplied.
* Purchase-event reconciliation differences against commerce orders.
* Event duplicates, invalid/expired tokens, client/server delivery lag, and queue/DLQ backlog.
* Compass search exposure counts that diverge from request logs.
* PostHog delivery failures and abnormal event-volume changes by client platform/version.

### 17.15 Experiment assignment

Experiment assignment should be deterministic using a stable hash of anonymous/user ID, experiment ID, and assignment version. Persist exposure with the assigned variant and configuration version. Avoid assignment changes mid-session.

Every experiment has:

* Hypothesis and owner.
* Surface and inclusion criteria.
* Primary metric and guardrails.
* Minimum duration/sample guidance.
* Variant configuration references.
* Stop/rollback conditions.
* Final decision and explanation.

Use interleaving for ranking-only comparisons when traffic is limited. Use A/B tests for broader UX/configuration changes. Never make a decision from widget click-through alone when total session conversion may be unchanged or harmed.

## 18. Safety, Privacy, and Security

### 18.1 Rx handling

Rx policy belongs in the shared eligibility layer:

* Rx products carry visible badges on applicable surfaces.
* They are excluded from recommendation widgets, promotional boosts, bestseller logic, discount badges, and sponsored placements.
* Exact search/category/concern discovery remains available according to regulatory policy.
* Prescription molecule searches route toward consult/prescription flows rather than promotional shelves.
* Rule validation, tests, and runtime policy all enforce this independently.

### 18.2 Claim-safe content

Content slot definitions include a `claimSensitive` flag and review status. Concern pages must have approved copy and a designated consultation module where a product grid could overstep clinical/compliance boundaries.

### 18.3 Personal data

* Use pseudonymous session/device IDs for anonymous discovery behavior.
* Do not place raw personal health data in the search index or client logs.
* Protect inspector access with role-based authorization and audit queries of sensitive information.
* Sensitive searches and Rx products do not appear in shared/recent-search surfaces without an explicit privacy policy.
* Profiles are consent-governed, deletable, and isolated from third-party engine data where required.

### 18.4 Security controls

* Service-to-service AWS IAM roles; no static production credentials.
* Secrets in AWS Secrets Manager.
* RDS encryption, S3 encryption, TLS in transit, private subnets for internal services.
* WAF/rate limiting on public endpoints.
* Admin role model: viewer, merchandiser, approver, product admin, engineer admin.
* Audit all privileged config changes, previews, publishes, reverts, and inspector access.

## 19. Observability and Operational Readiness

### 19.1 Service level objectives

Initial targets should be validated against real UX expectations:

| Metric                      | Initial target                                             |
| --------------------------- | ---------------------------------------------------------- |
| Search API p95 latency      | under 250 ms excluding client network                      |
| Autosuggest p95 latency     | under 150 ms excluding client network                      |
| Search availability         | 99.9% monthly                                              |
| Stock/price index freshness | under 5 minutes for 99% of updates                         |
| Event completeness          | over 99%                                                   |
| Click attribution validity  | over 99% of result clicks include search/page and position |
| Rx rule violations          | zero                                                       |
| Config publish propagation  | under 5 minutes, preferably seconds                        |

### 19.2 Alerts

Alert on:

* Search/suggest latency, error-rate, and timeout spikes.
* Zero-result and fallback-rate anomalies by query/category.
* Index queue age, DLQ size, index failures, and stale stock/price.
* Event gaps and duplicate-event spikes.
* Empty widget rate and serving-context integrity failures.
* Rule publication failures/cache propagation gaps.
* Safety-rule test failures and prohibited result exposure.
* Capacity headroom before planned sales events.

Every alert must have an owner, severity, runbook, and escalation policy.

### 19.3 Test strategy

| Test type              | Purpose                                                                  |
| ---------------------- | ------------------------------------------------------------------------ |
| Unit tests             | Query normalization, eligibility, rule evaluation, scoring functions.    |
| Contract tests         | Catalogue event and Discovery API schema compatibility.                  |
| Golden relevance tests | Important search/PLP/widget cases and safety assertions.                 |
| Integration tests      | Typesense indexing/filtering, PostgreSQL config loading, event emission. |
| End-to-end tests       | Search through UI, analytics attribution, admin preview/publish/revert.  |
| Load tests             | Diwali-scale expected traffic, cache behavior, queue pressure.           |
| Shadow tests           | Compare candidate engines/profiles with production-like traffic.         |
| Data quality tests     | Attribute coverage, product count reconciliation, stale data detection.  |

Golden cases should include direct product/brand matches, common typos, protected terms, concern/ingredient queries, zero-result recoveries, filters, variant grouping, pins/hides, and Rx exclusions.

## 20. Netcore-to-Typesense Migration and Rollout

Compass remains the stable client-facing facade throughout migration. Web and mobile clients continue calling the same Compass APIs; Compass normalizes provider responses, applies shared safety/attribution policy, and selects the provider internally.

```text theme={null}
Web / mobile -> Compass public API -> provider router
  -> Netcore adapter
  -> Typesense adapter
  -> shadow comparison recorder
  -> one canonical Compass response contract
```

### 20.1 Provider interface and modes

Compass defines a provider-neutral internal contract for search, autosuggest, and browse. Netcore and Typesense implement that contract; provider-specific response shapes never reach clients.

```text theme={null}
DiscoveryProvider
  search(request) -> normalized provider response
  suggest(request) -> normalized provider response
  browse(request) -> normalized provider response
```

Provider-routing modes:

| Mode                              | Customer response | Background behavior                                                | Intended use                                              |
| --------------------------------- | ----------------- | ------------------------------------------------------------------ | --------------------------------------------------------- |
| `netcore`                         | Netcore           | None                                                               | Default at migration start and emergency global fallback. |
| `shadow`                          | Netcore           | Typesense executes and comparison is recorded                      | Production parity validation with zero customer impact.   |
| `typesense`                       | Typesense         | Optional sampled Netcore comparison                                | Employees/internal QA and controlled tests.               |
| `typesense_with_netcore_fallback` | Typesense         | Netcore is used only after a Typesense failure or invalid response | Public canary and progressive rollout.                    |

Compass owns request validation, identity/session context, query/search IDs, Rx/listing/availability policy, response normalization, discovery context tokens, exposure events, and provider-comparison records. This guarantees behavior that must be consistent across engines.

### 20.2 Dual indexing and compatibility inventory

Keep the existing Netcore feed unchanged while a Compass index worker builds and maintains the Typesense index:

```text theme={null}
Atlas / Storehouse / Medusa change
  -> existing Netcore feed
  -> Compass Typesense index worker
  -> canonical DiscoveryProduct transform
  -> bulk Typesense upsert/delete with source_version
```

Every provider result must map to shared canonical identifiers before comparison or client response:

```text theme={null}
provider item ID -> canonical_product_id -> product_family_id -> display_variant_id
```

Maintain an explicit migration inventory. A Netcore behavior is not removed until its Compass/Typesense equivalent passes staging and shadow validation.

| Existing behavior                              | Target ownership/action                                                            |
| ---------------------------------------------- | ---------------------------------------------------------------------------------- |
| Synonyms and typo exceptions                   | Compass-governed terms; Typesense synonym/query configuration.                     |
| Protected brands, families, and molecules      | Compass protected-term policy before correction.                                   |
| Redirects                                      | Compass redirect rules.                                                            |
| Pins, hides, boosts, buries                    | Compass rule model, with Typesense curation used only as a retrieval optimization. |
| Listing, availability, Rx policy               | Shared Compass eligibility layer; never engine-specific.                           |
| Facets, filter values, category/brand identity | Canonical attribute dictionary and stable IDs.                                     |
| Sorts/ranking                                  | Versioned Compass ranking profiles and TypeSense query parameters.                 |
| Analytics                                      | Typesense native feedback plus PostHog/Compass attribution contract.               |

Reconcile dual indexes continuously: active/listable count, in-stock count, Rx count, category/brand/ingredient coverage, canonical product-family coverage, price/stock mismatch rate, and source-to-index lag.

### 20.3 Shadow comparison

In `shadow` mode, Netcore serves the user while Compass invokes Typesense asynchronously with a bounded timeout. The Typesense request never delay the visible response and uses `enable_analytics=false` so it cannot double-count popular/no-hit query data.

Store a comparison record with:

* Raw and normalized query, filters, sort, platform, surface, and query interpretation.
* Per-provider latency, errors, result count, product-family IDs/positions, display variants, facets, redirects, and fallback state.
* Top-3/top-10/top-24 canonical product overlap.
* Exact product and brand rank for golden/known queries.
* Safety, listing, stock, price, filter/facet, and variant-duplication mismatches.
* Active configuration/ranking profile versions.

Do not define success as identical rankings. The goal is Typesense non-inferiority against product expectations and safety requirements. Review high-volume, high-value, concern/ingredient, typo, Hinglish, Rx-sensitive, and high-difference query groups manually as well as automatically.

### 20.4 Sticky percentage rollout

Provider assignment is deterministic, server-side, and sticky. Use `user_id` when authenticated, otherwise `anonymous_id`, with `session_id` as the final fallback:

```text theme={null}
bucket = hash("discovery-provider-v1" + assignment_id) % 100
```

At a 25% rollout, buckets `0-24` receive `typesense_with_netcore_fallback`; remaining buckets receive Netcore. Preserve the assignment after sign-in and never switch a user between providers within a session. This protects pagination consistency, user experience, attribution, and experiment validity.

| Stage              |   Public Typesense traffic | Minimum observation                                    | Primary decision purpose                             |
| ------------------ | -------------------------: | ------------------------------------------------------ | ---------------------------------------------------- |
| Shadow             |                         0% | 1-2 weeks                                              | Data, result, safety, and latency parity.            |
| Internal beta      |                         0% | 1 week                                                 | Employee/QA workflow validation.                     |
| Canary             |                         1% | 24-72 hours                                            | Reliability, safety, freshness, and event integrity. |
| Early rollout      |                         5% | 3-7 days                                               | Query-quality and funnel guardrails.                 |
| Controlled rollout |              10%, then 25% | At least 1 week each                                   | Conversion/revenue non-inferiority.                  |
| Majority           |                        50% | Full business cycle; include campaign/sale if possible | Operational confidence at meaningful scale.          |
| Full web           |                       100% | 2-4 week stabilization                                 | Vendor fallback remains available.                   |
| Mobile             | Restart at 1% per platform | Independent progression                                | SDK/network and platform parity.                     |

At roughly 250,000 daily visitors, 1% is around 2,500 visits per day: enough for technical detection, not enough to claim small business-metric lift. Advance by decision gates, not elapsed time alone.

### 20.5 Feature flags and control precedence

Use separate controls for engine rollout and individual Typesense capabilities. Do not use one `use_typesense` flag for every behavior.

Provider controls:

```text theme={null}
discovery.provider.search
discovery.provider.autosuggest
discovery.provider.browse
discovery.global_kill_switch
```

Capability controls:

```text theme={null}
discovery.typesense.query_normalization
discovery.typesense.typo_tolerance
discovery.typesense.synonyms
discovery.typesense.protected_terms
discovery.typesense.product_grouping
discovery.typesense.filters
discovery.typesense.redirects
discovery.typesense.fallback_ladder
discovery.typesense.merchandising_rules
discovery.typesense.native_analytics
discovery.typesense.popularity_tiebreaker
discovery.typesense.hinglish_aliases
```

Flags are for rollout, experimentation, and emergency control. Permanent product decisions such as synonyms, pins, redirects, and campaigns belong in the Compass merchandising configuration model.

Flag evaluation order:

```text theme={null}
1. Emergency global kill switch
2. Safety/maintenance routing prerequisite
3. Internal QA/support allowlist or blocklist
4. Surface/platform restriction
5. Explicit experiment assignment
6. Sticky percentage rollout bucket
7. Default Netcore routing
```

Evaluate provider routing in Compass, not only in browser/mobile code. PostHog may provide experiment assignment, but safety-critical provider routing must continue to function if PostHog is unavailable. Log the selected provider, flag/configuration version, assignment reason, fallback outcome, request ID, and search ID on every request.

### 20.6 Backward compatibility

Maintain four compatibility guarantees during migration:

1. **Client API:** Compass endpoint paths, product-card fields, facets, sorting, pagination, and error semantics remain stable. New metadata is additive and optional.
2. **Identity:** Both providers map to the same canonical product/family/variant/brand/category identifiers.
3. **Behavior:** Existing Netcore rules are inventoried and reproduced or explicitly retired only after testing. Rx/safety behavior is centralized before public Typesense traffic.
4. **Data:** Netcore and Typesense index feeds run in parallel until decommission; all Typesense documents carry source product IDs, source versions, index timestamps, listability, purchasability, stock, and price.

### 20.7 Decision gates, fallback, and decommission

Hard-stop events immediately route the affected cohort to Netcore:

* Verified Rx or safety-rule violation.
* Invalid provider response, material price/availability breach, or index freshness breach.
* TypeSense error/timeout rate above threshold or cluster health/quorum incident.
* Attribution failure that makes the cohort unmeasurable.

Guardrail failures pause the rollout and trigger investigation:

* p95/p99 latency, zero-result, reformulation, search-exit, click, cart, revenue/session, and facet-empty-state regressions versus Netcore control.
* Event completeness, price/stock mismatch, index lag, or unresolved high-volume relevance differences.

Advance only when hard-stop metrics remain clean; guardrails are within tolerance; golden/high-volume review passes; Typesense is non-inferior on primary business outcomes; and discovery engineering, product, merchandising, catalogue/inventory, analytics, and clinical/compliance owners approve the review.

Use interleaving only for ranking-only comparisons after candidate sets, grouping, filters, and safety behavior have reached parity. Do not interleave incompatible result sets or different redirect/fallback experiences.

Netcore can be decommissioned only after 100% web traffic has completed the agreed stabilization period, mobile has completed or has an approved separate plan, safety mismatch is zero, search/business guardrails meet or exceed control, TypeSense capacity/recovery/snapshots are proven, vendor configurations are migrated/retired, and operations/support can diagnose through Compass rather than vendor tooling.

## 21. Delivery Plan

### Phase 0: Discovery foundation and engine proof of concept

Deliverables:

* Catalogue completeness audit and controlled attribute dictionary.
* Canonical `DiscoveryProduct` contract.
* 100+ golden search/recommendation safety cases.
* Baseline of current query, conversion, zero-result, latency, and reformulation metrics.
* Typesense proof of concept against real products and queries.
* Event schema and attribution design.
* Architecture decision record and production SLOs.

Exit criteria:

* Engine meets representative relevance and latency needs.
* The team can ingest/update a representative product set.
* The catalogue gaps blocking Phase 1 are assigned with owners.

### Phase 1: Search, autosuggest foundation, and safety

Build:

* Discovery API, Typesense collection/indexer, and catalogue delta ingestion.
* Search result integration on one chosen surface.
* Normalization, synonyms, protected terms, typo tolerance, and deterministic intent detection.
* Eligibility, stock/listability, product-family grouping, basic facets, and ranking profiles.
* Fallback ladder, redirects, pins/hides/boosts, and a narrow admin console.
* Search/suggestion event backbone, query dashboard, inspector, alerts, regression suite, staged rollout, and shadow mode.
* Rx and claim-safety rules.

Exit criteria:

* Golden cases pass.
* Safety violations are zero.
* Event completeness and attribution targets are met.
* Search quality matches or beats the current system under shadow/interleaving evaluation.
* A merchandiser can correct a known query problem without deploying code.

### Phase 2: Browse and SEO-safe PLPs

Build:

* Category, brand, concern, ingredient, and campaign page definitions.
* Configurable facets, quick filters, sorting, content slots, badges, and page-level rules.
* Search-inside-filter for high-cardinality facets.
* Stable URLs, redirects, canonical/index policy, and organic-traffic monitoring.
* Catalogue quality gates and assisted enrichment workflow.
* Page/filter health dashboards and page inspector.

### Phase 3: Recommendations

Build:

* Widget framework, strategies, placements, and measurement.
* Catalogue relationship data, hard exclusion filters, routine conflict/complement rules, and fallback chains.
* Curated and relationship-based widgets first, co-purchase candidates second.
* Offline review fixtures and widget experiments.
* Recently viewed/continue shopping and session intent adaptation.
* Hinglish coverage expanded based on Phase 1 query data.

### Phase 4: Full control plane and learning

Build:

* Campaign bundles, bulk imports, approval workflows, and rule-health monitoring.
* Ranking review workflow, bounded behavioral signals, profile-aware recommendations, and discovery profiles.
* Experiment registry, automatic proposal queues, holdouts, and weekly discovery review.
* Sponsorship only after explicit commercial, legal, relevance, and safety approval.

## 22. Cost and Scaling Strategy

### 22.1 Traffic planning assumptions

At roughly 250,000 daily website visitors, discovery is a meaningful production workload but not, by itself, a reason to introduce Kafka or OpenSearch.

Use measured funnel data for final capacity planning. Until it is available, size the initial benchmark and load tests using these conservative assumptions:

| Workload                     | Planning assumption                                            | Approximate daily volume        | Capacity implication                                                                                                                                                 |
| ---------------------------- | -------------------------------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Site visitors                | Known baseline                                                 | 250,000 visits                  | Traffic is bursty; average daily rate is not a sizing target.                                                                                                        |
| Executed searches            | 2-5 searches per visitor                                       | 500,000-1,250,000 searches      | Average is about 6-15 requests/second across a day; test 10-20x average for peak sale-event traffic.                                                                 |
| Autosuggest                  | 3-8 requests per executed search after client debouncing       | 1.5M-10M requests               | This is likely the highest request volume; debounce at 150-250 ms, cancel stale requests, cache short prefixes carefully, and target lower latency than full search. |
| Discovery interaction events | 5-15 server/client events per visitor                          | 1.25M-3.75M events              | PostHog and first-party ingestion need capacity testing; events remain asynchronous and never block the customer response.                                           |
| Catalogue deltas             | Depends on catalogue and inventory activity, not visitor count | Usually far below query traffic | SNS/SQS with idempotent workers and DLQs is sufficient for product, price, stock, and listing updates.                                                               |

Peak traffic must be determined from hourly production data, not daily averages. A sale, campaign, push notification, or SEO landing-page burst can reasonably produce 10-20 times average traffic. The Phase 1 load test should use the higher of the measured peak or this planning multiplier, with separate test mixes for search, autosuggest, filters, and product-click event ingestion.

### 22.2 Why SNS/SQS remains sufficient at this scale

SNS/SQS is the selected event backbone for Phases 0-3. It is appropriate because catalogue indexing is an asynchronous, idempotent work queue:

* Each product, price, inventory, or listing update is processed independently using `product_id + source_version` as its idempotency key.
* Ordering is required only per product, not globally. The indexer rejects stale versions, so normal SQS at-least-once delivery is safe.
* Worker concurrency can scale with queue depth through the existing Kubernetes/KEDA deployment model.
* SQS provides retries, visibility timeouts, buffering, and DLQ isolation already used by the platform.
* The primary high-volume path is synchronous query serving to Typesense, not catalogue event consumption. Kafka would not improve search-request latency.

Kafka or MSK should be evaluated only if the platform later needs a shared, long-retention, replayable stream for many consumers or low-latency behavioral aggregation for personalization/ranking. It is not a prerequisite for serving this visitor volume.

### 22.3 Cost priorities

At the start, engineering time and operational risk cost more than a small difference in infrastructure pricing. Self-hosted Typesense is appropriate while:

* Product schemas and relevance rules are evolving frequently.
* Traffic is moderate.
* The team is focused on building discovery capabilities rather than search-cluster operations.

Cost review should include total cost of ownership:

```text theme={null}
Typesense compute, persistent volumes, backups, and load-balancing price
+ compute and database cost
+ storage and analytics cost
+ engineering time for operations/tuning
+ incident/on-call cost
+ delayed delivery cost
```

### 22.4 Scaling paths

| Component  | Early stage                                                                                       | Growth stage                                                                                                                                     |
| ---------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| API        | Compass deployment with existing KEDA scaling on CPU/RPS/latency                                  | Split read-heavy query workloads and admin only if required; multi-region only with evidence.                                                    |
| Indexer    | Single consumer group with SQS batching                                                           | Parallel workers partitioned by product ID; backpressure/DLQ monitoring.                                                                         |
| PostgreSQL | RDS/Aurora with backups and read replica only if needed                                           | Read replicas for inspector/admin analytics, partition audit/event metadata.                                                                     |
| Redis      | Small HA ElastiCache                                                                              | Separate rate-limit/config/result-cache usage if contention emerges.                                                                             |
| Typesense  | Three self-hosted nodes with private Raft peering, persistent SSD/NVMe data and analytics volumes | Size CPU/RAM from benchmark data; move to five nodes only when two-node fault tolerance is justified; evaluate OpenSearch only against evidence. |
| Analytics  | Typesense analytics, PostHog, Compass operational records, and authoritative commerce events      | Add S3/Athena/dbt or the company warehouse only when the Section 17.4 triggers are met.                                                          |

Do not prematurely introduce Kafka, a separate Kubernetes platform, a data lakehouse, feature stores, or dedicated vector databases. Introduce them only when the existing queues, PostHog/warehouse reporting, or engine capabilities genuinely prove insufficient.

## 23. Open Decisions and Owners

| Decision                                       | Suggested owner                   | When needed                            |
| ---------------------------------------------- | --------------------------------- | -------------------------------------- |
| Product family vs flavour grouping policy      | Product + catalogue               | Before indexing contract is final      |
| Delivery-area handling: badge, demote, or hide | Product + operations              | Before location-aware inventory launch |
| Rx claim/promotion boundaries                  | Clinical/compliance + product     | Before Phase 1 public launch           |
| Attribute coverage thresholds                  | Catalogue + product               | Phase 0 completion                     |
| Search-first vs browse-first after Phase 1     | Product + analytics               | Phase 1 exit review                    |
| Sensitive recent-search behavior               | Privacy/compliance + product      | Before autosuggest launch              |
| Sponsored autosuggest ban                      | Product + commercial + compliance | Before any sponsorship design          |
| Search engine migration trigger thresholds     | Engineering + finance + product   | Before Phase 2 cost review             |

## 24. Immediate Next Steps

1. Confirm the Phase 1 surface: web search results is the recommended first integration.
2. Run the catalogue completeness audit and define the controlled attribute dictionary.
3. Collect 100-200 real, anonymized queries and build the first golden-query suite.
4. Create a Typesense proof of concept using the proposed `DiscoveryProduct` contract.
5. Define the event schema, Typesense analytics rules, and PostHog server/client ownership.
6. Build the eligibility/Rx policy module and test it independently before integrating search UI.
7. Establish the PostgreSQL configuration schema and a minimal query-rule preview workflow.
8. Shadow the existing vendor/system before any user traffic is shifted.

## 25. Definition of a Successful First Release

The first release is successful when it demonstrates all of the following in production or controlled rollout:

* Users can find exact products, brands, categories, concerns, and ingredients reliably.
* Common typos and approved aliases recover without damaging protected brands/molecules.
* Non-listable, non-purchasable, and prohibited Rx contexts never leak into results.
* Product variants are not duplicated in search.
* A zero-result query follows a measured, honest fallback path.
* Search clicks, add-to-carts, and purchases are attributable to the returned result list and position.
* Product/merchandising can publish, preview, inspect, and revert a query-level correction without engineering.
* The team can explain why a product appeared at its position.
* The platform can be rolled back or degraded safely if the engine or index pipeline fails.

That foundation is enough to begin browse and recommendations confidently. It is deliberately more valuable than shipping advanced ML, broad personalization, or a large control plane before the core retrieval, safety, data, and measurement loops are dependable.