Design a Feature Store

Problem Statement & Requirements

Think of a feature store like a prep kitchen in a restaurant. Instead of each chef (data scientist) individually chopping vegetables (computing features) from scratch, the prep kitchen prepares standardized ingredients that any chef can use. This ensures consistency, saves time, and prevents mistakes.

Functional Requirements

• Feature registration — Define features with schemas, descriptions, and ownership
• Online serving — Low-latency lookup of feature values for real-time inference
• Offline serving — Batch retrieval of historical features for training datasets
• Point-in-time joins — Retrieve features as they were at a specific timestamp
• Feature transformation — Define and execute batch + streaming transformations
• Discovery & reuse — Search and browse available features across teams

Non-Functional Requirements

• Online latency — <10ms p99 for feature vector retrieval
• Consistency — Training and serving features must be identical
• Scale — Millions of entities, thousands of features, millions QPS
• Freshness — Streaming features updated within seconds

Back-of-Envelope Estimation

System API Design

# Register a feature
POST /api/v1/features
{
  "name": "user_purchase_count_30d",
  "entity": "user",
  "dtype": "int64",
  "description": "Number of purchases in last 30 days",
  "source": { "type": "batch", "query": "SELECT user_id, COUNT(*) ..." },
  "owner": "fraud-team"
}

# Online: Get features for an entity (low-latency)
GET /api/v1/features/online?entity=user&id=user_123
    &features=purchase_count_30d,avg_order_value,account_age_days

# Offline: Generate training dataset with point-in-time join
POST /api/v1/features/offline
{
  "entity": "user",
  "features": ["purchase_count_30d", "avg_order_value"],
  "entity_df": "s3://data/training_entities.parquet",
  "timestamp_column": "event_timestamp"
}

Data Model

CREATE TABLE feature_definitions (
    feature_name  VARCHAR PRIMARY KEY,
    entity_type   VARCHAR,
    dtype         VARCHAR,
    description   TEXT,
    source_type   VARCHAR,  -- batch, streaming, on-demand
    transform_sql TEXT,
    owner         VARCHAR,
    tags          TEXT[],
    created_at    TIMESTAMP
);
-- Offline store (Parquet/Delta Lake, partitioned by date)
-- Schema: entity_id, feature_name, value, event_timestamp

-- Online store (Redis/DynamoDB)
-- Key: {entity_type}:{entity_id}
-- Value: {feature_name: value, ...} (hash map)

CREATE TABLE materialization_jobs (
    job_id        VARCHAR PRIMARY KEY,
    feature_name  VARCHAR,
    status        VARCHAR,
    started_at    TIMESTAMP,
    records       BIGINT,
    latency_ms    FLOAT
);

High-Level Architecture

Deep Dive: Core Components

Point-in-Time Correctness

def point_in_time_join(entity_df, feature_df):
    """Join features as-of each entity's timestamp."""
    # Sort features by time
    feature_df = feature_df.sort_values("event_timestamp")

    # For each entity row, find the latest feature
    # value BEFORE the entity's timestamp
    result = pd.merge_asof(
        entity_df.sort_values("event_timestamp"),
        feature_df,
        on="event_timestamp",
        by="entity_id",
        direction="backward"  # Only look at past values
    )
    return result

Online vs. Offline Store Tradeoffs

Feature Freshness: Batch vs. Streaming

Not all features need real-time updates. Categorize by freshness requirements:

• Static features (account_age, country): Update daily or on change
• Slow-moving (30-day purchase count): Batch update hourly
• Fast-moving (session click count): Stream with seconds-level freshness
• On-demand (real-time price): Compute at request time, don't store

Scaling & Optimization

Online Store at Scale

For millions of QPS: shard Redis by entity_id (consistent hashing), use read replicas, compress feature values (delta encoding for time-series), and batch multi-feature lookups into single MGET calls.

Efficient Materialization

• Incremental updates: Only recompute features for entities that changed
• Backfill: When adding a new feature, compute historical values in parallel
• Deduplication: Skip materialization if feature values haven't changed

Practice Problems

Quick Reference