Part 3: NoSQL – Flexible Data Management¶

Goal:

Design a flexible, scalable NoSQL system to handle Stripe’s unstructured and semi-structured data (logs, user interactions, ML features, feedback).
Enables real-time fraud detection, customer personalization, and predictive analytics with low-latency queries and seamless OLTP/OLAP integration.

🎯 Core Objectives¶

Flexible Schema: Support dynamic fields.
Efficient Querying: Optimize for nested/unstructured data (e.g., logs, clickstream).
Scalability: Handle high-volume, high-velocity data (e.g., millions of logs/day).
Real-Time Sync: Keep NoSQL synchronized with OLTP
Integration: Link with OLTP/OLAP via embedded IDs (e.g., merchant_id, customer_id).

Use Cases¶

These four data types have very different structures — technical logs, free text, numerical metrics, and event sequences — and their structure can evolve over time (new fields, new event types). A traditional relational database would require redefining the schema with every change, whereas a NoSQL document can absorb these variations without migration.

Use Case	Description	Objective
Log Data	Error logs, API calls, access logs.	Debugging, auditing, and detecting technical anomalies.
User Interaction Data	Session recordings, clickstream, mobile events.	Understand user behavior to optimize UX and power personalization.
ML Features	Device fingerprints, behavioral patterns.	Feed predictive models (e.g., fraud detection, recommendations).
Customer Feedback	Reviews(free text), survey responses, support tickets	Measure satisfaction, identify friction points, enable sentiment analysis.

📂 Data Model¶

We use a MongoDB Document Structure for the database, which is a flexible and schema-less database.

Schema Design Principles and benefits¶

Flexible Schema — Document database allow adding or removing fields without migrations (e.g., fraud_risk_score).
Binary Data can be can be stored as Base64-encoded strings (e.g., digital signatures).
Hierarchical Data : we use nested JSON for complex structures (e.g., a logs array within transactions) stored inside the parent document — keeping related, structured data together in one place.

Managing Relationships Between Documents¶

Embedding vs. Referencing : related data live inside the same document or in a separate document (we decide what to nest versus what to link)
Embed frequently accessed data (e.g., metadata within transactions).
Reference large or rarely used data (e.g., link to Merchant_id whixh reference an object within the table).
Indexing : We index reference items (merchant_id, customer_id) and frequently queried items (timestamp) in mongo db for faster queries.

Integration¶

Real-Time Sync (Keep NoSQL synchronized with OLTP) is done via CDC (Debezium + Kafka).

See Data Integration & ML

Machine Learning Integration¶

Process Overview¶

ML models interact with the NoSQL system in two ways: reading features to make predictions, and writing results back into documents.

flowchart LR
    MongoDB[(MongoDB Features)] -->|Read| Model[ML Model]
    Model -->|Write prediction| MongoDB
    Kafka[Kafka Stream] -->|Real-time features| Model

Features are stored/updated in MongoDB (e.g., ML Features documents).
The model reads relevant features (directly or via the Kafka enrichment stream).
Predictions (e.g., fraud_risk_score, is_fraud) are written back into the document.

Strategies¶

Real-Time Fraud Detection: the model scores transactions as enriched events arrive from Kafka Streams, writing fraud_risk_score into the transaction document within milliseconds.
Customer Personalization: User Interaction Data (clickstream, sessions) feeds a recommendation model that updates a preferences/segment field on the customer document.
Predictive Analytics: batch-aggregated data (from the Spark/Airflow pipeline) trains models on historical patterns (e.g., churn, LTV), with results stored in Customer_Dim or dedicated prediction documents.

Monitoring & Updating Models¶

Mechanism	Purpose
`model_version` field	Tracks which model produced a given prediction.
Drift monitoring (e.g., Evidently AI)	Detects when live data diverges from training data.
Champion/Challenger pattern	Runs a new model version alongside production to compare before promoting.
Scheduled retraining (Airflow)	Periodically retrains models on fresh batch data.

Query performance¶

Time-Series Optimization : We can use TimescaleDB to optimize queries of time-based metrics (data with horodate like logs)

Denormalization : We duplicate data (e.g., embed merchant_name in transactions) to avoid jointures for faster queries, but if the data change we have to update it in several tables.

Examples¶

Logs¶

{
  "_id": "log_98765",
  "timestamp": "2026-07-19T14:30:00Z",
  "level": "ERROR",
  "service": "payment_processing",
  "message": "Transaction failed: insufficient funds",
  "metadata": {
    "transaction_id": "txn_12345",
    "merchant_id": "m_67890",
    "error_code": "INSUFFICIENT_FUNDS",
    "stack_trace": "...",
    "device": {
      "ip": "192.0.2.1",
      "user_agent": "Mozilla/5.0..."
    }
  }
}

User Interaction¶

{
  "_id": "session_54321",
  "user_id": "u_12345",
  "session_start": "2026-07-19T14:00:00Z",
  "session_end": "2026-07-19T14:15:00Z",
  "events": [
    {
      "timestamp": "2026-07-19T14:01:23Z",
      "type": "page_view",
      "page": "/checkout",
      "device": "mobile"
    },
    {
      "timestamp": "2026-07-19T14:02:45Z",
      "type": "add_to_cart",
      "product_id": "p_98765"
    }
  ]
}

ML Features Example¶

{
  "_id": "feature_txn_12345",
  "transaction_id": "txn_12345",
  "features": {
    "amount": 99.99,
    "time_since_last_txn": 3600, // seconds
    "device_fingerprint": "abc123...",
    "user_risk_score": 0.85,
    "merchant_category": "e-commerce",
    "ip_geolocation": "US-CA"
  },
  "labels": {
    "is_fraud": true,
    "model_version": "v2.1"
  },
  "timestamp": "2026-07-19T14:30:00Z"
}

Customer Feedback Example¶

{
  "_id": "feedback_78901",
  "user_id": "u_12345",
  "type": "review",
  "rating": 5,
  "comment": "Fast and secure payment process!",
  "metadata": {
    "transaction_id": "txn_12345",
    "merchant_id": "m_67890",
    "submitted_at": "2026-07-19T14:30:00Z",
    "tags": ["positive", "speed", "security"]
  }
}

Sample NoSQL Queries¶

1.Find Users with 3+ Failed Transactions in 5 Minutes

// MongoDB Aggregation Pipeline
db.transactions.aggregate([
  {
    \$match: {
      status: "failed",
      timestamp: { \$gte: new Date("2026-07-19T14:00:00Z") }
    }
  },
  {
    \$group: {
      _id: "\$user_id",
      failed_count: { \$sum: 1 },
      first_failure: { \$min: "\$timestamp" },
      last_failure: { \$max: "\$timestamp" }
    }
  },
  {
    \$match: {
      failed_count: { \$gte: 3 },
      \$expr: { \$lt: [{ \$subtract: ["\$last_failure", "\$first_failure"] }, 300000] } // 5 minutes in ms
    }
  }
]);

2.Find High-Risk Transactions in the Last Hour

db.transactions.find({
  "metadata.fraud_risk_score": { \$gte: 0.9 },
  timestamp: { \$gte: new Date("2026-07-19T13:00:00Z") },
  status: "completed"
}).sort({ timestamp: -1 });