Cross-Cutting Concerns¶
The data architecture must include robust security measures, such as encryption, access controls, and audit logging.
Stripe must ensure that its data storage and processing meet regulatory requirements across different jurisdictions.
The system should support automated compliance reporting and real-time monitoring for security breaches.
Security & Compliance¶
| Requirement | Solution | Tools |
|---|---|---|
| Encryption | AES-256 (at rest), TLS 1.3 (in transit) | Snowflake, PostgreSQL |
| Access Control | RBAC + ABAC | HashiCorp Vault |
| Audit Logging | Immutable logs | Snowflake Audit Logs, pgAudit |
| PCI-DSS Compliance | Tokenization of PII | Protegrity |
| GDPR Compliance | Right to erasure scripts | Custom Automation |
Monitoring¶
-
Prometheus collects metrics from PostgreSQL, Kafka, Snowflake, and MongoDB.
-
Grafana visualizes these metrics for observability across the pipeline.
| Metric | Tool | Threshold | Applies To |
|---|---|---|---|
| Query Latency | Prometheus + Grafana | > 500ms | OLAP |
| Query Failure Rate | Snowflake Query History | > 1% | OLAP |
| Data Freshness (Replication Lag) | Kafka Lag Script | > 5 minutes | Pipeline (OLTP→OLAP) |
| Storage Usage | Snowflake Storage Metrics | > 80% capacity | OLAP |
| Transaction Throughput | pg_stat_activity / Prometheus | OLTP | |
| Replication/WAL Lag | pg_stat_replication | OLTP | |
| Connection Pool Saturation | PgBouncer stats | OLTP |
Tech Stack¶
| Component | Technology | Why? |
|---|---|---|
| OLTP Database | Normalized 3NF, PostgreSQL | ACID compliance, transactional integrity, minimize redundancy, support high throughput. |
| OLAP Database | Star schema, PostgreSQL | Handle unstructured data, support ML features and flexible queries. |
| NoSQL Database | MongoDB, Document | Flexible schema, JSON support, storage for unstructured data and ML features, horizontal scaling. |
| Data Pipeline | Kafka + Debezium (CDC), Spark (batch) | Real-time and batch data flow, error handling, transformation. |
| Batch Orchestration | Apache Airflow | Schedule and monitor ETL jobs. |
| Storage | Snowflake (Parquet) | Optimized for analytical queries. |
| Security & Compliance | AES-256, RBAC, Audit Logging, Protegrity | Ensure GDPR, PCI-DSS, CCPA compliance and data protection. |
| Machine Learning | TensorFlow Serving, MLflow, Kubernetes | Real-time fraud detection, customer personalization, model monitoring. |
| Caching | Redis | Low-latency caching. |
| Monitoring | Prometheus + Grafana | Track performance and costs. |