Observability

Distribution Shift

Identification

Statistical & Distance-Based Methods

  1. Population Stability Index (PSI) / Jensen-Shannon Divergence (JSD)

  2. Kolmogorov-Smirnov (KS) Test

Model Performance Monitoring

  1. Live A/B Testing with Shadow Models

  2. Error Analysis on Recent Queries

Embedding-Based Drift Detection

  1. Measuring Drift in Learned Representations (e.g., PCA, t-SNE)

  2. Contrastive Learning for Drift Detection

Operational Monitoring

  • ML Monitoring & Logging: Model drift detection, feature monitoring, data pipelines.

  • Data Drift and Concept Drift: Detection techniques, adaptive retraining pipelines.