There are three areas of interest:

  • The usual Model Performance, for example latency.
  • Model Outlier Detection: that is, for a single instance, whether it is out-of-distribution of training data.
  • And Model Drifting, which can be further distinguished in two:
    • Covariate Shift: the input data distribution changes, but p(y|x) remains the same.
    • Label Shift: the label distribution changes (that is, reality) but the conditional distribution stays the same p(x|y)