There are three areas of interest:

  • The usual model performance, for example latency.
  • Model outlier detection: that is, for a single instance, whether it is out-of-distribution of training data.
  • And Model drifting, which can be further distinguished in two:
    • Covariate shift: the input data distribution changes, but p(y|x) remains the same.
    • Label shift: the label distribution changes (that is, reality) but the conditional distribution stays the same p(x|y)