Querying the results
Once you have trained a model, you can run predictions using PromQL. A new grafanacloud-ml-metrics Prometheus data source will be present in your Grafana Cloud instance. This datasource understands PromQL, and translates specific queries into running models.
The two main series to be queried are <forecast_metric_name>:predicted which contains series related to the predictions of a specific forecast, and <forecast_metric_name>:actual which provides the actual values retrieved from the query.
Predictions
The :predicted metric contains three series differentiated by the ml_forecast label. For example, a forecast with a metric name of request_rate the resulting series will be:
request_rate:predicted{ml_forecast="yhat"}
request_rate:predicted{ml_forecast="yhat_lower"}
request_rate:predicted{ml_forecast="yhat_upper"}
Where yhat is the calculated the predicted value, and yhat_lower and yhat_upper are the confidence bounds of the prediction.
Actual
The :actual metric will query the datasource for up-to-date data of the real values. The results for this query do not have an ml_forecast label, so to compare the :predicted and :actual series you will likely want to use ignoring (ml_forecast).
For example, to calculate the residual, or error, of a prediction you can use the PromQL:
abs(request_rate:predicted{ml_forecast="yhat"} - ignoring (ml_forecast) request_rate:actual)
Anomaly Detection
Another common use case is determining anomalies, defined as when the actual value is outside of the predicted upper and lower bounds. This can be done with the query:
request_rate:actual < ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_lower"}
or
request_rate:actual > ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_upper"}
