Machine LearningQuerying

Querying the results

Once you have trained a model, you can run predictions using PromQL. A new grafanacloud-ml-metrics Prometheus datasource will be present in your Grafana Cloud instance. This datasource understands PromQL, and translates specific queries into running models.

The two main series to be queried are <forecast_metric_name>:predicted which contains series related to the predictions of a specific forecast, and <forecast_metric_name>:actual which provides the actual values retrieved from the query.

Predictions

The :predicted metric contains three series differentiated by the ml_forecast label. For example, a forecast with a metric name of request_rate the resulting series will be:

request_rate:predicted{ml_forecast="yhat"}
request_rate:predicted{ml_forecast="yhat_lower"}
request_rate:predicted{ml_forecast="yhat_upper"}

Where yhat is the calculated the predicted value, and yhat_lower and yhat_upper are the confidence bounds of the prediction.

Actual

The :actual metric will query the datasource for up-to-date data of the real values. The results for this query do not have an ml_forecast label, so to compare the :predicted and :actual series you will likely want to use without (ml_forecast).

For example, to calculate the residual, or error, of a prediction you can use the PromQL:

abs(request_rate:predicted{ml_forecast="yhat"} - without (ml_forecast) request_rate:actual)

Anomaly Detection

Another common use case is determining anomalies, defined as when the actual value is outside of the predicted upper and lower bounds. This can be done with the query:

request_rate:actual < ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_lower"}
or
request_rate:actual > ignoring (ml_forecast) request_rate:predicted{ml_forecast="yhat_upper"}