Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Open source

loki.source.cloudflare

loki.source.cloudflare pulls logs from the Cloudflare Logpull API and forwards them to other loki.* components.

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server and can be useful for enriching existing logs on an origin server.

Multiple loki.source.cloudflare components can be specified by giving them different labels.

Usage

river
loki.source.cloudflare "LABEL" {
  zone_id   = "ZONE_ID"
  api_token = "API_TOKEN"

  forward_to = RECEIVER_LIST
}

Arguments

loki.source.cloudflare supports the following arguments:

NameTypeDescriptionDefaultRequired
forward_tolist(LogsReceiver)List of receivers to send log entries to.yes
api_tokenstringThe API token to authenticate with.yes
zone_idstringThe Cloudflare zone ID to use.yes
labelsmap(string)The labels to associate with incoming log entries.{}no
workersintThe number of workers to use for parsing logs.3no
pull_rangedurationThe timeframe to fetch for each pull request."1m"no
fields_typestringThe set of fields to fetch for log entries."default"no

By default loki.source.cloudflare fetches logs with the default set of fields. Here are the different sets of fields_type available for selection, and the fields they include:

  • default includes:
"ClientIP", "ClientRequestHost", "ClientRequestMethod", "ClientRequestURI", "EdgeEndTimestamp", "EdgeResponseBytes", "EdgeRequestHost", "EdgeResponseStatus", "EdgeStartTimestamp", "RayID"
  • minimal includes all default fields and adds:
"ZoneID", "ClientSSLProtocol", "ClientRequestProtocol", "ClientRequestPath", "ClientRequestUserAgent", "ClientRequestReferer", "EdgeColoCode", "ClientCountry", "CacheCacheStatus", "CacheResponseStatus", "EdgeResponseContentType
  • extended includes all minimal fields and adds:
"ClientSSLCipher", "ClientASN", "ClientIPClass", "CacheResponseBytes", "EdgePathingOp", "EdgePathingSrc", "EdgePathingStatus", "ParentRayID", "WorkerCPUTime", "WorkerStatus", "WorkerSubrequest", "WorkerSubrequestCount", "OriginIP", "OriginResponseStatus", "OriginSSLProtocol", "OriginResponseHTTPExpires", "OriginResponseHTTPLastModified"
  • all includes all extended fields and adds:
 "BotScore", "BotScoreSrc", "ClientRequestBytes", "ClientSrcPort", "ClientXRequestedWith", "CacheTieredFill", "EdgeResponseCompressionRatio", "EdgeServerIP", "FirewallMatchesSources", "FirewallMatchesActions", "FirewallMatchesRuleIDs", "OriginResponseBytes", "OriginResponseTime", "ClientDeviceType", "WAFFlags", "WAFMatchedVar", "EdgeColoID", "RequestHeaders", "ResponseHeaders"`k

The component saves the last successfully-fetched timestamp in its positions file. If a position is found in the file for a given zone ID, the component restarts pulling logs from that timestamp. When no position is found, the component starts pulling logs from the current time.

Logs are fetched using multiple workers which request the last available pull_range repeatedly. It is possible to fall behind due to having too many log lines to process for each pull; adding more workers, decreasing the pull range, or decreasing the quantity of fields fetched can mitigate this performance issue.

The last timestamp fetched by the component is recorded in the loki_source_cloudflare_target_last_requested_end_timestamp debug metric.

All incoming Cloudflare log entries are in JSON format. You can make use of the loki.process component and a JSON procssing stage to extract more labels or change the log line format. A sample log looks like this:

{
	"CacheCacheStatus": "miss",
	"CacheResponseBytes": 8377,
	"CacheResponseStatus": 200,
	"CacheTieredFill": false,
	"ClientASN": 786,
	"ClientCountry": "gb",
	"ClientDeviceType": "desktop",
	"ClientIP": "100.100.5.5",
	"ClientIPClass": "noRecord",
	"ClientRequestBytes": 2691,
	"ClientRequestHost": "www.foo.com",
	"ClientRequestMethod": "GET",
	"ClientRequestPath": "/comments/foo/",
	"ClientRequestProtocol": "HTTP/1.0",
	"ClientRequestReferer": "https://www.foo.com/foo/168855/?offset=8625",
	"ClientRequestURI": "/foo/15248108/",
	"ClientRequestUserAgent": "some bot",
	"ClientSSLCipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
	"ClientSSLProtocol": "TLSv1.2",
	"ClientSrcPort": 39816,
	"ClientXRequestedWith": "",
	"EdgeColoCode": "MAN",
	"EdgeColoID": 341,
	"EdgeEndTimestamp": 1637336610671000000,
	"EdgePathingOp": "wl",
	"EdgePathingSrc": "macro",
	"EdgePathingStatus": "nr",
	"EdgeRateLimitAction": "",
	"EdgeRateLimitID": 0,
	"EdgeRequestHost": "www.foo.com",
	"EdgeResponseBytes": 14878,
	"EdgeResponseCompressionRatio": 1,
	"EdgeResponseContentType": "text/html",
	"EdgeResponseStatus": 200,
	"EdgeServerIP": "8.8.8.8",
	"EdgeStartTimestamp": 1637336610517000000,
	"FirewallMatchesActions": [],
	"FirewallMatchesRuleIDs": [],
	"FirewallMatchesSources": [],
	"OriginIP": "8.8.8.8",
	"OriginResponseBytes": 0,
	"OriginResponseHTTPExpires": "",
	"OriginResponseHTTPLastModified": "",
	"OriginResponseStatus": 200,
	"OriginResponseTime": 123000000,
	"OriginSSLProtocol": "TLSv1.2",
	"ParentRayID": "00",
	"RayID": "6b0a...",
  "RequestHeaders": [],
  "ResponseHeaders": [
    "x-foo": "bar"
  ],
	"SecurityLevel": "med",
	"WAFAction": "unknown",
	"WAFFlags": "0",
	"WAFMatchedVar": "",
	"WAFProfile": "unknown",
	"WAFRuleID": "",
	"WAFRuleMessage": "",
	"WorkerCPUTime": 0,
	"WorkerStatus": "unknown",
	"WorkerSubrequest": false,
	"WorkerSubrequestCount": 0,
	"ZoneID": 1234
}

Exported fields

loki.source.cloudflare does not export any fields.

Component health

loki.source.cloudflare is only reported as unhealthy if given an invalid configuration.

Debug information

loki.source.cloudflare exposes the following debug information:

  • Whether the target is ready and reading logs from the API.
  • The Cloudflare zone ID.
  • The last error reported, if any.
  • The stored positions file entry, as the combination of zone_id, labels and last fetched timestamp.
  • The last timestamp fetched.
  • The set of fields being fetched.

Debug metrics

  • loki_source_cloudflare_target_entries_total (counter): Total number of successful entries sent via the cloudflare target.
  • loki_source_cloudflare_target_last_requested_end_timestamp (gauge): The last cloudflare request end timestamp fetched, for calculating how far behind the target is.

Example

This example pulls logs from Cloudflare’s API and forwards them to a loki.write component.

river
loki.source.cloudflare "dev" {
  zone_id   = env("CF_ZONE_ID")
  api_token = local.file.api.content

  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}