Sending Fastly logs to a third party logging provider

Fastly supports a variety of third party services as recipients of log data emitted from the edge.

Logging endpoints can be configured via the API, web interface, or CLI, and are available to both VCL and Compute@Edge services.

VCL vs Compute@Edge

In VCL, data is logged with the log statement, while in Compute@Edge each language SDK offers an equivalent API:

  1. Fastly VCL
  2. Rust
  3. AssemblyScript
  4. JavaScript
log_fastly::init_simple("my_endpoint_name", log::LevelFilter::Info);
log::info!("{} {}", req.get_url(), req.get_client_ip_addr().unwrap());

STDIO in Compute@Edge

In Compute@Edge, it's also possible to emit data to STDOUT and STDERR. Creating a log endpoint called stdout or stderr will capture the log output on these interfaces. STDIO output is also streamable to a terminal using fastly log-tail. Learn more about testing Compute@Edge services.

Automatic logging in VCL

When creating a log endpoint for a VCL service, you can optionally provide a log string as the format property, in our custom log format, and Fastly will generate and include a log statement in your VCL automatically. This allows you to create operational logging in one single step - creating both a log instruction that generates log events, and a log destination to which to send those events - so that for every request, log output will be emitted to the log endpoint.

The generated log statement will normally be placed in the vcl_log subroutine, but this can be changed using the placement property available on all log endpoint types. Setting placement to "none" will make the log endpoint available but will not generate any log instructions. In that case, you will need to write a log statement in your VCL manually as shown in the examples above.

Logging destinations

Supported log endpoints include generic protocols (so you can operate your own log receiver), and dedicated connectors for popular third-party services.

Generic endpoints

Generic log endpoints can be used to instruct your Fastly service to send logs to a destination of your choice, including your own logging infrastructure.

HTTP challenge

When sending logs to an HTTP endpoint, Fastly requires proof that you control the domain name specified in the URL field. We verify this by sending a challenge request to /.well-known/fastly/logging/challenge. The response to a challenge request must include a SHA-256 hash (in a hex string format) of your Fastly service ID and it must appear on its own line in the response. If multiple Fastly services are configured to use the same log endpoint, multiple hex(sha256) values can be added to the challenge response. Alternatively, an asterisk (*) can be used on a line to allow any service to post to the HTTP endpoint. For example:

GET /.well-known/fastly/logging/challenge HTTP/1.1
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 132
ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c
06ae6402e02a9dad74edc71aa69c77c5747e553b0840bfc56feb7e65b23f0f61
*

The correct value for your challenge response can be generated on most unix-like systems by running sha256sum

$ echo -n "YOUR_FASTLY_SERVICE_ID" | sha256sum
ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c -

Syslog facility and severity

All log messages delivered to syslog endpoints have a facility of local0 (a user-defined code in the syslog standard) and a severity of info (level 6).

Dedicated integrations

We support a variety of third party services, which may be configured via the web interface, API or CLI:

NameInstructions
Amazon KinesisWeb interfaceAPICLI
Amazon S3Web interfaceAPICLI
DatadogWeb interfaceAPICLI
DigitalOcean SpacesWeb interfaceAPICLI
ElasticsearchWeb interfaceAPICLI
Google BigQueryWeb interfaceAPICLI
Google Cloud Pub/SubWeb interfaceAPICLI
Google Cloud StorageWeb interfaceAPICLI
Heroku LogplexWeb interfaceAPICLI
HoneycombWeb interfaceAPICLI
LogentriesWeb interfaceAPICLI
LogglyWeb interfaceAPICLI
Microsoft Azure Blob StorageWeb interfaceAPICLI
New RelicWeb interfaceAPICLI
PapertrailWeb interfaceAPICLI
Rackspace Cloud FilesWeb interfaceAPICLI
ScalyrWeb interfaceAPICLI
SplunkWeb interfaceAPICLI
SumoLogicWeb interfaceAPICLI

Consult your preferred provider's documentation for details on how to set up their service. Some details specific to particular providers are included in the web interface guides linked above.

Compatible integrations

Many third party services not explicitly supported by Fastly can also be used via a generic transport, or a compatible third-party for which we have a dedicated connector.

Automatic log generation

The format property of log endpoints accepts a proprietary syntax based on the Apache log format which is converted into edge code for VCL services, and inserted into the VCL subroutine identified by the placement property.

IMPORTANT: This section does not apply to Compute@Edge services, where Fastly cannot manipulate your code. Each Compute@Edge language offers an API within the Fastly SDK that allows data to be written to a log endpoint from your code. This must be done separately to the creation of the log endpoint.

The value of the format property is the literal string to log, with the following placeholders replaced with the appropriate dynamic value:

PlaceholderSource VCL variableDescription
%%A literal % character.
%areq.http.Fastly-Client-IPThe client IP address of the request.
%Aserver.ipThe local IP address.
%Bresp.body_bytes_writtenThe size of response in bytes, excluding HTTP headers.
%bresp.body_bytes_writtenThe size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent.
%{foo}Creq.http.Cookie:fooThe contents of cookie Foobar in the request sent to the server.
%Dtime.elapsed.usecThe time taken to serve the request, in microseconds.
%freq.url.pathThe URL path, e.g. /images/cat.jpg
%hreq.http.Fastly-Client-IPThe client IP address of the request.
%Hreq.protoThe request protocol.
%{foo}ireq.http.fooThe contents of the specified header in the request sent to the server.
%Ireq.bytes_readBytes received, including request headers and body.
%mreq.methodThe request method.
%{foo}oresp.http.fooThe contents of the specified header in the response.
%Oresp.bytes_writtenBytes sent, including headers. Will never be zero.
%pserver.portThe canonical port of the server serving the request. Always returns 80.
%{format}pserver.portThe canonical port of the server serving the request. Valid formats are "canonical", "local", or "remote". Always returns 80 for "canonical" and "local", and always returns "-" for "remote".
%qreq.urlThe query string (prepended with a ? if a query string exists, otherwise an empty string).
%rreq.request, req.url, req.protoThe first line of the request (unquoted).
%sresp.statusThe HTTP status code on the response. For requests that restart, this is the status of the original request. Use %>s for the final status.
%ttime.startThe time the request was received, in Standard English format (e.g., [01/Jan/1970:00:00:00 -0700]). The last number indicates the timezone offset from UTC.
%{format}ttime.start.msecThe time, in the form given by format, which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3), the following format tokens are supported: sec (number of seconds since the Epoch), msec (number of milliseconds since the Epoch), usec (number of microseconds since the Epoch), msec_frac (millisecond fraction), and usec_frac (microsecond fraction).
%Ttime.elapsed.secThe time taken to serve the request, in seconds.
%Ureq.url.pathThe URL path requested, not including any query string. Same as %f.
%vreq.http.hostThe value of the Host header on the client request.
%Vreq.http.hostSame as %v.
%{vcl}V{vcl}The literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., %{client.geo.country_code}V or %{tls.client.cipher}V). This %-directive is a Fastly extension and is not found in Apache. See useful variables to log for more examples.

Additionally, the following placeholders are recognized but not relevant to Fastly services, and always return constants intended to allow the output to remain parsable by tools intended to work with Apache formats:

  • Always returns "-": %{foo}e, %l, %{foo}n, %P, %{foo}P, %R, %u
  • Always returns 0: %k
  • Always returns +: %X

When using format to generate log instructions, format_version should be set to 2. The older, version 1 log format continues to be supported for compatibility but is not recommended for new configuration.

Examples

To replicate Apache's common log format, set format to:

%h - - %t "%r" %>s %b

This will produce a log line such as:

123.1.12.123 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

However, in modern usage, structured formats such as JSON are increasingly replacing the venerable space-separated log formats popularized by Apache. Where supported by the system you are sending logs to, we recommend constructing JSON or another format that can be parsed by a standard library. This also permits logging of a much greater variety of valuable data points.

HINT: If you are familiar with VCL, using only the raw-VCL placeholder %{.....}V instead of the Apache tokens may make it easier to understand and maintain your log configurations.

Here is an example of a format that produces JSON-formatted log lines, using only VCL placeholders:

{
"timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
"client_ip": "%{req.http.Fastly-Client-IP}V",
"geo_country": "%{client.geo.country_name}V",
"url": "%{json.escape(req.url)}V",
"request_referer": "%{json.escape(req.http.referer)}V",
"request_user_agent": "%{json.escape(req.http.User-Agent)}V",
"fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V,
"response_state": "%{json.escape(fastly_info.state)}V",
"response_status": %{resp.status}V,
"response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
"response_body_size": %{resp.body_bytes_written}V,
"request_method": "%{json.escape(req.method)}V",
"request_protocol": "%{json.escape(req.proto)}V",
"fastly_server": "%{json.escape(server.identity)}V",
"host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V"
}

Fastly does not attempt to parse your log output, so take care to ensure that if you are logging in a standard structured format, it is correctly escaped and valid. If not, your log destination may not understand it.

Explicit logging in code

Both Compute@Edge and VCL services support logging to a named log endpoint with an explicit line of code, as shown in the example at the top of this page. Where a log endpoint is configured with automatic log generation, you can still log to that endpoint manually as well.

VCL log statements are prefixed with syslog, the service ID, and the name of the destination endpoint, followed by a :: delimiter. This prefix, up to and including the delimiter, is removed from the log message before it is dispatched to the log destination.

// |--------------- Header -----------------| |-------------- Content --------------|
log "syslog " req.service_id " my_log_endpoint :: " req.http.Fastly-Client-IP " " req.url;

In Compute@Edge, all languages offer APIs to create log handles and specify the name of the log endpoint separately to the log message.

Log messages must be a single line. Newline characters or null bytes in log messages will terminate the message.

Useful variables to log

Fastly exposes a wealth of information for every request, and much of that data is useful to log. The full set of data available in VCL services is described in the VCL reference. Compute@Edge services have language-specific SDKs providing access to much of the same data and a set of environment variables which are common across all our supported languages.

Here we have collected a list of data available to Fastly services which are typically the most popular data to include in log messages. Those marked with ⭐ appear in the largest number of customer configurations.

WARNING: Be sure to take into account security, privacy, and compliance requirements when making decisions about the data you intend to include in logs. Some jurisdictions may have data protection regulations.

To use one of these variables in a VCL log message, either write it directly using the log statement or, if using log generation include it in a format property of a log endpoint, in the form %{VARNAME}V. Where data is indicated as available in Compute@Edge, the access syntax will depend on the language SDK in use. Consult the documentation for your preferred language.

DescriptionVCLCompute@Edge?
⭐ Client IP addressreq.http.fastly-client-ipYes
Client ISP ID (autonomous system number)client.as.numberNo
Client ISP name (the name of the organization associated with the client's autonomous system)client.as.nameNo
⭐ Country from which client request originatedclient.geo.country_codeYes
Client connection typeclient.geo.conn_typeYes
Approximate client latitudeclient.geo.latitudeYes
Approximate client longitudeclient.geo.longitudeYes
⭐ TLS versiontls.client.protocolNo
SNI server nametls.client.servernameNo
Cipher used in TLS requesttls.client.cipherNo
SHA-1 of cipher suite identifierstls.client.ciphers_shaNo
SHA-1 of TLS extension identifierstls.client.tlsexts_shaNo
Congestion control algorithmclient.socket.congestion_algorithmNo
Current congestion window size (packets)client.socket.cwndNo
Buffer space available for receiving dataclient.socket.tcpi_rcv_spaceNo
HTTP requests received on this connection (including the current one)client.requestsNo
Round-trip timeclient.socket.tcpi_rttNo
Round-trip time varianceclient.socket.tcpi_rttvarNo
Receiver-side estimate of round-trip timeclient.socket.tcpi_rcv_rttNo
Socket max segment size (receive)client.socket.tcpi_rcv_mssNo
Whether this is an IPv6 requestreq.is_ipv6No
⭐ Whether client request is HTTP/2fastly_info.is_h2No
Whether this is a generated H2 Push requestfastly_info.h2.is_pushNo
HTTP/2 stream IDfastly_info.h2.stream_idNo
Name of the service VCL currently loadedreq.vclNo
⭐ Version of the VCL currently executingreq.vcl.versionNo
⭐ Fastly POP identifierserver.datacenterYes
Cache node identifierserver.hostnameYes
Whether the request is being handled on a shield POPfastly.ff.visits_this_serviceNo
Size of client request headersreq.header_bytes_readNo
Size of client request bodyreq.body_bytes_readNo
URL pathreq.url.pathNo
HTTP Referer request header received in requestreq.http.refererYes
HTTP User-Agent header received in requestreq.http.user-agentYes
HTTP Accept header received in requestreq.http.acceptYes
HTTP Accept-Language header received in requestreq.http.accept-languageYes
HTTP Accept-Language header received in requestreq.http.accept-languageYes
HTTP Accept-Charset header received in requestreq.http.accept-charsetYes
HTTP If-Modified-Since header received in requestreq.http.if-modified-sinceYes
HTTP If-None-Match header received in requestreq.http.if-none-matchYes
⭐ Category of cache resultfastly_info.stateNo
⭐ Age of cached objectobj.ageNo
The number of times this object has been used on this cache serverobj.hitsNo
Cache freshness lifetime remainingobj.ttlNo
HTTP response status coderesp.statusNo
HTTP status text of responseresp.responseNo
HTTP Cache-Control header sent in responseresp.http.cache-controlNo
⭐ HTTP Content-Length header sent in responseresp.completedYes
⭐ HTTP Content-Type header sent in responseresp.http.content-typeYes
HTTP Expires header sent in responseresp.http.expiresYes
HTTP Last-Modified header sent in responseresp.http.last-modifiedYes
HTTP ETag header sent in responseresp.http.etagYes
Bytes delivered as the response bodyresp.body_bytes_writtenNo
Bytes delivered as the response headerresp.header_bytes_writtenNo
Total bytes delivered to clientresp.bytes_writtenNo
Whether the response has been completedresp.completedNo
Bitrate from Fastly to clientclient.socket.tcpi_delivery_rateNo
Estimated packet lossclient.socket.plossNo
Packet retransmission countclient.socket.tcpi_delta_retransNo
⭐ Start time of request in ISO8601 formatstrftime({"%Y-%m-%dT%H:%M:%S%z"}, time.start)No
Start time of request as a unix timestamptime.start.secNo
End time of the request in ISO8601 formatstrftime({"%Y-%m-%dT%H:%M:%S%z"}, time.end)No
Time elapsed since data last sent to clientclient.socket.tcpi_last_data_sentNo
⭐ Elapsed timetime.elapsed.usecNo

Storing intermediate state

Log statements rendered into VCL as part of log generation are placed in vcl_log. At this point in the VCL state machine, much of the request and response data that you might wish to log is available, but some data values which may exist ephemerally in other parts of the state machine are no longer accessible. It's also possible that your configuration may have modified the values of writable req.* properties in order to correctly route the request, but you wish to log the original state of the request as received by Fastly.

The following code example demonstrates how variables can be sampled throughout the VCL flow, providing additional insights into backend requests and responses, the shielding process, and more granular timing data:

Limitations and constraints

There is a limit on the maximum length of each log message which varies by platform. In VCL services it is 16KB; in Compute@Edge, it is 64KB.