Testing fiddles

The Tests option in a fiddle's request configuration allows you to write test assertions against the instrumentation data that is returned by Fiddle when you press RUN. Writing tests is a good way to express what parts of a fiddle are most important when sharing the solution with someone else, or to express a bug by demonstrating the gap between observed behavior and what you expected to see.

Each test should be written on a single line and be in the following format:

[LABEL] TARGET COMPARISON_TYPE REFERENCE_VALUE

For example:

[Cache TTL is positive] events.where(fnName=fetch)[0].attribs.ttl isMoreThan 0

The optional LABEL, which must be wrapped in square brackets if present, allows you to define a 'human readable' expression for the test. TARGET is the thing being tested, REFERENCE_VALUE is a literal value to compare it with, and COMPARISON_TYPE is the comparison you want to make.

You can write more than one test per request.

An example test in Fiddle

Test targets

Within a fiddle test case, TARGET should reference a property of the result data, based on the following schema:

NameTypeDescription
clientFetchObjThe request from the client to Fastly (and the response from Fastly).
├─ .reqStrHTTP request block, containing request method, path, HTTP version, header key/value pairs and request body.
├─ .respStrHTTP response header, contains response status line and response headers (not the body).
├─ .respTypeStrParsed Content-type response header value (mime type only).
├─ .isTextBoolWhether the response body can be treated as text.
├─ .isImageBoolWhether the response body can be treated as an image.
├─ .statusNumHTTP response status.
├─ .bodyPreviewStrUTF-8 text preview of the body (truncated at 1K).
├─ .bodyBytesReceivedNumAmount of data received.
├─ .bodyChunkCountNumNumber of chunks received.
├─ .completeBoolWhether the response is complete.
└─ .trailersStrHTTP response trailers.
originFetchesArrayOrigin fetches made during the request.
└─ [idx]ObjEach fetch is one object.
   ├─ .fetchIDStrUnique ID for this fetch.
   ├─ .traceIDStrID for the compute instance that triggered this fetch.
   ├─ .reqStrHTTP request block, containing request method, path, HTTP version, header key/value pairs and request body.
   ├─ .respStrHTTP response header, contains response status line and response headers (not the body).
   ├─ .remoteAddrStrResolved IP address of origin.
   ├─ .remotePortNumPort on origin server.
   ├─ .remoteHostStrHostname of origin server.
   └─ .elapsedTimeNumTotal time spent on origin fetch (ms).
eventsArrayFastly platform events related to the request, in order of execution.
└─ [idx]ObjEach array element is one event.
   ├─ .typeStrOne of vcl-sub, log, fetch or a Compute@Edge event type: ecp-start, ecp-end, geo, dict-open, dict-get
   ├─ .timeNumUnix timestamp in microseconds
   ├─ .popStrThree letter code identifying the Fastly POP location in which this event occurred, eg. 'LON', 'IAD', 'SYD'.
   ├─ .nodeIDNumNumeric identifier of the individual server on which this event occurred.
   ├─ .reqIDStrIdentifier for this client request. In a fiddle that sends more than one client request, there may be multiple different reqIDs.
   ├─ .traceIDStrID for the execution flow that triggered this event. In Compute@Edge, this represents the compute instance that is currently handling the request. In VCL, it represents one pass through a VCL state flow (i.e., it resets on a restart, ESI, or HTTP/2 server push)
   ├─ .seqIdxNumSequence number of this event with respect to other events with the same traceID and nodeID.
   ├─ .parentStrIf the event is part of a process that was initiated by a different process, this is the traceID of the parent compute process. Normally populated as a result of shielding
   ├─ .isAsyncBool(vcl-sub only) True if the event happened as part of an asynchronous task.
   ├─ .fnNameStr(vcl-sub only) For VCL events, the VCL subroutine that triggered the event. May be 'recv', 'hash', 'hit', 'miss', 'pass', 'fetch', 'deliver', 'error', or 'log'.
   ├─ .logsArray(vcl-sub only) Array of log messages logged from this event.
   │  └─ [idx]ObjEach array element is one log message.
   │     ├─ .contentStrThe content of the log message
   │     └─ .logNameStrThe name of the log destination, eg. 'stdout' or 'bigquery'
   ├─ .wafObj(vcl-sub only) If the Web application firewall ran during this event, this object provides the results from it.
   │  ├─ .didLogBoolWhether the WAF generated any log events, which happens if any WAF rules were matched.
   │  ├─ .didBlockBoolWhether the WAF acted to block the request from proceeding by triggering an error.
   │  ├─ .didFailNumWhether the WAF failed due to a problem evaluating a rule.
   │  ├─ .categoryScoresObjAccumulated scores for each of the WAF scoring categories. (see WAF rule data model).
   │  ├─ .scoringRulesArrayA list of matched scoring rules (see WAF rule data model).
   │  └─ .thresholdRulesArrayA list of matched threshold rules (see WAF rule data model).
   ├─ .attribsObjEvent-specific properties. These are displayed in the Fiddle UI for the event. For example: within miss events, a staleExists property is reported.
   └─ .prevAttribsObjEvent-specific properties where the value at the start of the subroutine was different to that recorded at the end.
logsArrayArray of strings, messages logged from all events.
insightsArrayInsight tags for this request. Insight tags identify recommendations or divergences from best practices. For example: [ "client-cc-missing", "invalid-header" ].

For example, to target the headers that are sent by Fastly to the backend in an origin request, use the .req property of the first item in the originFetches collection:

originFetches[0].req includes "My-Custom-Header"

In some cases targets are more complex. For example, the TTL set in vcl_fetch is reported as the .ttl property of the event object created by the vcl_fetch subroutine, but it's not possible to know ahead of time what index that event has within the events array. Instead, use a filter function to narrow down events to just the events associated with the vcl_fetch subroutine, and then pick the first one:

events.where(fnName=fetch)[0].attribs.ttl isMoreThan 0

HINT: If there is an .attribs property on an object, Fiddle will treat it as a special case and allow the target expression to directly access the child properties of attribs, to make targeting event properties easier. The example shown above can therefore also be written as:

events.where(fnName=fetch)[0].ttl isMoreThan 0

The where(fnName=fetch) is an example of using an aggregation function to transform the test data and make it easier to target the values that you want to test. The available functions are:

FunctionDetails
listBy(field)
Array ➔ Array
Takes an array of objects and makes an array of arrays, where in each sub-array, all objects share the same value of field. Items in the output array are ordered based on the order in which the values of the selected field first appear in the input.

e.g. events.listBy(vclflowkey)[1][0].fnName is "recv"
where(field=val)
Array ➔ Array
Takes an array of objects and filters it to leave only those where the property called field has value val.

e.g. events.where(fnName=recv)[0].url startsWith "/a/b"
groupBy(field)
Array ➔ Obj
Takes an array of objects and splits it into multiple arrays where each one has the same value for the property field, and organizes the resulting data into an object where the field values are keys.

e.g. events.groupBy(fnName).recv[0].url startsWith "/a/b"
transpose()
Array ➔ Obj
Takes an array of objects and makes an object of arrays.  Where multiple input objects share the same property name, that property becomes a top level property with an array containing all the values.

e.g. events.transpose().return notIncludes "error"
count()
Array ➔ Num
Takes an array and returns the length

e.g. originFetches.count() greaterThan 1
concat()
Array ➔ Str
Takes an array and returns a string representation of all array items joined together, delimited by newlines.

e.g. logs.concat() includes "Hello"

HINT: All timing properties reported by Fiddle are in microseconds - so 1 hour is reported as 3600000000 (3.6 billion microseconds). The most common timing property to target for testing is .ttl, which is provided on events where fnName is fetch.

Lastly, you need to assert something about this target data, which requires a comparison type and a reference value, together making up an assertion.

Assertions

We support the following options for COMPARISON_TYPE:

NameValue typeDescription
isAnyNon-strict equality (using ==)
isJSONNoneThe target is valid JSON (does not require a reference value, so just two parameters)
isTrueNoneThe target is true
isAtLeast, isAboveJSON numberTarget is numerically higher than (or at least) the reference value
isAtMost, isBelowJSON numberTarget is numerically lower than (or at most) the reference value
includesAnyThe target includes the reference value. Can be used to assert the inclusion of a value in an array, a substring in a string, or a subset of properties in an object.
matchesJS RegExpThe target matches a regular expression. Regex must be delimited with / and may be followed by modifiers, e.g. /abc/i
oneOfJSON arrayChecks that the value of the target is equal to at least one of the values in the reference array
startsWithJSON stringChecks that the value of the target starts with the reference string
endsWithJSON stringChecks that the value of the target ends with the reference string

All comparisons can be negated by prefixing them with not, e.g. notOneOf, isNotJSON, notMatches. The word 'is' will be ignored so can be used purely to improve readability.

The REFERENCE_VALUE must be expressed in a way that can be interpreted as the appropriate type:

  • JSON number: A plain number (3, 5.7, -42)
  • JSON string: A double quoted string ("foo"; contained double-quotes must be escaped as \")
  • JSON boolean: The word true or false
  • JSON array: A JSON-parseable array containing any combination of numbers, strings and booleans as defined above ([1, "foo"])
  • JS RegExp: Either a JavaScript regex literal (/(foo|bar)/ism) or the bare pattern, not quoted or delimited (foo|bar). In the latter case, the s, m and i flags will be applied to the pattern automatically.

Labels

Labelling your tests is optional but helps to illustrate your intent, as well as make test results easier to read. The LABEL portion of the test expression may contain any character other than square brackets ([]), and since test expressions must be one line, labels may also not include newline characters.

[Response is OK] clientFetch.status is 200

Labelled tests in the test results

Example tests

If you want to test...Try this example
HTTP response status on the client fetchclientFetch.status is 200
Request headers on the client fetchclientFetch.req notIncludes "Fastly-FF: "
Response headers on the client fetchclientFetch.resp includes "Age: "
HTTP response status of a backend fetchevents.where(fnName=fetch)[0].status isOneOf [301,302,307,308]
Request headers on a backend fetchoriginFetches[0].req includes "GET /"
Response headers on a backend fetchoriginFetches[0].resp matches /\w+@gmail\.com/
TTL set in a vcl_fetch subroutineevents.where(fnName=fetch)[0].ttl isLessThan 86400000000
URL of the request following a VCL restartevents.where(fnName=recv)[1].url is "/foo"
(After a restart, vcl_recv will run a second time, so the [1] targets the second occurrence
Number of origin fetchesoriginFetches.count() is 2
Response body from Fastly to clientclientFetch.bodyPreview contains "<meta"
(You will only get the first 1K of the response)
Value of fastly.info_state in deliverevents.where(fnName=deliver)[0].state is "HIT"
Number of times deliver is runevents.where(fnName=deliver).count() is 1

Debugging tests

When a test fails, Fiddle will display a representation of the observed, 'actual' value of the target, so you can see why it didn't match. That observed value will often be undefined because the target didn't match anything. In that case, it can be helpful to shorten your target expression, to reveal the shape of the data structure and allow you to see what you can target:

Test failure showing actual value of target

Testing best practices

When constructing tests for your fiddle, consider the following best practices:

  • Label your tests to better illustrate your intent
  • Try to avoid asserting the nature of the implementation of the solution. Instead, assert the achievement of the overall goal. It should be possible to change the way the solution is achieved without having to change the tests.
  • For solutions that act on the user's IP address, for example to block users from certain regions from accessing certain content, use the Client IP override in request options to ensure that the test results are reliably reproducible.
  • Fiddle data arrives asynchronously, but not all instrumentation delays are created equal! Tests that target clientFetch will be fastest. Those that target originFetches will be slightly slower, and events is slowest. Fiddle will keep repeating your tests until they pass or a timeout is reached, to allow async data to arrive. Speed up your fiddle by using clientFetch and originFetch assertions instead of testing events or logs.

WAF rule data model

The .waf property of a VCL flow event contains details of the execution of the Web application firewall. WAF rules are classified as scoring or threshold and also into one of the following categories:

  • sql: SQL injection
  • rfi: Remote file inclusion
  • lfi: Local file inclusion
  • rce: Remote command execution
  • php: PHP injection
  • session: Session hijacking
  • http: HTTP violation
  • xss: Cross-site scripting

The .waf.categoryScores property provides total scores for each category plus anomaly, a total of all categories. This is a good way to test that a request triggers a rule in a particular category:

events.where(fnName=miss)[0].waf.categoryScores.xss isAbove 0

If you simply want to know whether the WAF was triggered at all, use one of the top level properties of the .waf data block:

events.where(fnName=miss)[0].waf.didBlock isTrue

Individual matched rules are reported in .waf.scoringRules and .waf.thresholdRules. These are both arrays of objects that have the following data model:

NameTypeApplies toDescription
ruleIDNumBothID of the matched rule
messageStringBothDescription of the matched rule
categoryScoresObjScoringFor scoring rules, an object with the total accumulated scores for each category after the rule has run. The scores reported here therefore include the points contributed by this rule, and also any rules matched prior to this one.
incrementalScoresObjScoringFor scoring rules, an object with the number of points this rule has contributed to each category.
categoryScoreNumThresholdFor threshold rules, the score accumulated in the category associated with the rule.
thresholdNumThresholdFor threshold rules, the score required to trigger the rule. For a threshold rule to be included in the WAF report, it must have been triggered, and therefore categoryScore will always be no less than threshold.

To check that a particular rule was triggered, the transpose() aggregation function can be useful:

events.where(fnName=miss)[0].waf.scoringRules.transpose().ruleID includes 1