Testing fiddles
The Tests option in a fiddle's request configuration allows you to write test assertions against the instrumentation data that is returned by Fiddle when you press RUN. Writing tests is a good way to express what parts of a fiddle are most important when sharing the solution with someone else, or to express a bug by demonstrating the gap between observed behavior and what you expected to see.
Each test should be written on a single line and be in the following format:
[LABEL] TARGET COMPARISON_TYPE REFERENCE_VALUE
For example:
[Cache TTL is positive] events.where(fnName=fetch)[0].attribs.ttl isMoreThan 0
The optional LABEL
, which must be wrapped in square brackets if present, allows you to define a 'human readable' expression for the test. TARGET
is the thing being tested, REFERENCE_VALUE
is a literal value to compare it with, and COMPARISON_TYPE
is the comparison you want to make.
You can write more than one test per request.
Test targets
Within a fiddle test case, TARGET
should reference a property of the result data, based on the following schema:
Name | Type | Description |
---|---|---|
clientFetch | Obj | The request from the client to Fastly (and the response from Fastly). |
├─ .req | Str | HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body. |
├─ .resp | Str | HTTP response header, contains response status line and response headers (not the body). |
├─ .respType | Str | Parsed Content-type response header value (mime type only). |
├─ .isText | Bool | Whether the response body can be treated as text. |
├─ .isImage | Bool | Whether the response body can be treated as an image. |
├─ .status | Num | HTTP response status. |
├─ .bodyPreview | Str | UTF-8 text preview of the body (truncated at 1K). |
├─ .bodyBytesReceived | Num | Amount of data received. |
├─ .bodyChunkCount | Num | Number of chunks received. |
├─ .complete | Bool | Whether the response is complete. |
└─ .trailers | Str | HTTP response trailers. |
originFetches | Array | Origin fetches made during the request. |
└─ [idx] | Obj | Each fetch is one object. |
├─ .fetchID | Str | Unique ID for this fetch. |
├─ .traceID | Str | ID for the compute instance that triggered this fetch. |
├─ .req | Str | HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body. |
├─ .resp | Str | HTTP response header, contains response status line and response headers (not the body). |
├─ .remoteAddr | Str | Resolved IP address of origin. |
├─ .remotePort | Num | Port on origin server. |
├─ .remoteHost | Str | Hostname of origin server. |
└─ .elapsedTime | Num | Total time spent on origin fetch (ms). |
events | Array | Fastly platform events related to the request, in order of execution. |
└─ [idx] | Obj | Each array element is one event. |
├─ .type | Str | One of vcl-sub , log , fetch or a Compute@Edge event type: ecp-start , ecp-end , geo , dict-open , dict-get |
├─ .time | Num | Unix timestamp in microseconds |
├─ .pop | Str | Three letter code identifying the Fastly POP location in which this event occurred, eg. 'LON', 'IAD', 'SYD'. |
├─ .nodeID | Num | Numeric identifier of the individual server on which this event occurred. |
├─ .reqID | Str | Identifier for this client request. In a fiddle that sends more than one client request, there may be multiple different reqIDs. |
├─ .traceID | Str | ID for the execution flow that triggered this event. In Compute@Edge, this represents the compute instance that is currently handling the request. In VCL, it represents one pass through a VCL state flow (i.e., it resets on a restart , ESI, or HTTP/2 server push) |
├─ .seqIdx | Num | Sequence number of this event with respect to other events with the same traceID and nodeID . |
├─ .parent | Str | If the event is part of a process that was initiated by a different process, this is the traceID of the parent compute process. Normally populated as a result of shielding |
├─ .isAsync | Bool | (vcl-sub only) True if the event happened as part of an asynchronous task. |
├─ .fnName | Str | (vcl-sub only) For VCL events, the VCL subroutine that triggered the event. May be 'recv', 'hash', 'hit', 'miss', 'pass', 'fetch', 'deliver', 'error', or 'log'. |
├─ .logs | Array | (vcl-sub only) Array of log messages logged from this event. |
│ └─ [idx] | Obj | Each array element is one log message. |
│ ├─ .content | Str | The content of the log message |
│ └─ .logName | Str | The name of the log destination, eg. 'stdout' or 'bigquery' |
├─ .waf | Obj | (vcl-sub only) If the Web application firewall ran during this event, this object provides the results from it. |
│ ├─ .didLog | Bool | Whether the WAF generated any log events, which happens if any WAF rules were matched. |
│ ├─ .didBlock | Bool | Whether the WAF acted to block the request from proceeding by triggering an error . |
│ ├─ .didFail | Num | Whether the WAF failed due to a problem evaluating a rule. |
│ ├─ .categoryScores | Obj | Accumulated scores for each of the WAF scoring categories. (see WAF rule data model). |
│ ├─ .scoringRules | Array | A list of matched scoring rules (see WAF rule data model). |
│ └─ .thresholdRules | Array | A list of matched threshold rules (see WAF rule data model). |
├─ .attribs | Obj | Event-specific properties. These are displayed in the Fiddle UI for the event. For example: within miss events, a staleExists property is reported. |
└─ .prevAttribs | Obj | Event-specific properties where the value at the start of the subroutine was different to that recorded at the end. |
logs | Array | Array of strings, messages logged from all events. |
insights | Array | Insight tags for this request. Insight tags identify recommendations or divergences from best practices. For example: [ "client-cc-missing", "invalid-header" ] . |
For example, to target the headers that are sent by Fastly to the backend in an origin request, use the .req
property of the first item in the originFetches
collection:
originFetches[0].req includes "My-Custom-Header"
In some cases targets are more complex. For example, the TTL set in vcl_fetch
is reported as the .ttl
property of the event object created by the vcl_fetch
subroutine, but it's not possible to know ahead of time what index that event has within the events
array. Instead, use a filter function to narrow down events
to just the events associated with the vcl_fetch
subroutine, and then pick the first one:
events.where(fnName=fetch)[0].attribs.ttl isMoreThan 0
HINT: If there is an .attribs
property on an object, Fiddle will treat it as a special case and allow the target expression to directly access the child properties of attribs
, to make targeting event properties easier. The example shown above can therefore also be written as:
events.where(fnName=fetch)[0].ttl isMoreThan 0
The where(fnName=fetch)
is an example of using an aggregation function to transform the test data and make it easier to target the values that you want to test. The available functions are:
Function | Details |
---|---|
listBy(field) Array ➔ Array | Takes an array of objects and makes an array of arrays, where in each sub-array, all objects share the same value of field . Items in the output array are ordered based on the order in which the values of the selected field first appear in the input.e.g. events.listBy(vclflowkey)[1][0].fnName is "recv" |
where(field=val) Array ➔ Array | Takes an array of objects and filters it to leave only those where the property called field has value val .e.g. events.where(fnName=recv)[0].url startsWith "/a/b" |
groupBy(field) Array ➔ Obj | Takes an array of objects and splits it into multiple arrays where each one has the same value for the property field , and organizes the resulting data into an object where the field values are keys.e.g. events.groupBy(fnName).recv[0].url startsWith "/a/b" |
transpose() Array ➔ Obj | Takes an array of objects and makes an object of arrays. Where multiple input objects share the same property name, that property becomes a top level property with an array containing all the values. e.g. events.transpose().return notIncludes "error" |
count() Array ➔ Num | Takes an array and returns the length e.g. originFetches.count() greaterThan 1 |
concat() Array ➔ Str | Takes an array and returns a string representation of all array items joined together, delimited by newlines. e.g. logs.concat() includes "Hello" |
HINT: All timing properties reported by Fiddle are in microseconds - so 1 hour is reported as 3600000000
(3.6 billion microseconds). The most common timing property to target for testing is .ttl
, which is provided on events where fnName
is fetch
.
Lastly, you need to assert something about this target data, which requires a comparison type and a reference value, together making up an assertion.
Assertions
We support the following options for COMPARISON_TYPE
:
Name | Value type | Description |
---|---|---|
is | Any | Non-strict equality (using == ) |
isJSON | None | The target is valid JSON (does not require a reference value, so just two parameters) |
isTrue | None | The target is true |
isAtLeast , isAbove | JSON number | Target is numerically higher than (or at least) the reference value |
isAtMost , isBelow | JSON number | Target is numerically lower than (or at most) the reference value |
includes | Any | The target includes the reference value. Can be used to assert the inclusion of a value in an array, a substring in a string, or a subset of properties in an object. |
matches | JS RegExp | The target matches a regular expression. Regex must be delimited with / and may be followed by modifiers, e.g. /abc/i |
oneOf | JSON array | Checks that the value of the target is equal to at least one of the values in the reference array |
startsWith | JSON string | Checks that the value of the target starts with the reference string |
endsWith | JSON string | Checks that the value of the target ends with the reference string |
All comparisons can be negated by prefixing them with not
, e.g. notOneOf
, isNotJSON
, notMatches
. The word 'is' will be ignored so can be used purely to improve readability.
The REFERENCE_VALUE
must be expressed in a way that can be interpreted as the appropriate type:
- JSON number: A plain number (
3
,5.7
,-42
) - JSON string: A double quoted string (
"foo"
; contained double-quotes must be escaped as\"
) - JSON boolean: The word
true
orfalse
- JSON array: A JSON-parseable array containing any combination of numbers, strings and booleans as defined above (
[1, "foo"]
) - JS RegExp: Either a JavaScript regex literal (
/(foo|bar)/ism
) or the bare pattern, not quoted or delimited (foo|bar
). In the latter case, thes
,m
andi
flags will be applied to the pattern automatically.
Labels
Labelling your tests is optional but helps to illustrate your intent, as well as make test results easier to read. The LABEL
portion of the test expression may contain any character other than square brackets ([]
), and since test expressions must be one line, labels may also not include newline characters.
[Response is OK] clientFetch.status is 200
Example tests
If you want to test... | Try this example |
---|---|
HTTP response status on the client fetch | clientFetch.status is 200 |
Request headers on the client fetch | clientFetch.req notIncludes "Fastly-FF: " |
Response headers on the client fetch | clientFetch.resp includes "Age: " |
HTTP response status of a backend fetch | events.where(fnName=fetch)[0].status isOneOf [301,302,307,308] |
Request headers on a backend fetch | originFetches[0].req includes "GET /" |
Response headers on a backend fetch | originFetches[0].resp matches /\w+@gmail\.com/ |
TTL set in a vcl_fetch subroutine | events.where(fnName=fetch)[0].ttl isLessThan 86400000000 |
URL of the request following a VCL restart | events.where(fnName=recv)[1].url is "/foo" (After a restart , vcl_recv will run a second time, so the [1] targets the second occurrence |
Number of origin fetches | originFetches.count() is 2 |
Response body from Fastly to client | clientFetch.bodyPreview contains "<meta" (You will only get the first 1K of the response) |
Value of fastly.info_state in deliver | events.where(fnName=deliver)[0].state is "HIT" |
Number of times deliver is run | events.where(fnName=deliver).count() is 1 |
Debugging tests
When a test fails, Fiddle will display a representation of the observed, 'actual' value of the target, so you can see why it didn't match. That observed value will often be undefined
because the target didn't match anything. In that case, it can be helpful to shorten your target expression, to reveal the shape of the data structure and allow you to see what you can target:
Testing best practices
When constructing tests for your fiddle, consider the following best practices:
- Label your tests to better illustrate your intent
- Try to avoid asserting the nature of the implementation of the solution. Instead, assert the achievement of the overall goal. It should be possible to change the way the solution is achieved without having to change the tests.
- For solutions that act on the user's IP address, for example to block users from certain regions from accessing certain content, use the Client IP override in request options to ensure that the test results are reliably reproducible.
- Fiddle data arrives asynchronously, but not all instrumentation delays are created equal! Tests that target
clientFetch
will be fastest. Those that targetoriginFetches
will be slightly slower, andevents
is slowest. Fiddle will keep repeating your tests until they pass or a timeout is reached, to allow async data to arrive. Speed up your fiddle by usingclientFetch
andoriginFetch
assertions instead of testingevents
orlogs
.
WAF rule data model
The .waf
property of a VCL flow event contains details of the execution of the Web application firewall. WAF rules are classified as scoring or threshold and also into one of the following categories:
sql
: SQL injectionrfi
: Remote file inclusionlfi
: Local file inclusionrce
: Remote command executionphp
: PHP injectionsession
: Session hijackinghttp
: HTTP violationxss
: Cross-site scripting
The .waf.categoryScores
property provides total scores for each category plus anomaly
, a total of all categories. This is a good way to test that a request triggers a rule in a particular category:
events.where(fnName=miss)[0].waf.categoryScores.xss isAbove 0
If you simply want to know whether the WAF was triggered at all, use one of the top level properties of the .waf
data block:
events.where(fnName=miss)[0].waf.didBlock isTrue
Individual matched rules are reported in .waf.scoringRules
and .waf.thresholdRules
. These are both arrays of objects that have the following data model:
Name | Type | Applies to | Description |
---|---|---|---|
ruleID | Num | Both | ID of the matched rule |
message | String | Both | Description of the matched rule |
categoryScores | Obj | Scoring | For scoring rules, an object with the total accumulated scores for each category after the rule has run. The scores reported here therefore include the points contributed by this rule, and also any rules matched prior to this one. |
incrementalScores | Obj | Scoring | For scoring rules, an object with the number of points this rule has contributed to each category. |
categoryScore | Num | Threshold | For threshold rules, the score accumulated in the category associated with the rule. |
threshold | Num | Threshold | For threshold rules, the score required to trigger the rule. For a threshold rule to be included in the WAF report, it must have been triggered, and therefore categoryScore will always be no less than threshold . |
To check that a particular rule was triggered, the transpose()
aggregation function can be useful:
events.where(fnName=miss)[0].waf.scoringRules.transpose().ruleID includes 1