1. Home
2. Solutions
3. Tutorials

Instructions

The principle of A/B testing is that you have one or more tests (aspects that you are testing, such as "how many articles should be displayed on each page"). Each test has two or more buckets (such as "10", "15" and "20"). In each test, the visitor should be randomly assigned to a bucket, taking into account whether some buckets are weighted more than others. When running more than one test at the same time, you must ensure that the tests don't influence each other. The same visitor, once assigned to a set of buckets, should stick with them, so that they don't perceive that the website is changing for no reason.

Starting point

For this tutorial we'll assume you're starting with a Compute JavaScript project that simply forwards the client request to a named backend:

/// <reference types="@fastly/js-compute" />

addEventListener("fetch", (event) => event.respondWith(handleRequest(event)));
async function handleRequest(event) {
  const req = event.request;

  // Send the request to `origin_0`.
  const backendResponse = await fetch(req, {
    backend: "origin_0"
  });

  return backendResponse;
}

Give the visitor an ID

Consistently mapping a visitor to the same set of A/B test buckets depends on the visitor having the same identity from one request to the next. If you are already doing authentication in your Fastly service, you can use the ID that you get from that. If you want to include anonymous visitors in your A/B testing, you'll likely still need to create a tracking ID for them.

Before making the fetch to the backend, check for an existing usable ID from a cookie. The Request object doesn't provide a standard interface for interacting with individual cookies, but you can parse it like this:

7 lines
  // Parse cookies from the "Cookie" header
  const cookies = (req.headers.get('Cookie') ?? '')
    .split(';')
    .reduce((acc, val) => {
      const segment = val.trim();
      const pos = segment.indexOf('=');
      if(pos !== -1) {
        const key = segment.slice(0, pos);
        const value = decodeURIComponent(segment.slice(pos + 1));
        acc[key] = value;
      }
      return acc;
    }, {});
36 lines

Check the cookie named ab_cid, and if a value doesn't exist, generate one using the uuid package.

1 lines
import { v4 as uuidV4 } from 'uuid';
19 lines
  // Allocate the visitor a unique identifier if they don't already have one
  let newCookie = false;
  let abTestVisitorId = cookies['ab_cid'];
  if(abTestVisitorId == null) {
    abTestVisitorId = uuidV4();
    newCookie = true;
  }
28 lines

If the cookie wasn't included in the request, you'll want to include one with the response, to identify future visits from the same visitor. After the fetch, and before the response goes to the browser, modify the response by adding a Set-Cookie header:

49 lines
  if(newCookie) {
    backendResponse.headers.append('Set-Cookie', `ab_cid=${encodeURIComponent(abTestVisitorId)}; Max-Age: 31536000; Path=/; Secure; HttpOnly`);
    backendResponse.headers.set('Cache-Control', 'no-store');
  }
3 lines

Remove the visitor ID from origin requests

The ID that you are using to identify the visitor for the purposes of A/B testing shouldn't be sent to your origin servers, because if the origin server were to generate different content based on the ID, rather than the bucket allocation, then Fastly can't cache the resulting output efficiently. To remove that possibility, add the following code before the fetch to the origin:

29 lines
  if(cookies['ab_cid'] != null) {
    // If request contained A/B cookie, then we hide it from the origin
    delete cookies['ab_cid'];
    const reqCookie = Object.entries(cookies)
      .map(([name, value]) => {
        return name + '=' + encodeURIComponent(value);
      })
      .join('; ');
    if(reqCookie !== '') {
      req.headers.set('Cookie', reqCookie);
    } else {
      req.headers.delete('Cookie');
    }
  }
13 lines

Define some tests

Now it's time to define the tests you want to run:

• The name of each test
• For each test, a number of buckets, each with a value and the relative weighting

You can lay out this information using a simple JavaScript object.

3 lines
const AB_TESTS = {
  itemcount: [
    { value: '10', weight: 50 },
    { value: '15', weight: 50 },
  ],
  buttonsize: [
    { value: 'small', weight: 50 },
    { value: 'medium', weight: 25 },
    { value: 'large', weight: 25 },
  ],
};
67 lines

The code above shows an object with a key for each test, where each value is an array of buckets for each test. In this example, the tests are named itemcount and buttonsize.

For each test, define the buckets by specifying the value (a string) and their relative weight (a number) for each. The test object above contains a test called buttonsize with three buckets, where you want 50% of visitors in the first, 25% in the second, and 25% in the third. The relative weights in this example are written to add to 100 to match their percentages, but they don't need to.

Apply the test to the current visitor

Add a function that can map the current visitor to their selected bucket in each test. Unlike VCL, the Compute platform gives you access to loop constructs and object enumeration. You can iterate over the tests you defined:

15 lines
function applyTestToVisitor(visitorId) {
  const results = {};
  for(const [testId, entries] of Object.entries(AB_TESTS)) {

    /*
      Find the bucket that the visitor belongs to for this test (next step)
    */

    results[testId] = bucket.value;
  }
  return results;
}
54 lines

For each test, you'll need to randomly assign a bucket to the visitor. By seeding the random number generator using the combination of the visitor ID and test name, the random selection will be deterministic, so you can avoid having to remember the selections per visitor.

2 lines
import seedrandom from 'seedrandom';                              // 1
15 lines
  for(const [testId, entries] of Object.entries(AB_TESTS)) {      // 2
    const rng = seedrandom(visitorId + testId);                   // 3

    const weightsTotal =
      entries.reduce((acc, entry) => acc + entry.weight, 0);      // 4
    let random = rng() * weightsTotal;                            // 5
    const bucket = entries.find(entry => {                        // 6
      random = random - entry.weight;
      return random < 0;
    });

    results[testId] = bucket.value;                               // 7
  }
62 lines

1. Import the seedrandom package, a pseudorandom number generator that can be seeded.
2. Repeat the following steps for each test.
3. visitorId + testId combines the visitor ID with the name of the test. Use this to seed the random number generator.
4. For this test, add up the weight values of all the buckets to find the total.
5. Generate a random number between 0 and this total, by calling the random number generator to return a number between 0 and 1, and multiplying that number by the total.
6. Use this number to find the corresponding bucket. You can do this by iterating the buckets for the test, successively subtracting the weight value of each bucket from the random number generated above until the value falls below 0.
7. Add the bucket's value to the results object using the test name as the key.

At the end of this, result is an object that contains the results for each test for this visitor, e.g.:

{
  "itemcount": "10",
  "buttonsize": "small"
}

Send test allocation to origin in an HTTP header

For your origin server to be able to act on the test allocations, you need to add the allocation data to the requests to the origin server. It's important to include only one test's allocation in each header, so that Fastly can store content with the minimum number of permutations.

Before the fetch call, insert the following:

75 lines
  const testResults = applyTestToVisitor(abTestVisitorId);

  for(const [testId, result] of Object.entries(testResults)) {
    req.headers.set(`Fastly-ABTest-${testId}`, result);
  }
13 lines

Use the allocations on your origin server

The information about the bucket allocations is presented to your origin server as HTTP headers, prefixed by Fastly-ABTest-. In the examples above, you created tests called itemcount and buttonsize, so you will see two headers with their corresponding values, e.g.: Fastly-ABTest-itemcount: 10 and Fastly-ABTest-buttonsize: small. You can use this information to adjust the response you generate.

It's vital that you tell Fastly which tests you used to determine the content of the page, so we can cache multiple variants of the response where appropriate. Sometimes, the visitor might request a resource that is not affected by any A/B tests - perhaps an image - and in this case you don't need to do anything. We will cache just one copy of the resource, and use it to satisfy all requests. However, if you do use any Fastly-ABTest- headers to decide what to output, then you need to tell us that you did this, using a Vary header:

Vary: Fastly-ABTest-buttonsize

In this case, you're saying that in producing the response, you checked for the test defined in the Fastly-ABTest-buttonsize header. For this test, Fastly needs to keep 3 copies of this resource, one for each of the possible button size buckets (small, medium and large). However, you're also implicitly saying that the page does not have a list of items on it and therefore the itemcount test is irrelevant.

To indicate that you used multiple tests in generating the page, just comma-separate the header names in your Vary header:

Vary: Fastly-ABTest-itemcount, Fastly-ABTest-buttonsize

Now, Fastly needs to store six variations of the resource: two itemcount possibilities, for each of three buttonsize possibilities (2 x 3 = 6). Clearly, the more permutations, the more variants must be stored.

Next steps

Where you include a Vary header for an A/B test on the response, you could consider adding a surrogate key as well. This will allow pages that have been subject to particular tests to be purged without affecting other resources (e.g., if you decided to make the 'large' buttonsize even larger. However, since the bucket allocation and bucket value are part of the request and can be considered by the cache, you don't need to purge in order to update either your weightings or bucket values (e.g., if you decided to introduce a new itemcount of 25).

See also

Blog posts:

• A/B testing at the edge (covers other techniques)
• GOV.UK guide to how A/B testing works (customer blog)

Quick install

The embedded fiddle below shows the complete solution. Feel free to run it, and click the INSTALL tab to customize and upload it to a Fastly service in your account:

Once you have the code in your service, you can further customize it if you need to.

All code on this page is provided under both the BSD and MIT open source licenses.