Request enrichment

IMPORTANT: The content on this page is written for version 0.7.1 of the fastly crate. If you have previously used this example, your project may be using an older SDK version. View the changelog to learn how to migrate your program.

You need to fetch data from external APIs and add extra headers with additional useful information to the origin

Illustration of concept

WARNING: This information is part of a limited availability release. Portions of this API may be subject to changes and improvements over time. Fields marked deprecated may be removed in the future and their use is discouraged. For more information, see our product and feature lifecycle descriptions.

Requests passing through Fastly can be transformed in many useful ways, and one of the most common is to add information to a request that was not included by the client by appending additional HTTP headers before sending the request on to the backend.

Fastly exposes a variety of information automatically, such as the geolocation and network related information available in Rust via the Geo interface. This information is simple to add to a client request before passing it to an origin:

let client_ip = req.get_client_ip_addr().unwrap();
let geo = geo_lookup(client_ip).unwrap();
let country_code = geo.country_code();
req.set_header("Fastly-Geo-Country", country_code);

There are also countless data sources that can provide valuable information and add intelligence to your applications. If these sources expose an API, we can query it at the edge to enrich requests with the data that service provides. This might be one of your own services, like an A/B testing API, or a third-party service.

For the purpose of this tutorial, we will detect requests that contain passwords, send an API request to the Have I Been Pwned (HIBP) API to check whether the password has been leaked, and then add a header to the request before it is forwarded to the origin.

Have I Been "Pwned"?

"Have I been pwned" (HIBP) is a community service that maintains a database of compromised passwords. It provides an API endpoint that allows passwords to be checked against that database in a privacy-preserving way. This works using a k-anonymity principle, which we can invoke like this:

  1. Take the original, cleartext credential, such as '123456'.
  2. Make a SHA1 hash out of the credential, which results in a 40 character string.
  3. Split the hash into two strings: the first 5 characters, and the remaining 35 characters.
  4. Send the first 5 characters to the HIBP API.
  5. HIBP returns a list of all the SHA1 hashes in its database that begin with those 5 characters.
  6. Use the last 35 characters to determine whether the full hash is in the list.

This mechanism allows a precise trade-off to be made between information leakage and functionality. Let's see how this works using command-line tools:

$ printf '123456' | openssl sha1
(stdin)= 7c4a8d09ca3762af61e59520943dc26494f8941b
$ curl https://api.pwnedpasswords.com/range/7c4a8 | grep -i 'd09ca3762af61e59520943dc26494f8941b'
D09CA3762AF61E59520943DC26494F8941B:24230577

IMPORTANT: While it's possible to use HIBP anonymously, we strongly recommend using an API key. For production use, go to the HIBP API key page and obtain an API key.

Based on the response, we can tell that the password has been reported compromised 24,230,577 times. So '123456' turns out to be a bad idea for a password.

Set up a Rust based C@E project

This tutorial assumes that you have already have the Fastly CLI installed. If you are new to the platform, read our Getting Started guide.

Initialize a project

If you haven't already created a Rust-based Compute@Edge project, run fastly compute init in a new directory in your terminal and follow the prompts to provision a new service using the default Rust starter kit and with httpbin as a backend:

$ mkdir hibp_enrichment && cd hibp_enrichment
$ fastly compute init
Name: [hibp_enrichment]
Description: Check the HIBP API for pwned passwords and send enriched information to the origin
Author: My Name
Language:
[1] Rust
[2] AssemblyScript (beta)
[3] Other ('bring your own' Wasm binary)
Choose option: [1]
Starter kit:
[1] Default (https://github.com/fastly/compute-starter-kit-rust-default.git)
Choose option or type URL: [1]
✓ Initializing...
✓ Fetching package template...
✓ Updating package manifest...
✓ Initializing package...

This will create some files for you, which you'll need to edit as you go through the rest of the tutorial:

  • fastly.toml describes your project and tells the Fastly CLI where to deploy it to.
  • Cargo.toml is the Rust package manifest, where you declare your dependencies.
  • src/main.rs is the source code of your project. This will have some example code in it, which you can remove.

Create a Fastly Compute@Edge service

Create your new Compute@Edge service. Make a note of the service ID that is returned from the following command.

$ fastly service create -n 'my-enrichment-demo'

Add the service ID into the fastly.toml file:

fastly.toml
TOML
# This file describes a Fastly Compute@Edge package. To learn more visit:
# https://developer.fastly.com/reference/fastly-toml/
authors = [...]
description = "Check the HIBP API for compromised passwords and send enriched information to the origin"
language = "rust"
manifest_version = 1
name = "hibp_enrichment"
service_id = "[your_service_id]"

Configure the backends

You'll need two backends for this demo: the HIBP API and your own origin server that will answer the user's request. We'll use httpbin.org as a stand in for our backend, but feel free to substitute your own. You can add these using the fastly backend create command:

$ fastly backend create --service-id=[your_service_id] --name=api --address=api.pwnedpasswords.com --port=443 --use-ssl --version=1
$ fastly backend create --service-id=[your_service_id] --name=primary --address=httpbin.org --port=443 --use-ssl --version=1

Write the code

Dependencies

Add these dependencies to your Cargo.toml file:

Cargo.toml
TOML
fastly = "0.7.0"
sha1 = "^0.6"
percent-encoding = "^2.1.0"

Afterwards, import them at the top of src/main.rs. If you haven't already, remove all the existing content of main.rs and replace it with the following:

main.rs
Rust
use fastly::http::{header, Method, StatusCode};
use fastly::{mime, Error, Request, Response};
use percent_encoding::percent_decode_str;
use sha1::Sha1;

The fastly dependency provides the SDK for the Fastly platform; percent-encoding decodes the percent-encoded password that the user submits in the login form POST; and sha1 to performs the hashing function necessary to be compatible with the HIBP API.

Set up the backends and constants

You created two backends on the Fastly service, called api and primary. Since they are referenced as strings in Rust, assigning them to constants will help to avoid typos later:

main.rs
Rust
// The name of the backend servers associated with this service.
// This must match the backend names you configured using `fastly backend create`.
const BACKEND_APP_SERVER: &str = "primary";
const BACKEND_SECURITY_CHECK: &str = "api";
// Credential prefix length
const PREFIX_LENGTH: usize = 5;

Add helper functions

To find a password in the request, you'll need to be able to parse the request body and extract a field by name. In VCL, we have the subfield function, but there's no equivalent in Rust so now's a good time to write one:

main.rs
Rust
// Helper function to parse for password field
fn sub_field<'a>(content: &'a str, field_name: &str, separator_character: &str) -> Option<&'a str> {
content
.split(separator_character)
.find_map(|sub_field| field_value(sub_field, field_name))
}
fn field_value<'a>(content: &'a str, field_name: &str) -> Option<&'a str> {
let mut i = content.split('=');
let name = i.next()?.trim();
if name == field_name {
let value = i.next()?.trim();
Some(value)
} else {
None
}
}

You'll also need to be able to compute a SHA1 hash of the password for the HIBP API. The SHA1 crate does that but it's helpful to have an easy way to get it as a string:

main.rs
Rust
/// Generate SHA1 hash from string s
fn hash_sha1(s: &str) -> String {
let mut hasher = Sha1::new();
hasher.update(s.as_bytes());
hasher.digest().to_string()
}

Fetch enriched data from the API

A Compute@Edge program in Rust receives a Request. Since the objective here is to create an improved (enriched) request, a good signature for the enrichment function would be Request -> Result<Request, Error>. This enables the enrichment logic to be nicely encapsulated and can be invoked elegantly as part of processing the incoming request, in conjunction with other similar handlers.

The function will therefore take a Request, add a Fastly-Password-Status header to it, and then return it to the calling scope.

main.rs
Rust
// Process login with threat check
fn process_credential(mut req: Request) -> Result<Request, Error> {
// Get body as a string from the request
let body_string = req.take_body_str();
// Get credential from body string
match sub_field(&body_string, "password", "&") {
Some("") | None => {
// No credential or empty credential found
println!("No valid credential is found");
req.set_header("fastly-password-status", "no-credential");
}
Some(plain_cred) => {
// Decode the percent-encoded password
let decoded_plain_cred = percent_decode_str(&plain_cred).decode_utf8().unwrap();
// Generate sha1 hash of credential
let hashed_cred: String = hash_sha1(&decoded_plain_cred).to_uppercase();
// Split the hash of credential to left and right part at position PREFIX_LENGTH
let (hash_left, hash_right) = hashed_cred.split_at(PREFIX_LENGTH);
// Prepare the request for threat check
// (If you use HIBP in production please use an API key)
let api_url = format!("https://api.pwnedpasswords.com/range/{}", hash_left);
let api_req = Request::get(api_url);
// Send threat check request to API with the left-hand-side of the SHA1 hash
let mut api_resp = api_req.send(BACKEND_SECURITY_CHECK)?;
let api_resp_body = api_resp.take_body().into_string();
// Check if the response body contains the right-hand-side of the sha1 hash
let result = if api_resp_body.contains(hash_right) {
"compromised-credential"
} else {
"safe-credential"
};
req.set_header("fastly-password-status", result);
// Uncomment for debugging. For production use, avoid logging credentials
//println!("Checked credential {}, result is {}", plain_cred, result);
}
};
// Return the body string to the request before giving the request back to the main function
req.set_body(body_string);
Ok(req)
}

First, take the request body from the request and use the helper function defined earlier to search it for a credential. For the purposes of this tutorial, we'll assume that the body is application/x-www-form-urlencoded and that the field name we want is always password, so a body that would match would be username=Jo&password=123456. Special characters such as "!" and "$" are often used in passwords, but these special characters will be percent-encoded when a user submits a credential for this tutorial. Therefore, we must decode the percent-encoded password before interacting with the HIBP API.

If there's no credential found in the body, it would be very inefficient to make an unnecessary API request, so in this case, you can create a fast path by adding a Fastly-Password-Status: no-credential header and returning immediately.

Where a credential is found, the other helper function defined earlier can be used to compute a SHA1 hash as a 40 character string. The HIBP API takes a 5-character prefix of that as an input, so divide the hash into a 5-character hash_left and a 35-character hash_right. The request to the API will return a list of 'right hand sides' of all hashes in the database that start with the supplied 'left hand side'. It's then easy enough to check whether hash_right is in the list, and if so, conclude that the credential is compromised.

Use the enrichment function

The entry point for a Compute@Edge progam is the main function. A simple scenario here is to pass every request directly to a backend, and then to return whatever the backend responds with. You need only make a small modification to this - insert a call to the enrichment function, which will modify the request, before you send it to the origin.

main.rs
Rust
#[fastly::main]
fn main(mut req: Request) -> Result<Response, Error> {
// Pass all requests through the credential detection, which
// modifies the request to enrich it with new information
req = process_credential(req)?;
// Send request to the primary origin as normal
req.set_header(header::HOST, BACKEND_APP_SERVER);
Ok(req.send(BACKEND_APP_SERVER)?)
}

Commonly, backends require that the Host header sent in the backend request matches the hostname of the backend. Fastly doesn't modify the Host header by default, so you likely also want to do this.

You now have a complete Compute@Edge program, which receives a Request, enriches it, forwards it to an origin, and then uses the returned Response to reply to the client.

Add a login page

Normally, your backend (the primary backend here) would serve pages that would invite a user to submit a password somehow. But since HTTPBin (the primary backend we are using in this tutorial) doesn't do that, you could, as a convenient way to test the demo, add a pre-canned login page to the application, and store it in your Compute@Edge program. Start by creating a login.html page in the src/ directory:

login.html
html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Compromised password detection demo</title>
</head>
<body>
<form action="/post" method="post">
<div class="container">
<label for="username"><b>Username</b></label>
<input type="text" placeholder="Enter Username" name="username" required />
<label for="password"><b>Password</b></label>
<input type="password" placeholder="Enter Password" name="password" required />
<button type="submit">Login</button>
</div>
</form>
</body>
</html>

Then add a section to the main() function to intercept GET requests to the / path and return the login page instead of forwarding the request to origin.

main.rs
Rust
#[fastly::main]
fn main(mut req: Request) -> Result<Response, Error> {
// For the demo, serve a basic login form on the root path
if req.get_method() == Method::GET && req.get_path() == "/" {
let page_html = include_str!("login.html");
return Ok(Response::from_status(StatusCode::OK)
.with_content_type(mime::TEXT_HTML_UTF_8)
.with_body(page_html));
}
// Pass all requests through the credential detection, which
// modifies the request to enrich it with new information
req = process_credential(req)?;
// Send request to the primary origin as normal
req.set_header(header::HOST, BACKEND_APP_SERVER);
Ok(req.send(BACKEND_APP_SERVER)?)
}

Build and deploy

Congratulations! You now have a mechanism to see if the submitted credentials are part of the known compromised credentials.

$ fastly compute build
✓ Initializing...
✓ Verifying package manifest...
✓ Verifying local rust toolchain...
✓ Building package using rust toolchain...
✓ Creating package archive...
SUCCESS: Built rust package hibp-enrichment (pkg/hibp-enrichment.tar.gz)
$ fastly compute deploy
✓ Initializing...
✓ Reading package manifest...
✓ Fetching latest version...
✓ Validating package...
✓ Cloning latest version...
✓ Uploading package...
✓ Activating version...
✓ Updating package manifest...
Manage this service at:
https://manage.fastly.com/configure/services/XXXXXXXX
View this service at:
https://XXXXX.edgecompute.app
SUCCESS: Deployed package (service XXXXXX, version 1)

Try it out

Navigate to the URL shown under "View this service at" in the output above, and you should see the login page. When you log in, your request will be forwarded to HTTPBin, which simply echos back to you what it received. This enables you to see that the origin server has received an additional header with the credential in it.

Have I Been Pwned Origin Request Enrichment

Try it with no password, with the password '123456', and with something strong and random. You should be able to trigger all three of the possible values of Fastly-Password-Status.

Next Steps

Now that you understand how to use an API request to enrich data that is sent to the origin, you could combine this with other Fastly sources such as proxy description to gain visibility into if a given client is coming from a proxy. You could also add API requests to other 3rd party sources or your own sources, and perform them in parallel.