Serve robots.txt from the edge

Include full text of robots.txt in VCL, serve as a synthetic response to avoid robots.txt requests hitting your origin.

VCL

Use this solution in your VCL service (click RUN below to test this solution or clone it to make changes):

Compute@Edge

Use this solution in your Compute@Edge service:

  1. Rust
match (req.get_method(), req.get_path()) {
(&Method::GET, "/robots.txt") => {
Ok(Response::from_body("User-agent: BadBot\nDisallow: /\n"))
}
_ => Ok(
Response::from_body("The page you requested could not be found")
.with_status(StatusCode::NOT_FOUND),
),
}