1. Home
  2. Solutions
  3. Examples

Serve robots.txt from the edge

Include full text of robots.txt in VCL, serve as a synthetic response to avoid robots.txt requests hitting your origin.

VCL

Use this solution in your VCL service (click RUN below to test this solution or clone it to make changes):

Compute@Edge

Use this solution in your Compute@Edge service:

  1. Rust
    match (req.get_method(), req.get_path()) {
        (&Method::GET, "/robots.txt") => {
            Ok(Response::from_body("User-agent: BadBot\nDisallow: /\n"))
        }
        _ => Ok(
            Response::from_body("The page you requested could not be found")
                .with_status(StatusCode::NOT_FOUND),
        ),
    }