Serve robots.txt from the edge

Include full text of robots.txt in VCL, serve as a synthetic response to avoid robots.txt requests hitting your origin.


Use this solution in your VCL service (click RUN below to test this solution or clone it to make changes):


Use this solution in your Compute@Edge service:

  1. Rust
match (req.get_method(), req.get_path()) {
(&Method::GET, "/robots.txt") => {
Ok(Response::from_body("User-agent: BadBot\nDisallow: /\n"))
_ => Ok(
Response::from_body("The page you requested could not be found")