Serve robots.txt from the edge
Include full text of robots.txt in VCL, serve as a synthetic response to avoid robots.txt requests hitting your origin.
VCL
Use this solution in your VCL service (click RUN below to test this solution or clone it to make changes):
Compute@Edge
Use this solution in your Compute@Edge service:
- Rust
match (req.get_method(), req.get_path()) { (&Method::GET, "/robots.txt") => { Ok(Response::from_body("User-agent: BadBot\nDisallow: /\n")) } _ => Ok( Response::from_body("The page you requested could not be found") .with_status(StatusCode::NOT_FOUND), ), }