Modifying robots.txt

Is there a way to modify robots.txt?

Crawling at 1 req/second is quite high for our application as we don’t get that much traffic.

I also notice a lot of crawlers hitting token/* and auth/start/

Are they of any use to crawlers?

To be honest I’ve done this kind of stuff with an Apache alias.

E.g.

<VirtualHost *:443>
...
    Alias /robots.txt /path/to/your/other/robots.txt

    ...

     RewriteEngine on
     RewriteRule ^/robots.txt     - [L]
     RewriteRule ^(.*)$ /index.php?__path__=$1 [B,L,QSA]

     ...

</VirtualHost>

Note that you have introduce a new RewriteRule for that alias.

Then reloaded Apache.

No.