REDbot and robots.txt

REDbot has been getting a lot of load recently, but some of it isn’t what it was designed for.

Some people seem to be using it for Referer spam, especially to major search engines. Because they’re doing this from a script, it overloads the public instance of REDbot, making it unavailable for other people.

So, RED now checks site robots.txt files before accessing them.

If you want to be able to check a site with REDbot, make sure you allow the User-Agent “RED.”



Long overdue, RED now support HTTP over SSL and TLS, thanks to support in Thor, the underlying HTTP library.

For example, try Google.

At the moment, it treats HTTPS just like HTTP, but over time we’ll add checks for certificate validity and other HTTPS-specific concerns.

As always, please report any concerns, either using the feedback button on the home page, or in REDbot’s issues list.