REDbot has been getting a lot of load recently, but some of it isn’t what it was designed for.
Some people seem to be using it for Referer spam, especially to major search engines. Because they’re doing this from a script, it overloads the public instance of REDbot, making it unavailable for other people.
So, RED now checks site robots.txt files before accessing them.
If you want to be able to check a site with REDbot, make sure you allow the User-Agent “RED.”