Why is so much of the White House site hidden from search engines??


This just strikes me as rather odd. The purpose of robots.txt is to exclude your site from indexing by search engines. This is useful if parts of your site contain data not of general interest. That there are entries in robots.txt is perfectly normal – but 2200 lines?!?