Robots.txt

Overview The Robot Exclusion Standard, Robots Exclusion Protocol or robots.txt protocol is a suggestion for "honest" browsers which should prevent web crawlers from accessing the parts of a website listed in the roboos.txt file. Discovery Methodology Browse to the robots.txt file in Mutillidae and read the contents Exploitation Follow the paths in the robots.txt file to see if any sensitive directories or files are exposed. Try to list the contents of directories since servers will sometimes be misconfigured to show directory contents. Example Robots.txt is located at http://[server]/mutillidae/robots.txt with default installation. On Samurai WTF the path will be http://mutillidae/robots.txt.