Documentation for Robots File.
How to create your robots.txt file
In this article we go over the steps needed to create the robots.txt file for your website. This...