When getting serious about Search Engine Optimization (SEO) there are a few things that your website should have. One of them consists of a robots.txt file that should be properly compiled and placed on the web server where your domain’s files are located. Why should you have a robots.txt file to begin with? The robots.txt indicates specific information about the url structure of your domain and the way Googlebot should crawl it. It’s important that you create one and do it properly to ensure that your urls from your domain are properly indexed.