Ahrefs allows you locate the best words to work with on your web site so that folks can easily obtain you on search engines like google. txt file is then parsed and will instruct the robot regarding which pages will not be to be crawled. Like a online search engine https://www.youtube.com/watch?v=0eLlfoHTk_E