The website crawl provides us a wealth of knowledge, but to just take this audit to the following stage, we have to seek advice from the search engines. However, engines like google do not like to offer unrestricted use of their servers so we will just have to accept https://www.youtube.com/watch?v=EVeTP0aV8gY