Search engines bots, crawlers and spiders helps your site getting discovered by users. However, some search engines are very aggressive and consume a lot of server bandwidth. And sometimes, bots …