Dear Members,
Here i have a question regarding the web crawling technique in Asp.net Sites. I have an asp.net application(Windows Server 2012 withIIS 8.0. ), in which i have been facing a strange issue i.e., CPU usage reaches 99% when no user available to access the site.
After searching for several option, i found one solution i.e., i have added robot.txt with the following text in the IIS webroot folder.
robots.txt
User-agent: * Disallow: /
Using this solution googlebot is not getting registered in the IIS logs (CPU usage reduced 50%) . But still the bingbot keep on registering in the IIS log. Is there any to stop the bingbot?
My another question is,
Does the website crawling (some thing like googlebot) can trigger IIS worker process, that could utilize the CPU usage 99% ?