Quantcast
Channel: Performance
Viewing all articles
Browse latest Browse all 447

"Web crawler" increases IIS Worker Process usage.

$
0
0

Dear Members,

Here i have a question regarding the web crawling technique in Asp.net Sites.  I have an asp.net application(Windows Server 2012  withIIS 8.0. ), in which i have been facing a strange issue i.e., CPU usage reaches 99% when no user available to access the site. 

After searching for several option, i found one solution i.e., i have added robot.txt with the following text in the IIS webroot folder.

robots.txt

User-agent: *
Disallow: /

Using this solution googlebot is not getting registered in the IIS logs (CPU usage reduced 50%) . But still the bingbot keep on registering in the IIS log. Is there any to stop the bingbot? 

My another question is,

Does the website crawling (some thing like googlebot) can trigger IIS worker process, that could utilize the CPU usage 99% ?


Viewing all articles
Browse latest Browse all 447

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>