Answer by LuckyBrain for how to check if my website is being accessed using a...
I had a similar issue in my web application because I created some bulky data in the database for each user that browsed into the site and the crawlers were provoking loads of useless data being...
View ArticleAnswer by jbcurtin for how to check if my website is being accessed using a...
This question is a great place to start:Detecting 'stealth' web-crawlersOriginal post:This would take a bit to engineer a solution.I can think of three things to look for right off the bat:One, the...
View ArticleAnswer by Venatu for how to check if my website is being accessed using a...
You can use the robots.txt file to block access to crawlers, or you can use javascript to detect the browser agent, and switch based on that. If I understood the first option is more appropriate,...
View Articlehow to check if my website is being accessed using a crawler?
how to check if a certain page is being accessed from a crawler or a script that fires contineous requests?I need to make sure that the site is only being accessed from a web browser.Thanks.
View Article
