Yahoo reduces Load on Websites from its Crawlers
Yahoo Search engine has reported major changes in its crawler program which keeps visiting websites to gather fresh data. As per the official statement issued by Yahoo Search Engineer Priyank Garg, the company has implemented major changes in their crawlers. In addition, yahoo will reduce the number of machines which were crawling websites.
The statement further added, "We have been rolling out fresh updates to our crawl, index and ranking systems -- ones that are relatively significant for webmasters. We are reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days."
Yahoo has taken these steps after webmasters suggested them to reduce the activity of Yahoo crawlers on their websites.
Yahoo recently added support for 'Dynamic URL Rewriting' in its Site Explorer.
The new crawler is more efficient and will significantly reduce load on websites. Webmasters operating websites with lot of information and pages were having lot of bandwidth cost due to excessive crawling by Yahoo.