I was looking in my hitlogs and I noticed that Yanga WorldSearch Bot v1.1/beta was fetching a whole lot of pages and (ab)using a truckload of bandwidth.
I searched around and found blogs with debates about it being “legit” or not.
It’s legit
The Yanga gang do actually have a search engine interface where you can search among the pages they crawl. A few searches on things like my own name did produce results. The evidence suggests that this is in fact a useful crawler which is used to provide a public service.
..and misbehaved
It must be mentioned that this crawler ate more than four thousand pages off one website during the last 24 hours. That really is a whole lot of pages. The logs further indicate that their crawler is very stupid and unpolite.
I’ll allow Yanga for now, and I recommend allowing it since it does appear to be useful – if your server can handle the immense load it puts on it when eating pages. Supporting alternatives to the heavily-censored Google search-engine is a good thing. I do, however, recommend that those who host sites with heavy PHP/MySQL usage on weak servers just -j DROP their IPs as it does strain servers to the point where users may notice a slowdown.
Sphere: Related Content
Explore posts in the same categories: Web spiders
anonymous
