Advanced distributed crawling system for KAD network
Many distributed hash tables have been proposed recently, but only very few of them have been applied actually. KAD, a Kademlia based DHT, is only an exception which is widely used in eMule peer-to-peer system. The measurement of the network will be able to guidance for the designing of such systems. In order to make the results of measurement more accurate, it is essential to develop a high-performance crawler. In this paper, we introduce a new distributed KAD crawler and apply a new condition for stopping crawling, so that the crawler can collect information faster and more completely. On the basis of this, a crawler is developed and full KAD ID space is crawled. Then on the basis of statistical analysis, we have found some interesting phenomena.
Author's Name: Wu, Q., Chen, X.
Volume: Volume 7
Issues: Issue 3
Keywords: DHT, Distributed crawler, KAD network, Network measurement