detecting nepotistic links by language model disagreement
In this short note we demonstrate the applicability of hyperlink downweighting by means of language model disagreement. The method filters out hyperlinks with no relevance to the target page without the need of white and blacklists or human interaction. We fight various forms of nepotism such as common maintainers, ads, link exchanges or misused affiliate programs. Our method is tested on a 31 M page crawl of the .de domain with a manually classified 1000-page random sample.
links links language modeling language modeling demonstrate applications hyperlink method filtering relevance paging human interaction program crawl classifiation random sample link spam language model linking anchor text