Spyder

Stuart at work has been playing with a nice Python web crawler recently. I’ve used Harvestman before but it’s not the most straightforward thing to work with. Spyder has a really nifty callback based approach and a couple of hooks which allows you to write code like:

pre. crawler = Spyder() crawler.register(my_custom_method, ‘post_fetch_html’) crawler.run(URL)

On a side note I wish Launchpad was as clean and tidy as GitHub though. I can see GitHub adding some of the features that Launchpad has eventually, but I hope they fit them in around the existing features.