DoCrawler
This is a toy web crawler written in Go.
It crawls a single host (i.e. anything.com) and outputs a site map in JSON format.
For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
Design
- The main crawler loop is in
docrawler.go:docrawl(). - From there I maintain a hash of links we've already crawled.
- Each link gets an
httpItem{}struct instanced, which holds its crawl state. - A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter.
- These goroutines listen for
*httpItem{}on one channel, crawl it, fill out its results, and send it back todocrawl()on another channel. docrawl()is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on.
How do I get set up?
After cloning this repository, run:
make vendor && make
In order to run the tests:
make test
If you'd like to automatically run tests any time a file is changed:
make autotest
To deploy, just copy the executable bin/docrawler to a destination of your choosing.
Developed on OS X. Also tested on Windows 7 under MinGW+bash.
Who do I talk to?
Written by Stephen Waits. Please contact me at mailto:steve@waits.net with any questions.