39 lines
1.5 KiB
Markdown
39 lines
1.5 KiB
Markdown
# DoCrawler #
|
|
|
|
This is a toy web crawler written in Go.
|
|
|
|
It crawls a single host (i.e. anything.com) and outputs a site map in JSON format.
|
|
|
|
For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
|
|
|
|
## Design ##
|
|
|
|
* The main crawler loop is in `docrawler.go:docrawl()`.
|
|
* From there I maintain a hash of links we've already crawled.
|
|
* Each link gets an `httpItem{}` struct instanced, which holds its crawl state.
|
|
* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter.
|
|
* These goroutines listen for `*httpItem{}` on one channel, crawl it, fill out its results, and send it back to `docrawl()` on another channel.
|
|
* `docrawl()` is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on.
|
|
|
|
## How do I get set up? ##
|
|
|
|
After cloning this repository, run:
|
|
|
|
make vendor && make
|
|
|
|
In order to run the tests:
|
|
|
|
make test
|
|
|
|
If you'd like to automatically run tests any time a file is changed:
|
|
|
|
make autotest
|
|
|
|
To deploy, just copy the executable `bin/docrawler` to a destination of your choosing.
|
|
|
|
*Developed on OS X. Also tested on Windows 7 under MinGW+bash.*
|
|
|
|
## Who do I talk to? ##
|
|
|
|
Written by Stephen Waits. Please contact me at <mailto:steve@waits.net> with any questions.
|