diff --git a/README.md b/README.md new file mode 100644 index 0000000..f64d7f1 --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# DoCrawler # + +This is a toy web crawler written in Go. + +It crawls a single host (i.e. anything.com) and outputs a site map in JSON format. + +For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com). + +## Design ## + +* The main crawler loop is in `docrawler.go:docrawl()`. +* From there I maintain a hash of links we've already crawled. +* Each link gets an `httpItem{}` struct instanced, which holds its crawl state. +* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter. +* These goroutines listen for `*httpItem{}` on one channel, crawl it, fill out its results, and send it back to `docrawl()` on another channel. +* `docrawl()` is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on. + +## How do I get set up? ## + +After cloning this repository, run: + + make vendor && make + +In order to run the tests: + + make test + +If you'd like to automatically run tests any time a file is changed: + + make autotest + +To deploy, just copy the executable `bin/docrawler` to a destination of your choosing. + +*Developed on OS X. Also tested on Windows 7 under MinGW+bash.* + +## Who do I talk to? ## + +Written by Stephen Waits. Please contact me at with any questions.