add readme

README.md

@@ -0,0 +1,38 @@
+# DoCrawler #
+
+This is a toy web crawler written in Go.
+
+It crawls a single host (i.e. anything.com) and outputs a site map in JSON format.
+
+For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
+
+## Design ##
+
+* The main crawler loop is in `docrawler.go:docrawl()`.
+* From there I maintain a hash of links we've already crawled.
+* Each link gets an `httpItem{}` struct instanced, which holds its crawl state.
+* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter.
+* These goroutines listen for `*httpItem{}` on one channel, crawl it, fill out its results, and send it back to `docrawl()` on another channel.
+* `docrawl()` is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on.
+
+## How do I get set up? ##
+
+After cloning this repository, run:
+
+    make vendor && make
+
+In order to run the tests:
+
+    make test
+
+If you'd like to automatically run tests any time a file is changed:
+
+    make autotest
+
+To deploy, just copy the executable `bin/docrawler` to a destination of your choosing.
+
+*Developed on OS X. Also tested on Windows 7 under MinGW+bash.*
+
+## Who do I talk to? ##
+
+Written by Stephen Waits. Please contact me at <mailto:steve@waits.net> with any questions.