Merge from develop

docs/TODO.md

@@ -6,10 +6,10 @@
   * should only crawl these pages once
   * but should be included in the links part
 * when sorting links, "uniq" them as well
+* be smart about index.html?
 
 ## Cleanup
 
-* rename Page -> crawlerState
 * helper function to do the whole crawl process
 * document code better
   * test with godoc
@@ -17,7 +17,6 @@
 
 ## Features
 
-* some sort of status during crawl?
 * command line options, flag parsing
 * check RFCs on URL character sets, compliance, etc.
 * packagize parts of app