* so now http://a.com/index.html and http://a.com/ are understood equivalents
* as are http://a.com/x.html#blog and http://a.com/x.html (etc.)
* this meant a little more bookkeeping in docrawler, but not horrible
* stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping
* and url tests were written to support this
* crawled is now a map to httpItem's (instead of struct{}) so we can connect pages mid-crawl
* added duplicate link to html site, test updated to match
* now any httpItem with linkType == tUnknown is listed as "Broken" in the output
* outputs status
* crawlStatus() function created (split out of docrawl())
* changed from using bool flags to linkType in single httpItem enum, tests updated to match this
* unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added
* renamed a lot of things from 'page' to 'item'
* include refurl in httpItem
* []*Location result is now sorted
* Broken field in Page{} now,
* anything that fails to fetch (with http 200) or parse (should not happen) is considered broken
* tests updated appropriately
* discovered my "image.png" file was in the wrong directory because it showed up as broken. fixed!
* bug fix: only fetch if type=text/html (after HEAD response)
* added Skipped field to Page to remember this state
* marshaled into "Remote" field in JSON
* bug fix: skipped (etc.) sites weren't getting added to the main page list
* tests had to be reworked. because the crawl is non-deterministic, the tests started breaking. tests made a bit more robust, but are slightly weaker now.
* the lists of URLs in the output are now sorted
* ... and the parseLinks is no longer doing any sorting (no point in it)
* parser tests updated accordingly
* added a "broken" link to index.html - not currently supported, will fix next!
swaits