* Broken field in Page{} now,
* anything that fails to fetch (with http 200) or parse (should not happen) is considered broken
* tests updated appropriately
* discovered my "image.png" file was in the wrong directory because it showed up as broken. fixed!
* bug fix: only fetch if type=text/html (after HEAD response)
* added Skipped field to Page to remember this state
* marshaled into "Remote" field in JSON
* bug fix: skipped (etc.) sites weren't getting added to the main page list
* tests had to be reworked. because the crawl is non-deterministic, the tests started breaking. tests made a bit more robust, but are slightly weaker now.
* the lists of URLs in the output are now sorted
* ... and the parseLinks is no longer doing any sorting (no point in it)
* parser tests updated accordingly
* added a "broken" link to index.html - not currently supported, will fix next!
* the json marshaller cannot deal with circular data structures, and no automatic .String() conversion
* so we copy our crawl data slice into another slice which will be marshaled into json more easily
* made the page slice we use during crawling actually circular (as it should be) rather than putting duplicates in for each page discovered
* test for conversion to output data structure, and hacky cheater test for json output
* removed Base *Page from our Page struct as it's only needed temporarily
* moved Page{} and NewPage() into page.go
* queued and crawled are both maps now. used to track which pages we already know about
* tests updated to match the multipage crawling
* added another file to the testsite (scripts/blah.js)
* fetch() renamed to fetchPage. now takes *Page instead of url string
* fetch_filetype() renamed to fetchFiletype. now takes *Page instead of url string
* parse() renamed to parseLinks()
* tests updated appropriately
* minor cleanup (lint)
* NewPage added for conveniently creating new *Page structs from url strings
* refined Page struct. now need to refactor everything else to use this
* wrote docrawl() function. NOT intended to be anything like the final design. just sketching it out to get a better idea of needs, design, etc.
* replaced fmt.Print* with t.Log* in tests (oops!)
* added a header fetcher function to get the file type (since we only want to parse html (with tests)
* added small test for the URL resolver where the base URL is empty
* added placeholder files in testsite/ for about.html and image.png
swaits