* the json marshaller cannot deal with circular data structures, and no automatic .String() conversion
* so we copy our crawl data slice into another slice which will be marshaled into json more easily
* made the page slice we use during crawling actually circular (as it should be) rather than putting duplicates in for each page discovered
* test for conversion to output data structure, and hacky cheater test for json output
* removed Base *Page from our Page struct as it's only needed temporarily
* moved Page{} and NewPage() into page.go
* queued and crawled are both maps now. used to track which pages we already know about
* tests updated to match the multipage crawling
* added another file to the testsite (scripts/blah.js)
* fetch() renamed to fetchPage. now takes *Page instead of url string
* fetch_filetype() renamed to fetchFiletype. now takes *Page instead of url string
* parse() renamed to parseLinks()
* tests updated appropriately
* minor cleanup (lint)
* NewPage added for conveniently creating new *Page structs from url strings
* refined Page struct. now need to refactor everything else to use this
* wrote docrawl() function. NOT intended to be anything like the final design. just sketching it out to get a better idea of needs, design, etc.
* replaced fmt.Print* with t.Log* in tests (oops!)
* added a header fetcher function to get the file type (since we only want to parse html (with tests)
* added small test for the URL resolver where the base URL is empty
* added placeholder files in testsite/ for about.html and image.png
* moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly)
* updated idea in Design.md
* renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming
* parse now sorts its results. test updated.
* moved regexp compilation to init instead of every time parse() is called
* misc comments and cleanup
* added test for fetching a known file and verifying its content matches
* add test for parsing a known string and verify URLs are extracted
* add test for a request which doesn't exist and verify 404 is received (canary test)
* add comments throughout test (for each test function)
* wrote (stupidly) simple fetcher which takes a URL, GETs it, and returns the body and error
* wrote (stupidly) simple parser which takes a string, parses out (src|href|xhref)="...", and returns a slice of the captured links
* added needed html files
swaits