* crawled is now a map to httpItem's (instead of struct{}) so we can connect pages mid-crawl
* added duplicate link to html site, test updated to match
* now any httpItem with linkType == tUnknown is listed as "Broken" in the output
* outputs status
* crawlStatus() function created (split out of docrawl())
* changed from using bool flags to linkType in single httpItem enum, tests updated to match this
* unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added
* renamed a lot of things from 'page' to 'item'
* include refurl in httpItem
* []*Location result is now sorted
* Broken field in Page{} now,
* anything that fails to fetch (with http 200) or parse (should not happen) is considered broken
* tests updated appropriately
* discovered my "image.png" file was in the wrong directory because it showed up as broken. fixed!
* bug fix: only fetch if type=text/html (after HEAD response)
* added Skipped field to Page to remember this state
* marshaled into "Remote" field in JSON
* bug fix: skipped (etc.) sites weren't getting added to the main page list
* tests had to be reworked. because the crawl is non-deterministic, the tests started breaking. tests made a bit more robust, but are slightly weaker now.
* the lists of URLs in the output are now sorted
* ... and the parseLinks is no longer doing any sorting (no point in it)
* parser tests updated accordingly
* added a "broken" link to index.html - not currently supported, will fix next!
* the json marshaller cannot deal with circular data structures, and no automatic .String() conversion
* so we copy our crawl data slice into another slice which will be marshaled into json more easily
* made the page slice we use during crawling actually circular (as it should be) rather than putting duplicates in for each page discovered
* test for conversion to output data structure, and hacky cheater test for json output
* removed Base *Page from our Page struct as it's only needed temporarily
* moved Page{} and NewPage() into page.go
* queued and crawled are both maps now. used to track which pages we already know about
* tests updated to match the multipage crawling
* added another file to the testsite (scripts/blah.js)
* fetch() renamed to fetchPage. now takes *Page instead of url string
* fetch_filetype() renamed to fetchFiletype. now takes *Page instead of url string
* parse() renamed to parseLinks()
* tests updated appropriately
* minor cleanup (lint)
* NewPage added for conveniently creating new *Page structs from url strings
* refined Page struct. now need to refactor everything else to use this
* wrote docrawl() function. NOT intended to be anything like the final design. just sketching it out to get a better idea of needs, design, etc.
* replaced fmt.Print* with t.Log* in tests (oops!)
* added a header fetcher function to get the file type (since we only want to parse html (with tests)
* added small test for the URL resolver where the base URL is empty
* added placeholder files in testsite/ for about.html and image.png
* moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly)
* updated idea in Design.md
* renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming
* parse now sorts its results. test updated.
* moved regexp compilation to init instead of every time parse() is called
* misc comments and cleanup
* added test for fetching a known file and verifying its content matches
* add test for parsing a known string and verify URLs are extracted
* add test for a request which doesn't exist and verify 404 is received (canary test)
* add comments throughout test (for each test function)
* wrote (stupidly) simple fetcher which takes a URL, GETs it, and returns the body and error
* wrote (stupidly) simple parser which takes a string, parses out (src|href|xhref)="...", and returns a slice of the captured links
* added needed html files
swaits