swaits
1b09e6c204
* so now http://a.com/index.html and http://a.com/ are understood equivalents * as are http://a.com/x.html#blog and http://a.com/x.html (etc.) * this meant a little more bookkeeping in docrawler, but not horrible * stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping * and url tests were written to support this
297 B
297 B
To Do
Bugs
Cleanup
- helper function to do the whole crawl process
- document code better
- test with godoc
- omit fields in json when nil? (Broken, etc.)
Features
- command line options, flag parsing
- check RFCs on URL character sets, compliance, etc.
- packagize parts of app