swaits
72cac47961
* outputs status * crawlStatus() function created (split out of docrawl()) * changed from using bool flags to linkType in single httpItem enum, tests updated to match this * unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added * renamed a lot of things from 'page' to 'item' * include refurl in httpItem * []*Location result is now sorted
496 B
496 B
To Do
Bugs
- deal with anchor issue (repeatedly crawling index.html#about, index.html#contact, etc.)
- should only crawl these pages once
- but should be included in the links part
- when sorting links, "uniq" them as well
- be smart about index.html?
Cleanup
- helper function to do the whole crawl process
- document code better
- test with godoc
Features
- command line options, flag parsing
- check RFCs on URL character sets, compliance, etc.
- packagize parts of app