docrawler

0 Fork 0

T

swaits 1b09e6c204 deal with anchors and index.html "page aliasing: proprerly

* so now http://a.com/index.html and http://a.com/ are understood equivalents
* as are http://a.com/x.html#blog and http://a.com/x.html (etc.)
* this meant a little more bookkeeping in docrawler, but not horrible
* stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping
* and url tests were written to support this

2015-02-17 01:26:20 -08:00

.vendor

add coverage to Makefile (make cover)

2015-02-15 12:40:56 -08:00

docs

deal with anchors and index.html "page aliasing: proprerly

2015-02-17 01:26:20 -08:00

src/docrawler

deal with anchors and index.html "page aliasing: proprerly

2015-02-17 01:26:20 -08:00

util

won't recrawl pages that are the same after stripping #anchor (blah.html#about, blah.html#contact, etc.)