5fb78a041e
add sh and zsh targets to make file to conveniently get at the same environment as make; moved cover.out to $/tmp, updated .hgignore to match
swaits2015-02-20 05:31:08 -08:00
7c943fee07
very minor banner change
swaits2015-02-19 11:29:30 -08:00
3e35fdb488
some cleanup and refactoring * moved crawlItem into fetch.go * renamed fetchPage fetchItem * doCrawl now takes number of workers as a parameter * command line flag parsing added
swaits2015-02-17 10:36:48 -08:00
6c9434e39a
flow: Created branch 'feature/cleanup_and_refactoring'.
swaits2015-02-17 09:53:28 -08:00
1b09e6c204
deal with anchors and index.html "page aliasing: proprerly * so now http://a.com/index.html and http://a.com/ are understood equivalents * as are http://a.com/x.html#blog and http://a.com/x.html (etc.) * this meant a little more bookkeeping in docrawler, but not horrible * stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping * and url tests were written to support this
swaits2015-02-17 01:26:20 -08:00
4e335cfc8a
reviewed coverage report. remove some dead code. add a few comments to areas of code unlikely to hit.
swaits2015-02-17 00:44:39 -08:00
db314ec938
flow: Created branch 'feature/fix_anchor_duplication'.
swaits2015-02-17 00:36:58 -08:00
2c2cfc126c
links are now uniq'd before marshaling * crawled is now a map to httpItem's (instead of struct{}) so we can connect pages mid-crawl * added duplicate link to html site, test updated to match * now any httpItem with linkType == tUnknown is listed as "Broken" in the output
swaits2015-02-16 22:47:56 -08:00
7e18005473
remove page.go (merge miss from /develop)
swaits2015-02-16 22:24:59 -08:00
72cac47961
docrawl() completely refactored to use goroutine per URL * outputs status * crawlStatus() function created (split out of docrawl()) * changed from using bool flags to linkType in single httpItem enum, tests updated to match this * unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added * renamed a lot of things from 'page' to 'item' * include refurl in httpItem * []*Location result is now sorted
swaits2015-02-16 22:13:52 -08:00
2e99708417
won't recrawl pages that are the same after stripping #anchor (blah.html#about, blah.html#contact, etc.) * but broken * committing on feature branch and will come back to this
swaits2015-02-16 16:26:16 -08:00
d9009cdbaa
flow: Created branch 'feature/unique_links_from_each_page'.
swaits2015-02-16 15:50:30 -08:00
56467ea175
Merge from develop. First version satisfying all basic requirements.
swaits2015-02-16 06:27:10 -08:00
f3152b6f44
broken links now recorded as such * Broken field in Page{} now, * anything that fails to fetch (with http 200) or parse (should not happen) is considered broken * tests updated appropriately * discovered my "image.png" file was in the wrong directory because it showed up as broken. fixed! * bug fix: only fetch if type=text/html (after HEAD response)
swaits2015-02-15 22:25:50 -08:00
f9fd0955e8
flow: Created branch 'feature/fix_broken_links'.
swaits2015-02-15 22:09:25 -08:00
7498b648d7
urls with hosts that don't match the first URL are now skipped and marked as remote in the output * added Skipped field to Page to remember this state * marshaled into "Remote" field in JSON * bug fix: skipped (etc.) sites weren't getting added to the main page list * tests had to be reworked. because the crawl is non-deterministic, the tests started breaking. tests made a bit more robust, but are slightly weaker now. * the lists of URLs in the output are now sorted * ... and the parseLinks is no longer doing any sorting (no point in it) * parser tests updated accordingly * added a "broken" link to index.html - not currently supported, will fix next!
swaits2015-02-15 22:07:37 -08:00
794f586979
flow: Created branch 'feature/limit_host_part'.
swaits2015-02-15 20:44:54 -08:00
be11b13443
json sitemap output now functional * the json marshaller cannot deal with circular data structures, and no automatic .String() conversion * so we copy our crawl data slice into another slice which will be marshaled into json more easily * made the page slice we use during crawling actually circular (as it should be) rather than putting duplicates in for each page discovered * test for conversion to output data structure, and hacky cheater test for json output * removed Base *Page from our Page struct as it's only needed temporarily
swaits2015-02-15 18:57:34 -08:00
7d9bd98488
flow: Created branch 'feature/output_sitemap'.
swaits2015-02-15 17:25:52 -08:00
8a6cac5df0
minor test cleanup; add "make stats" target to run cloc against project
swaits2015-02-15 16:19:58 -08:00
c95ad7ab95
now crawling multiple pages! * moved Page{} and NewPage() into page.go * queued and crawled are both maps now. used to track which pages we already know about * tests updated to match the multipage crawling * added another file to the testsite (scripts/blah.js)
swaits2015-02-15 15:26:21 -08:00
0c48b94ece
refactoring and cleanup. needed before proceeding. * fetch() renamed to fetchPage. now takes *Page instead of url string * fetch_filetype() renamed to fetchFiletype. now takes *Page instead of url string * parse() renamed to parseLinks() * tests updated appropriately * minor cleanup (lint) * NewPage added for conveniently creating new *Page structs from url strings
swaits2015-02-15 14:54:52 -08:00
614f05f1d7
minor Makefile cleanup
swaits2015-02-15 14:02:08 -08:00
9fb19b0067
Merge from develop. Plus hack fix to make golint work again.
swaits2015-02-15 12:59:52 -08:00
12cd121271
add coverage to Makefile (make cover) * also tweaked GOPATH as needed * re-vendored with cover tool included * ignore $/cover.out
swaits2015-02-15 12:40:56 -08:00
dbf49d259b
flow: Created branch 'feature/add_test_coverage'.
swaits2015-02-15 12:18:00 -08:00
0456723f0b
minimally functional crawler - crawls single page, finds links, stores in data structure * refined Page struct. now need to refactor everything else to use this * wrote docrawl() function. NOT intended to be anything like the final design. just sketching it out to get a better idea of needs, design, etc. * replaced fmt.Print* with t.Log* in tests (oops!) * added a header fetcher function to get the file type (since we only want to parse html (with tests) * added small test for the URL resolver where the base URL is empty * added placeholder files in testsite/ for about.html and image.png
swaits2015-02-15 12:12:30 -08:00
main