swaits/docrawler
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
112 Commits 19 Branches 0 Tags
5fb78a041e1193cecbff1f2489c28cf969e8140f
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
swaits 5fb78a041e add sh and zsh targets to make file to conveniently get at the same environment as make; moved cover.out to $/tmp, updated .hgignore to match 2015-02-20 05:31:08 -08:00
swaits a7ca3de99e change the functions basically just taking a single *httpItem parameter to use them as receivers instead 2015-02-18 07:55:22 -08:00
swaits f9488c5c11 flow: Merged <feature> 'cleanup_and_refactoring' to <develop> ('develop'). 2015-02-17 10:38:04 -08:00
swaits 154ef4321e flow: Closed <feature> 'cleanup_and_refactoring'. 2015-02-17 10:38:04 -08:00
swaits 3e35fdb488 some cleanup and refactoring 
* moved crawlItem into fetch.go
* renamed fetchPage fetchItem
* doCrawl now takes number of workers as a parameter
* command line flag parsing added
 2015-02-17 10:36:48 -08:00
swaits 6c9434e39a flow: Created branch 'feature/cleanup_and_refactoring'. 2015-02-17 09:53:28 -08:00
swaits 04e26b214e flow: Merged <feature> 'use_crawler_workers' to <develop> ('develop'). 2015-02-17 09:28:34 -08:00
swaits b502cfd5b1 flow: Closed <feature> 'use_crawler_workers'. 2015-02-17 09:28:34 -08:00
swaits 13b84ee452 completion of crawler worker mode. tests still passing! 2015-02-17 09:28:24 -08:00
swaits 3f85a0e7c0 switch to using a limited number of crawler worker functions 2015-02-17 09:24:52 -08:00
swaits e80ef1b1b9 flow: Created branch 'feature/use_crawler_workers'. 2015-02-17 09:24:26 -08:00
swaits e38595db7e realized we don't need the second itemMap ("results"), just the one ("crawled"). tests still pass. 2015-02-17 08:56:46 -08:00
swaits c13c5cf7dd better usage, command line error handling 2015-02-17 02:10:50 -08:00
swaits 71e0ea4975 flow: Merged <feature> 'make_json_pretty' to <develop> ('develop'). 2015-02-17 01:50:46 -08:00
swaits 1c0512a98d flow: Closed <feature> 'make_json_pretty'. 2015-02-17 01:50:46 -08:00
swaits 1a66b03d0d add indentation to json output. update test. 2015-02-17 01:50:39 -08:00
swaits 21e2acd4a5 flow: Created branch 'feature/make_json_pretty'. 2015-02-17 01:34:30 -08:00
swaits 3a56205fa8 update Design doc status for each item 2015-02-17 01:32:45 -08:00
swaits e2a799f02f flow: Merged <feature> 'fix_anchor_duplication' to <develop> ('develop'). 2015-02-17 01:26:44 -08:00
swaits f72d5d5eb0 flow: Closed <feature> 'fix_anchor_duplication'. 2015-02-17 01:26:44 -08:00
swaits 1b09e6c204 deal with anchors and index.html "page aliasing: proprerly 
* so now http://a.com/index.html and http://a.com/ are understood equivalents
* as are http://a.com/x.html#blog and http://a.com/x.html (etc.)
* this meant a little more bookkeeping in docrawler, but not horrible
* stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping
* and url tests were written to support this
 2015-02-17 01:26:20 -08:00
swaits 4e335cfc8a reviewed coverage report. remove some dead code. add a few comments to areas of code unlikely to hit. 2015-02-17 00:44:39 -08:00
swaits db314ec938 flow: Created branch 'feature/fix_anchor_duplication'. 2015-02-17 00:36:58 -08:00
swaits 42a40be62f flow: Merged <feature> 'remove_error_from_parselinks' to <develop> ('develop'). 2015-02-16 23:08:37 -08:00
swaits ee3cac7e29 flow: Closed <feature> 'remove_error_from_parselinks'. 2015-02-16 23:08:37 -08:00
swaits 6edd18e361 remove error from return of parseLinks as it's not needed 2015-02-16 23:08:33 -08:00
swaits f16be33dbd flow: Created branch 'feature/remove_error_from_parselinks'. 2015-02-16 23:06:45 -08:00
swaits 6665729e3f rename docrawl -> doCrawl; add some missing comments 2015-02-16 23:02:03 -08:00
swaits 48be4bc77d flow: Merged <feature> 'unique_links_from_each_page' to <develop> ('develop'). 2015-02-16 22:48:04 -08:00
swaits 6d16ebb8a1 flow: Closed <feature> 'unique_links_from_each_page'. 2015-02-16 22:48:04 -08:00
swaits 2c2cfc126c links are now uniq'd before marshaling 
* crawled is now a map to httpItem's (instead of struct{}) so we can connect pages mid-crawl
* added duplicate link to html site, test updated to match
* now any httpItem with linkType == tUnknown is listed as "Broken" in the output
 2015-02-16 22:47:56 -08:00
swaits 7e18005473 remove page.go (merge miss from /develop) 2015-02-16 22:24:59 -08:00
swaits 81ae0c433b Merge from develop 2015-02-16 22:23:19 -08:00
swaits 9938ae06f6 flow: Merged <feature> 'refactor_state_management' to <develop> ('develop'). 2015-02-16 22:14:23 -08:00
swaits e3c730bdfc flow: Closed <feature> 'refactor_state_management'. 2015-02-16 22:14:23 -08:00
swaits 72cac47961 docrawl() completely refactored to use goroutine per URL 
* outputs status
* crawlStatus() function created (split out of docrawl())
* changed from using bool flags to linkType in single httpItem enum, tests updated to match this
* unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added
* renamed a lot of things from 'page' to 'item'
* include refurl in httpItem
* []*Location result is now sorted
 2015-02-16 22:13:52 -08:00
swaits d7cc760e5b rename page.go -> item.go 2015-02-16 18:17:01 -08:00
swaits 58614802b6 renamed Page -> httpItem, and changed its members to unexported as well 
* tests all passing
 2015-02-16 18:14:12 -08:00
swaits 66f78b562f flow: Created branch 'feature/refactor_state_management'. 2015-02-16 16:34:09 -08:00
swaits d3aeafeba5 update TODO 2015-02-16 16:28:36 -08:00
swaits 2e99708417 won't recrawl pages that are the same after stripping #anchor (blah.html#about, blah.html#contact, etc.) 
* but broken
* committing on feature branch and will come back to this
 2015-02-16 16:26:16 -08:00
swaits d9009cdbaa flow: Created branch 'feature/unique_links_from_each_page'. 2015-02-16 15:50:30 -08:00
swaits fe46e7af15 add a TODO list 2015-02-16 06:13:59 -08:00
swaits 77b50a22b4 split tests out from everything else in cloc (make stats) 2015-02-15 23:12:01 -08:00
swaits 81e673a6bc flow: Merged <feature> 'url_on_command_line' to <develop> ('develop'). 2015-02-15 22:58:06 -08:00
swaits e867840994 flow: Closed <feature> 'url_on_command_line'. 2015-02-15 22:58:06 -08:00
swaits 5813e948f0 will attempt to crawl any URLs passed in on command line 2015-02-15 22:58:01 -08:00
swaits e3556df1b1 flow: Created branch 'feature/url_on_command_line'. 2015-02-15 22:31:18 -08:00
swaits b159dbf636 design doc extra line removed 2015-02-15 22:30:53 -08:00
swaits bdc75eca4a flow: Merged <feature> 'fix_broken_links' to <develop> ('develop'). 2015-02-15 22:27:04 -08:00
swaits c2b2897f5d flow: Closed <feature> 'fix_broken_links'. 2015-02-15 22:27:04 -08:00
swaits 569037d29f lint and fmt 2015-02-15 22:26:44 -08:00
swaits f3152b6f44 broken links now recorded as such 
* Broken field in Page{} now,
* anything that fails to fetch (with http 200) or parse (should not happen) is considered broken
* tests updated appropriately
* discovered my "image.png" file was in the wrong directory because it showed up as broken. fixed!
* bug fix: only fetch if type=text/html (after HEAD response)
 2015-02-15 22:25:50 -08:00
swaits f9fd0955e8 flow: Created branch 'feature/fix_broken_links'. 2015-02-15 22:09:25 -08:00
swaits 2f2d1036f6 flow: Merged <feature> 'limit_host_part' to <develop> ('develop'). 2015-02-15 22:09:10 -08:00
swaits 148e833454 flow: Closed <feature> 'limit_host_part'. 2015-02-15 22:09:10 -08:00
swaits 7498b648d7 urls with hosts that don't match the first URL are now skipped and marked as remote in the output 
* added Skipped field to Page to remember this state
* marshaled into "Remote" field in JSON
* bug fix: skipped (etc.) sites weren't getting added to the main page list
* tests had to be reworked. because the crawl is non-deterministic, the tests started breaking. tests made a bit more robust, but are slightly weaker now.
* the lists of URLs in the output are now sorted
* ... and the parseLinks is no longer doing any sorting (no point in it)
* parser tests updated accordingly
* added a "broken" link to index.html - not currently supported, will fix next!
 2015-02-15 22:07:37 -08:00
swaits 794f586979 flow: Created branch 'feature/limit_host_part'. 2015-02-15 20:44:54 -08:00
swaits 4598e6d8fe flow: Merged <feature> 'output_sitemap' to <develop> ('develop'). 2015-02-15 20:42:55 -08:00
swaits 2ae0502aba flow: Closed <feature> 'output_sitemap'. 2015-02-15 20:42:55 -08:00
swaits be11b13443 json sitemap output now functional 
* the json marshaller cannot deal with circular data structures, and no automatic .String() conversion
* so we copy our crawl data slice into another slice which will be marshaled into json more easily
* made the page slice we use during crawling actually circular (as it should be) rather than putting duplicates in for each page discovered
* test for conversion to output data structure, and hacky cheater test for json output
* removed Base *Page from our Page struct as it's only needed temporarily
 2015-02-15 18:57:34 -08:00
swaits 7d9bd98488 flow: Created branch 'feature/output_sitemap'. 2015-02-15 17:25:52 -08:00
swaits 8a6cac5df0 minor test cleanup; add "make stats" target to run cloc against project 2015-02-15 16:19:58 -08:00
swaits 828d776b94 flow: Merged <feature> 'improve_test_coverage' to <develop> ('develop'). 2015-02-15 16:04:09 -08:00
swaits 1e54b04490 flow: Closed <feature> 'improve_test_coverage'. 2015-02-15 16:04:09 -08:00
swaits ccc65f9715 coverage improved from 85.2% to 91.1% 2015-02-15 16:03:10 -08:00
swaits e4ce693396 flow: Created branch 'feature/improve_test_coverage'. 2015-02-15 15:38:01 -08:00
swaits cca2bcaf58 flow: Merged <feature> 'create_simple_sitemap' to <develop> ('develop'). 2015-02-15 15:31:27 -08:00
swaits f058de8e6f flow: Closed <feature> 'create_simple_sitemap'. 2015-02-15 15:31:27 -08:00
swaits 8a71e4cb5c missed a go fmt 2015-02-15 15:30:13 -08:00
swaits c95ad7ab95 now crawling multiple pages! 
* moved Page{} and NewPage() into page.go
* queued and crawled are both maps now. used to track which pages we already know about
* tests updated to match the multipage crawling
* added another file to the testsite (scripts/blah.js)
 2015-02-15 15:26:21 -08:00
swaits 0c48b94ece refactoring and cleanup. needed before proceeding. 
* fetch() renamed to fetchPage. now takes *Page instead of url string
* fetch_filetype() renamed to fetchFiletype. now takes *Page instead of url string
* parse() renamed to parseLinks()
* tests updated appropriately
* minor cleanup (lint)
* NewPage added for conveniently creating new *Page structs from url strings
 2015-02-15 14:54:52 -08:00
swaits 614f05f1d7 minor Makefile cleanup 2015-02-15 14:02:08 -08:00
swaits 9fb19b0067 Merge from develop. Plus hack fix to make golint work again. 2015-02-15 12:59:52 -08:00
swaits d8b47abf6e flow: Merged <feature> 'add_test_coverage' to <develop> ('develop'). 2015-02-15 12:41:02 -08:00
swaits 67817048f9 flow: Closed <feature> 'add_test_coverage'. 2015-02-15 12:41:02 -08:00
swaits 12cd121271 add coverage to Makefile (make cover) 
* also tweaked GOPATH as needed
* re-vendored with cover tool included
* ignore $/cover.out
 2015-02-15 12:40:56 -08:00
swaits dbf49d259b flow: Created branch 'feature/add_test_coverage'. 2015-02-15 12:18:00 -08:00
swaits 0456723f0b minimally functional crawler - crawls single page, finds links, stores in data structure 
* refined Page struct. now need to refactor everything else to use this
* wrote docrawl() function. NOT intended to be anything like the final design. just sketching it out to get a better idea of needs, design, etc.
* replaced fmt.Print* with t.Log* in tests (oops!)
* added a header fetcher function to get the file type (since we only want to parse html (with tests)
* added small test for the URL resolver where the base URL is empty
* added placeholder files in testsite/ for about.html and image.png
 2015-02-15 12:12:30 -08:00
swaits 7f6978ad27 Merge from develop 2015-02-15 10:03:36 -08:00
swaits d60700bf1a flow: Merged <feature> 'url_parse_and_clean' to <develop> ('develop'). 2015-02-15 10:02:25 -08:00
swaits afa5cfcc89 flow: Closed <feature> 'url_parse_and_clean'. 2015-02-15 10:02:25 -08:00
swaits 65b119ace6 rename getURL() to resolveURL() because "get" may imply actually GETting it 2015-02-15 10:02:20 -08:00
swaits b398e2ddf4 url resolution (with baseref) now functional 
* test cases (relative and absolute) added
* decent cleanup (using url.URL throughout instead of strings)
 2015-02-15 10:00:06 -08:00
swaits 24f557d927 basic url cleaning test passes 2015-02-15 08:29:41 -08:00
swaits 42799df06e add initial failing test for cleanURL 2015-02-15 08:20:00 -08:00
swaits 5cad35e382 flow: Created branch 'feature/url_parse_and_clean'. 2015-02-15 08:14:29 -08:00
swaits 2b58869a04 Merge from develop 2015-02-15 08:08:23 -08:00
swaits 0e58ea1282 bugfix: enable case insensitivity on REs for urls and title 2015-02-15 08:07:42 -08:00
swaits b3ca6c41ae flow: Merged <feature> 'extract_titles_from_pages' to <develop> ('develop'). 2015-02-15 08:03:28 -08:00
swaits b70a89815e flow: Closed <feature> 'extract_titles_from_pages'. 2015-02-15 08:03:28 -08:00
swaits b01fcb2132 add parsing of <title>...</title> to parse() 2015-02-15 08:03:19 -08:00
swaits 7501abf121 flow: Created branch 'feature/extract_titles_from_pages'. 2015-02-15 07:48:33 -08:00
swaits d1f14120df Merge from develop 2015-02-15 07:48:10 -08:00
swaits 39ef723ec6 flow: Merged <feature> 'split_tests' to <develop> ('develop'). 2015-02-15 07:45:16 -08:00
swaits 57b7130ac7 flow: Closed <feature> 'split_tests'. 2015-02-15 07:45:16 -08:00
swaits 042812f38e move simple parse test to parse_test.go which apply only to the parser. tests in dowcrawler_test.go remain integration/functional tests. 2015-02-15 07:45:06 -08:00
swaits 0cd3cb510a flow: Created branch 'feature/split_tests'. 2015-02-15 07:40:57 -08:00
swaits 5ef46be8c4 add (presently failing) tests for discovering a simple site map 2015-02-15 00:58:01 -08:00
swaits 249aca229b flow: Created branch 'feature/create_simple_sitemap'. 2015-02-15 00:43:50 -08:00