swaits
6ed857bb77
Merge from develop
2015-02-17 10:38:23 -08:00
swaits
f9488c5c11
flow: Merged <feature> 'cleanup_and_refactoring' to <develop> ('develop').
2015-02-17 10:38:04 -08:00
swaits
154ef4321e
flow: Closed <feature> 'cleanup_and_refactoring'.
2015-02-17 10:38:04 -08:00
swaits
3e35fdb488
some cleanup and refactoring
...
* moved crawlItem into fetch.go
* renamed fetchPage fetchItem
* doCrawl now takes number of workers as a parameter
* command line flag parsing added
2015-02-17 10:36:48 -08:00
swaits
6c9434e39a
flow: Created branch 'feature/cleanup_and_refactoring'.
2015-02-17 09:53:28 -08:00
swaits
425fb674dd
Merge from develop
2015-02-17 09:28:56 -08:00
swaits
04e26b214e
flow: Merged <feature> 'use_crawler_workers' to <develop> ('develop').
2015-02-17 09:28:34 -08:00
swaits
b502cfd5b1
flow: Closed <feature> 'use_crawler_workers'.
2015-02-17 09:28:34 -08:00
swaits
13b84ee452
completion of crawler worker mode. tests still passing!
2015-02-17 09:28:24 -08:00
swaits
3f85a0e7c0
switch to using a limited number of crawler worker functions
2015-02-17 09:24:52 -08:00
swaits
e80ef1b1b9
flow: Created branch 'feature/use_crawler_workers'.
2015-02-17 09:24:26 -08:00
swaits
af148c3a8a
Merge from develop
2015-02-17 08:57:14 -08:00
swaits
e38595db7e
realized we don't need the second itemMap ("results"), just the one ("crawled"). tests still pass.
2015-02-17 08:56:46 -08:00
swaits
c58bf3b364
Merge from develop
2015-02-17 02:11:19 -08:00
swaits
c13c5cf7dd
better usage, command line error handling
2015-02-17 02:10:50 -08:00
swaits
82d901ac38
Merge from develop. Functional for initial release.
2015-02-17 01:51:24 -08:00
swaits
71e0ea4975
flow: Merged <feature> 'make_json_pretty' to <develop> ('develop').
2015-02-17 01:50:46 -08:00
swaits
1c0512a98d
flow: Closed <feature> 'make_json_pretty'.
2015-02-17 01:50:46 -08:00
swaits
1a66b03d0d
add indentation to json output. update test.
2015-02-17 01:50:39 -08:00
swaits
21e2acd4a5
flow: Created branch 'feature/make_json_pretty'.
2015-02-17 01:34:30 -08:00
swaits
3a56205fa8
update Design doc status for each item
2015-02-17 01:32:45 -08:00
swaits
e2a799f02f
flow: Merged <feature> 'fix_anchor_duplication' to <develop> ('develop').
2015-02-17 01:26:44 -08:00
swaits
f72d5d5eb0
flow: Closed <feature> 'fix_anchor_duplication'.
2015-02-17 01:26:44 -08:00
swaits
1b09e6c204
deal with anchors and index.html "page aliasing: proprerly
...
* so now http://a.com/index.html and http://a.com/ are understood equivalents
* as are http://a.com/x.html#blog and http://a.com/x.html (etc.)
* this meant a little more bookkeeping in docrawler, but not horrible
* stripAnchorFromURL because stripURL because it now handles anchor and path suffix stripping
* and url tests were written to support this
2015-02-17 01:26:20 -08:00
swaits
4e335cfc8a
reviewed coverage report. remove some dead code. add a few comments to areas of code unlikely to hit.
2015-02-17 00:44:39 -08:00
swaits
db314ec938
flow: Created branch 'feature/fix_anchor_duplication'.
2015-02-17 00:36:58 -08:00
swaits
42a40be62f
flow: Merged <feature> 'remove_error_from_parselinks' to <develop> ('develop').
2015-02-16 23:08:37 -08:00
swaits
ee3cac7e29
flow: Closed <feature> 'remove_error_from_parselinks'.
2015-02-16 23:08:37 -08:00
swaits
6edd18e361
remove error from return of parseLinks as it's not needed
2015-02-16 23:08:33 -08:00
swaits
f16be33dbd
flow: Created branch 'feature/remove_error_from_parselinks'.
2015-02-16 23:06:45 -08:00
swaits
6665729e3f
rename docrawl -> doCrawl; add some missing comments
2015-02-16 23:02:03 -08:00
swaits
48be4bc77d
flow: Merged <feature> 'unique_links_from_each_page' to <develop> ('develop').
2015-02-16 22:48:04 -08:00
swaits
6d16ebb8a1
flow: Closed <feature> 'unique_links_from_each_page'.
2015-02-16 22:48:04 -08:00
swaits
2c2cfc126c
links are now uniq'd before marshaling
...
* crawled is now a map to httpItem's (instead of struct{}) so we can connect pages mid-crawl
* added duplicate link to html site, test updated to match
* now any httpItem with linkType == tUnknown is listed as "Broken" in the output
2015-02-16 22:47:56 -08:00
swaits
7e18005473
remove page.go (merge miss from /develop)
2015-02-16 22:24:59 -08:00
swaits
81ae0c433b
Merge from develop
2015-02-16 22:23:19 -08:00
swaits
9938ae06f6
flow: Merged <feature> 'refactor_state_management' to <develop> ('develop').
2015-02-16 22:14:23 -08:00
swaits
e3c730bdfc
flow: Closed <feature> 'refactor_state_management'.
2015-02-16 22:14:23 -08:00
swaits
72cac47961
docrawl() completely refactored to use goroutine per URL
...
* outputs status
* crawlStatus() function created (split out of docrawl())
* changed from using bool flags to linkType in single httpItem enum, tests updated to match this
* unified where we set httpItem.linkType (inside fetchType), appropriate errors and returns added
* renamed a lot of things from 'page' to 'item'
* include refurl in httpItem
* []*Location result is now sorted
2015-02-16 22:13:52 -08:00
swaits
d7cc760e5b
rename page.go -> item.go
2015-02-16 18:17:01 -08:00
swaits
58614802b6
renamed Page -> httpItem, and changed its members to unexported as well
...
* tests all passing
2015-02-16 18:14:12 -08:00
swaits
66f78b562f
flow: Created branch 'feature/refactor_state_management'.
2015-02-16 16:34:09 -08:00
swaits
d3aeafeba5
update TODO
2015-02-16 16:28:36 -08:00
swaits
2e99708417
won't recrawl pages that are the same after stripping #anchor (blah.html#about, blah.html#contact, etc.)
...
* but broken
* committing on feature branch and will come back to this
2015-02-16 16:26:16 -08:00
swaits
d9009cdbaa
flow: Created branch 'feature/unique_links_from_each_page'.
2015-02-16 15:50:30 -08:00
swaits
56467ea175
Merge from develop. First version satisfying all basic requirements.
2015-02-16 06:27:10 -08:00
swaits
fe46e7af15
add a TODO list
2015-02-16 06:13:59 -08:00
swaits
77b50a22b4
split tests out from everything else in cloc (make stats)
2015-02-15 23:12:01 -08:00
swaits
81e673a6bc
flow: Merged <feature> 'url_on_command_line' to <develop> ('develop').
2015-02-15 22:58:06 -08:00
swaits
e867840994
flow: Closed <feature> 'url_on_command_line'.
2015-02-15 22:58:06 -08:00
swaits