126 Commits
Author SHA1 Message Date
swaits 5cad35e382 flow: Created branch 'feature/url_parse_and_clean'. 2015-02-15 08:14:29 -08:00
swaits 2b58869a04 Merge from develop 2015-02-15 08:08:23 -08:00
swaits 0e58ea1282 bugfix: enable case insensitivity on REs for urls and title 2015-02-15 08:07:42 -08:00
swaits b3ca6c41ae flow: Merged <feature> 'extract_titles_from_pages' to <develop> ('develop'). 2015-02-15 08:03:28 -08:00
swaits b70a89815e flow: Closed <feature> 'extract_titles_from_pages'. 2015-02-15 08:03:28 -08:00
swaits b01fcb2132 add parsing of <title>...</title> to parse() 2015-02-15 08:03:19 -08:00
swaits 7501abf121 flow: Created branch 'feature/extract_titles_from_pages'. 2015-02-15 07:48:33 -08:00
swaits d1f14120df Merge from develop 2015-02-15 07:48:10 -08:00
swaits 39ef723ec6 flow: Merged <feature> 'split_tests' to <develop> ('develop'). 2015-02-15 07:45:16 -08:00
swaits 57b7130ac7 flow: Closed <feature> 'split_tests'. 2015-02-15 07:45:16 -08:00
swaits 042812f38e move simple parse test to parse_test.go which apply only to the parser. tests in dowcrawler_test.go remain integration/functional tests. 2015-02-15 07:45:06 -08:00
swaits 0cd3cb510a flow: Created branch 'feature/split_tests'. 2015-02-15 07:40:57 -08:00
swaits 5ef46be8c4 add (presently failing) tests for discovering a simple site map 2015-02-15 00:58:01 -08:00
swaits 249aca229b flow: Created branch 'feature/create_simple_sitemap'. 2015-02-15 00:43:50 -08:00
swaits 38224a8987 misc. minor cleanup
* moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly)
* updated idea in Design.md
* renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming
* parse now sorts its results. test updated.
* moved regexp compilation to init instead of every time parse() is called
* misc comments and cleanup
2015-02-15 00:40:36 -08:00
swaits 85f4b69e61 flow initialization: Added configuration file. 2015-02-14 23:50:10 -08:00
swaits d716fbc6f8 flow initialization: Created <develop> trunk: develop 2015-02-14 23:50:10 -08:00
swaits 8e3995b45c add tests for very simple fetching and parsing
* added test for fetching a known file and verifying its content matches
* add test for parsing a known string and verify URLs are extracted
* add test for a request which doesn't exist and verify 404 is received (canary test)
* add comments throughout test (for each test function)
* wrote (stupidly) simple fetcher which takes a URL, GETs it, and returns the body and error
* wrote (stupidly) simple parser which takes a string, parses out (src|href|xhref)="...", and returns a slice of the captured links
* added needed html files
2015-02-14 23:49:35 -08:00
swaits 9f411ee00b clean up the output of tests 2015-02-14 22:39:55 -08:00
swaits 32d3f82325 move some magic numbers to constants in test harness 2015-02-14 22:18:31 -08:00
swaits 21e1fb3451 initial vendor commit 2015-02-14 21:53:22 -08:00
swaits 747c805a82 add "make env" to dump the "go env" from within the make environment 2015-02-14 21:52:58 -08:00
swaits aaa5f639cf oops! ignored the wrong bin directory in .hgignore. fixed. 2015-02-14 21:50:02 -08:00
swaits 3bce3ccd2a add test harness web server
* plus "make autotest" runs "make test" once first
* add .hgignore
2015-02-14 21:46:52 -08:00
swaits da7c5ba57c design doc update 2015-02-14 21:11:10 -08:00
swaits b0e64c7973 initial commit, empty go project, plus simple design doc 2015-02-14 21:00:18 -08:00