swaits
65b119ace6
rename getURL() to resolveURL() because "get" may imply actually GETting it
2015-02-15 10:02:20 -08:00
swaits
b398e2ddf4
url resolution (with baseref) now functional
...
* test cases (relative and absolute) added
* decent cleanup (using url.URL throughout instead of strings)
2015-02-15 10:00:06 -08:00
swaits
24f557d927
basic url cleaning test passes
2015-02-15 08:29:41 -08:00
swaits
42799df06e
add initial failing test for cleanURL
2015-02-15 08:20:00 -08:00
swaits
5cad35e382
flow: Created branch 'feature/url_parse_and_clean'.
2015-02-15 08:14:29 -08:00
swaits
2b58869a04
Merge from develop
2015-02-15 08:08:23 -08:00
swaits
0e58ea1282
bugfix: enable case insensitivity on REs for urls and title
2015-02-15 08:07:42 -08:00
swaits
b3ca6c41ae
flow: Merged <feature> 'extract_titles_from_pages' to <develop> ('develop').
2015-02-15 08:03:28 -08:00
swaits
b70a89815e
flow: Closed <feature> 'extract_titles_from_pages'.
2015-02-15 08:03:28 -08:00
swaits
b01fcb2132
add parsing of <title>...</title> to parse()
2015-02-15 08:03:19 -08:00
swaits
7501abf121
flow: Created branch 'feature/extract_titles_from_pages'.
2015-02-15 07:48:33 -08:00
swaits
d1f14120df
Merge from develop
2015-02-15 07:48:10 -08:00
swaits
39ef723ec6
flow: Merged <feature> 'split_tests' to <develop> ('develop').
2015-02-15 07:45:16 -08:00
swaits
57b7130ac7
flow: Closed <feature> 'split_tests'.
2015-02-15 07:45:16 -08:00
swaits
042812f38e
move simple parse test to parse_test.go which apply only to the parser. tests in dowcrawler_test.go remain integration/functional tests.
2015-02-15 07:45:06 -08:00
swaits
0cd3cb510a
flow: Created branch 'feature/split_tests'.
2015-02-15 07:40:57 -08:00
swaits
5ef46be8c4
add (presently failing) tests for discovering a simple site map
2015-02-15 00:58:01 -08:00
swaits
249aca229b
flow: Created branch 'feature/create_simple_sitemap'.
2015-02-15 00:43:50 -08:00
swaits
38224a8987
misc. minor cleanup
...
* moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly)
* updated idea in Design.md
* renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming
* parse now sorts its results. test updated.
* moved regexp compilation to init instead of every time parse() is called
* misc comments and cleanup
2015-02-15 00:40:36 -08:00
swaits
85f4b69e61
flow initialization: Added configuration file.
2015-02-14 23:50:10 -08:00
swaits
d716fbc6f8
flow initialization: Created <develop> trunk: develop
2015-02-14 23:50:10 -08:00
swaits
8e3995b45c
add tests for very simple fetching and parsing
...
* added test for fetching a known file and verifying its content matches
* add test for parsing a known string and verify URLs are extracted
* add test for a request which doesn't exist and verify 404 is received (canary test)
* add comments throughout test (for each test function)
* wrote (stupidly) simple fetcher which takes a URL, GETs it, and returns the body and error
* wrote (stupidly) simple parser which takes a string, parses out (src|href|xhref)="...", and returns a slice of the captured links
* added needed html files
2015-02-14 23:49:35 -08:00
swaits
9f411ee00b
clean up the output of tests
2015-02-14 22:39:55 -08:00
swaits
32d3f82325
move some magic numbers to constants in test harness
2015-02-14 22:18:31 -08:00
swaits
21e1fb3451
initial vendor commit
2015-02-14 21:53:22 -08:00
swaits
747c805a82
add "make env" to dump the "go env" from within the make environment
2015-02-14 21:52:58 -08:00
swaits
aaa5f639cf
oops! ignored the wrong bin directory in .hgignore. fixed.
2015-02-14 21:50:02 -08:00
swaits
3bce3ccd2a
add test harness web server
...
* plus "make autotest" runs "make test" once first
* add .hgignore
2015-02-14 21:46:52 -08:00
swaits
da7c5ba57c
design doc update
2015-02-14 21:11:10 -08:00
swaits
b0e64c7973
initial commit, empty go project, plus simple design doc
2015-02-14 21:00:18 -08:00
swaits