Commit Graph

Select branches

Hide Pull Requests

develop

feature/add_test_coverage

feature/cleanup_and_refactoring

feature/create_simple_sitemap

feature/extract_titles_from_pages

feature/fix_anchor_duplication

feature/fix_broken_links

feature/improve_test_coverage

feature/limit_host_part

feature/make_json_pretty

feature/output_sitemap

feature/refactor_state_management

feature/remove_error_from_parselinks

feature/split_tests

feature/unique_links_from_each_page

feature/url_on_command_line

feature/url_parse_and_clean

feature/use_crawler_workers

main

Mono Color

• 65b119ace6 rename getURL() to resolveURL() because "get" may imply actually GETting it swaits 2015-02-15 10:02:20 -08:00
• b398e2ddf4 url resolution (with baseref) now functional * test cases (relative and absolute) added * decent cleanup (using url.URL throughout instead of strings) swaits 2015-02-15 10:00:06 -08:00
• 24f557d927 basic url cleaning test passes swaits 2015-02-15 08:29:41 -08:00
• 42799df06e add initial failing test for cleanURL swaits 2015-02-15 08:20:00 -08:00
• 5cad35e382 flow: Created branch 'feature/url_parse_and_clean'. swaits 2015-02-15 08:14:29 -08:00
• 2b58869a04 Merge from develop swaits 2015-02-15 08:08:23 -08:00
• 0e58ea1282 bugfix: enable case insensitivity on REs for urls and title swaits 2015-02-15 08:07:42 -08:00
• b3ca6c41ae flow: Merged <feature> 'extract_titles_from_pages' to <develop> ('develop'). swaits 2015-02-15 08:03:28 -08:00
• b70a89815e flow: Closed <feature> 'extract_titles_from_pages'. feature/extract_titles_from_pages swaits 2015-02-15 08:03:28 -08:00
• b01fcb2132 add parsing of <title>...</title> to parse() swaits 2015-02-15 08:03:19 -08:00
• 7501abf121 flow: Created branch 'feature/extract_titles_from_pages'. swaits 2015-02-15 07:48:33 -08:00
• d1f14120df Merge from develop swaits 2015-02-15 07:48:10 -08:00
• 39ef723ec6 flow: Merged <feature> 'split_tests' to <develop> ('develop'). swaits 2015-02-15 07:45:16 -08:00
• 57b7130ac7 flow: Closed <feature> 'split_tests'. feature/split_tests swaits 2015-02-15 07:45:16 -08:00
• 042812f38e move simple parse test to parse_test.go which apply only to the parser. tests in dowcrawler_test.go remain integration/functional tests. swaits 2015-02-15 07:45:06 -08:00
• 0cd3cb510a flow: Created branch 'feature/split_tests'. swaits 2015-02-15 07:40:57 -08:00
• 5ef46be8c4 add (presently failing) tests for discovering a simple site map swaits 2015-02-15 00:58:01 -08:00
• 249aca229b flow: Created branch 'feature/create_simple_sitemap'. swaits 2015-02-15 00:43:50 -08:00
• 38224a8987 misc. minor cleanup * moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly) * updated idea in Design.md * renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming * parse now sorts its results. test updated. * moved regexp compilation to init instead of every time parse() is called * misc comments and cleanup swaits 2015-02-15 00:40:36 -08:00
• d716fbc6f8 flow initialization: Created <develop> trunk: develop swaits 2015-02-14 23:50:10 -08:00
• 85f4b69e61 flow initialization: Added configuration file. swaits 2015-02-14 23:50:10 -08:00
• 8e3995b45c add tests for very simple fetching and parsing * added test for fetching a known file and verifying its content matches * add test for parsing a known string and verify URLs are extracted * add test for a request which doesn't exist and verify 404 is received (canary test) * add comments throughout test (for each test function) * wrote (stupidly) simple fetcher which takes a URL, GETs it, and returns the body and error * wrote (stupidly) simple parser which takes a string, parses out (src|href|xhref)="...", and returns a slice of the captured links * added needed html files swaits 2015-02-14 23:49:35 -08:00
• 9f411ee00b clean up the output of tests swaits 2015-02-14 22:39:55 -08:00
• 32d3f82325 move some magic numbers to constants in test harness swaits 2015-02-14 22:18:31 -08:00
• 21e1fb3451 initial vendor commit swaits 2015-02-14 21:53:22 -08:00
• 747c805a82 add "make env" to dump the "go env" from within the make environment swaits 2015-02-14 21:52:58 -08:00
• aaa5f639cf oops! ignored the wrong bin directory in .hgignore. fixed. swaits 2015-02-14 21:50:02 -08:00
• 3bce3ccd2a add test harness web server * plus "make autotest" runs "make test" once first * add .hgignore swaits 2015-02-14 21:46:52 -08:00
• da7c5ba57c design doc update swaits 2015-02-14 21:11:10 -08:00
• b0e64c7973 initial commit, empty go project, plus simple design doc swaits 2015-02-14 21:00:18 -08:00