misc. minor cleanup

* moved $/testfilter.sh to $/util/testfilter.sh (updated Makefile accordingly)
* updated idea in Design.md
* renamed TestParse/TestFetch to TestSimpleParse/TestSimpleFetch.. more tests coming
* parse now sorts its results. test updated.
* moved regexp compilation to init instead of every time parse() is called
* misc comments and cleanup
This commit is contained in:
swaits committed 2015-02-15 00:40:36 -08:00
1 parent d716fbc6f8
commit 38224a8987
6 files changed
+20 -12

No files matched your search

+1
View File
@@ -39,6 +39,7 @@ These are features I'd like to include.
* use channels to form a pipeline
* pipeline chain should be such that when there's no more work to do the job dispatcher closes the next channel in the pipeline, thereby closing the entire chain of channels (and exiting all goroutines)
* pipeline may allows us to scale concurrency at the bottlenecks
* could use Interfaces for things like Parser, Getter, but I think best avoided until needed
* launch goroutine per fetch
* when there's no Jobs left, and nothing being fetched or parsed, convert the data struct into something useful and output it