From da7c5ba57c456c00b5061b6cfbdce694b0504dc1 Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Sat, 14 Feb 2015 21:11:10 -0800 Subject: [PATCH] design doc update --- docs/Design.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/Design.md b/docs/Design.md index 4b2a587..183f389 100644 --- a/docs/Design.md +++ b/docs/Design.md @@ -35,11 +35,20 @@ These are features I'd like to include. * use Go context pattern * supports http & https +## Architecture + +* use channels to form a pipeline +* pipeline chain should be such that when there's no more work to do the job dispatcher closes the next channel in the pipeline, thereby closing the entire chain of channels (and exiting all goroutines) +* could use Interfaces for things like Parser, Getter, but I think best avoided until needed +* launch goroutine per fetch +* when there's no Jobs left, and nothing being fetched or parsed, convert the data struct into something useful and output it + + ## Wishlist These are features that sound cool but are probably out of scope. * distributed * sitemap support https://en.wikipedia.org/wiki/Sitemaps -* noindex tag -* noidext http header +* noindex tag (this might be pretty easy) +* noindex http header (actually this seems pretty easy)