swaits 3e35fdb488 some cleanup and refactoring ...

* moved crawlItem into fetch.go
* renamed fetchPage fetchItem
* doCrawl now takes number of workers as a parameter
* command line flag parsing added

2015-02-17 10:36:48 -08:00

.vendor

add coverage to Makefile (make cover)

2015-02-15 12:40:56 -08:00

docs

update Design doc status for each item

2015-02-17 01:32:45 -08:00

src/docrawler

some cleanup and refactoring

2015-02-17 10:36:48 -08:00

util

won't recrawl pages that are the same after stripping #anchor (blah.html#about, blah.html#contact, etc.)

2015-02-16 16:26:16 -08:00

.flow

flow initialization: Added configuration file.

2015-02-14 23:50:10 -08:00

.hgignore

add coverage to Makefile (make cover)

2015-02-15 12:40:56 -08:00

Makefile

split tests out from everything else in cloc (make stats)

2015-02-15 23:12:01 -08:00

S

Description

Simple web crawler written in Go.

MIT

5.1 MiB

0 Stars 1 Watchers 0 Forks

Languages

Go 96.5%

JavaScript 1.3%

HTML 0.8%

CSS 0.6%

Emacs Lisp 0.3%

Other 0.3%