Compare commits
6
Commits
bddd144dee
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b2642dcfac
|
||
|
|
9cc9f3d7a3
|
||
|
|
0d0c8af1d6 | ||
|
|
f8af5be62d | ||
|
|
a5496b51b8 | ||
|
|
32c214b3ef |
No files matched your search
@@ -0,0 +1,19 @@
|
||||
Copyright 2015 Stephen Waits <steve@waits.net>
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the “Software”), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -6,7 +6,7 @@ It crawls a single host (i.e. anything.com) and outputs a site map in JSON forma
|
||||
|
||||
For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
|
||||
|
||||
### An Example
|
||||
### Example Usage and Output
|
||||
|
||||
❯ bin/docrawler https://goregex.com/
|
||||
|
||||
@@ -56,14 +56,16 @@ For each page crawled, it distinguishes between links to other pages, links to a
|
||||
* The main crawler loop is in `docrawler.go:docrawl()`.
|
||||
* From there I maintain a hash of links we've already crawled.
|
||||
* Each link gets an `httpItem{}` struct instanced, which holds its crawl state.
|
||||
* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter.
|
||||
* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable via command line parameter.
|
||||
* These goroutines listen for `*httpItem{}` on one channel, crawl it, fill out its results, and send it back to `docrawl()` on another channel.
|
||||
* `docrawl()` is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on.
|
||||
|
||||
## How do I get set up? ##
|
||||
|
||||
After cloning this repository, run:
|
||||
Clone this repository, fetch the dependencies, and build:
|
||||
|
||||
hg clone https://bitbucket.org/swaits/docrawler/
|
||||
cd docrawler
|
||||
make vendor && make
|
||||
|
||||
This will `fmt`, `lint`, `vet`, and `build` the source into an executable at `bin/docrawler`.
|
||||
@@ -76,11 +78,11 @@ If you'd like to automatically run tests any time a file is changed:
|
||||
|
||||
make autotest
|
||||
|
||||
To see stats about the code, install cloc (`brew install cloc` on OS X) and then:
|
||||
To see stats about the code, install `cloc` (with `brew install cloc` on OS X) and then:
|
||||
|
||||
make stats
|
||||
|
||||
To look at the test coverage reports in your browser:
|
||||
To view at the test coverage reports in your browser:
|
||||
|
||||
make cover
|
||||
|
||||
@@ -94,6 +96,7 @@ To deploy, just copy the executable `bin/docrawler` to a destination of your cho
|
||||
* It's portable.
|
||||
* It lets me package the app in a way that's trivially easy for anyone to build.
|
||||
* It makes it so I can easily vendor third-party packages outside of the main `./src` tree.
|
||||
* Everyone is vendoring their own way in Go. I'm not in love with any particular style. But this is simple enough for this case.
|
||||
|
||||
## Who do I talk to? ##
|
||||
|
||||
|
||||
Reference in new issue
Block a user