Compare commits

..
6 Commits
Author SHA1 Message Date
swaits b2642dcfac chore: remove Sourcehut CI manifest
Migrated to git.swaits.com; Sourcehut builds no longer run.
2026-08-31 17:16:25 -06:00
swaits 9cc9f3d7a3 add github mirror 2024-01-29 10:38:18 -07:00
swaits 0d0c8af1d6 add LICENSE 2020-04-24 09:42:05 -07:00
swaits f8af5be62d Merge from develop 2015-02-20 06:17:23 -08:00
swaits a5496b51b8 more readme updates :) 2015-02-20 06:17:05 -08:00
swaits 32c214b3ef Merge from default 2015-02-20 06:05:19 -08:00
2 changed files with 27 additions and 5 deletions

No files matched your search

+19
View File
@@ -0,0 +1,19 @@
Copyright 2015 Stephen Waits <steve@waits.net>
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the “Software”), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+8 -5
View File
@@ -6,7 +6,7 @@ It crawls a single host (i.e. anything.com) and outputs a site map in JSON forma
For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
### An Example
### Example Usage and Output
❯ bin/docrawler https://goregex.com/
@@ -56,14 +56,16 @@ For each page crawled, it distinguishes between links to other pages, links to a
* The main crawler loop is in `docrawler.go:docrawl()`.
* From there I maintain a hash of links we've already crawled.
* Each link gets an `httpItem{}` struct instanced, which holds its crawl state.
* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable from via command line parameter.
* A number of crawler goroutines are fired off in the beginning so that we can control precisely how many http fetches happen at a single time. This number is configurable via command line parameter.
* These goroutines listen for `*httpItem{}` on one channel, crawl it, fill out its results, and send it back to `docrawl()` on another channel.
* `docrawl()` is predominantly a for-select loop, selecting on a ticker (which is used to update status to the console and check for crawl completion), and on the "work finished" channel that the crawlers send data back on.
## How do I get set up? ##
After cloning this repository, run:
Clone this repository, fetch the dependencies, and build:
hg clone https://bitbucket.org/swaits/docrawler/
cd docrawler
make vendor && make
This will `fmt`, `lint`, `vet`, and `build` the source into an executable at `bin/docrawler`.
@@ -76,11 +78,11 @@ If you'd like to automatically run tests any time a file is changed:
make autotest
To see stats about the code, install cloc (`brew install cloc` on OS X) and then:
To see stats about the code, install `cloc` (with `brew install cloc` on OS X) and then:
make stats
To look at the test coverage reports in your browser:
To view at the test coverage reports in your browser:
make cover
@@ -94,6 +96,7 @@ To deploy, just copy the executable `bin/docrawler` to a destination of your cho
* It's portable.
* It lets me package the app in a way that's trivially easy for anyone to build.
* It makes it so I can easily vendor third-party packages outside of the main `./src` tree.
* Everyone is vendoring their own way in Go. I'm not in love with any particular style. But this is simple enough for this case.
## Who do I talk to? ##