update readme

This commit is contained in:
swaits committed 2015-02-20 06:04:45 -08:00
1 parent 61514fb6c9
commit bddd144dee
1 file changed
+62
+62
View File
@@ -6,6 +6,51 @@ It crawls a single host (i.e. anything.com) and outputs a site map in JSON forma
For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com). For each page crawled, it distinguishes between links to other pages, links to assets, broken links, and remote links (i.e. someotherhost.com).
### An Example
❯ bin/docrawler https://goregex.com/
D.O. Crawler 1.0 Copyright (c) 2015 Stephen Waits <steve@waits.net> 2015-02-17
[
{
"URL": "https://goregex.com/",
"Title": "GoRegEx.com | Go Regular Expression Tester",
"Links": [
"https://goregex.com/"
],
"Assets": [
"https://goregex.com/assets/css/bootstrap-responsive.min.css",
"https://goregex.com/assets/css/bootstrap.min.css",
"https://goregex.com/assets/css/goregex.css",
"https://goregex.com/assets/img/apple-touch-icon-114x114.png",
"https://goregex.com/assets/img/apple-touch-icon-120x120.png",
"https://goregex.com/assets/img/apple-touch-icon-144x144.png",
"https://goregex.com/assets/img/apple-touch-icon-152x152.png",
"https://goregex.com/assets/img/apple-touch-icon-180x180.png",
"https://goregex.com/assets/img/apple-touch-icon-57x57.png",
"https://goregex.com/assets/img/apple-touch-icon-60x60.png",
"https://goregex.com/assets/img/apple-touch-icon-72x72.png",
"https://goregex.com/assets/img/apple-touch-icon-76x76.png",
"https://goregex.com/assets/img/favicon-160x160.png",
"https://goregex.com/assets/img/favicon-16x16.png",
"https://goregex.com/assets/img/favicon-192x192.png",
"https://goregex.com/assets/img/favicon-32x32.png",
"https://goregex.com/assets/img/favicon-96x96.png",
"https://goregex.com/assets/img/favicon.ico",
"https://goregex.com/assets/js/goregex.js"
],
"Broken": null,
"Remote": [
"http://code.google.com/p/re2/wiki/Syntax",
"http://golang.org/",
"http://golang.org/pkg/regexp/",
"https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js",
"https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"
]
}
]
## Design ## ## Design ##
* The main crawler loop is in `docrawler.go:docrawl()`. * The main crawler loop is in `docrawler.go:docrawl()`.
@@ -21,6 +66,8 @@ After cloning this repository, run:
make vendor && make make vendor && make
This will `fmt`, `lint`, `vet`, and `build` the source into an executable at `bin/docrawler`.
In order to run the tests: In order to run the tests:
make test make test
@@ -29,10 +76,25 @@ If you'd like to automatically run tests any time a file is changed:
make autotest make autotest
To see stats about the code, install cloc (`brew install cloc` on OS X) and then:
make stats
To look at the test coverage reports in your browser:
make cover
To deploy, just copy the executable `bin/docrawler` to a destination of your choosing. To deploy, just copy the executable `bin/docrawler` to a destination of your choosing.
*Developed on OS X. Also tested on Windows 7 under MinGW+bash.* *Developed on OS X. Also tested on Windows 7 under MinGW+bash.*
### Why a Makefile? Aren't those old???
* It's clean.
* It's portable.
* It lets me package the app in a way that's trivially easy for anyone to build.
* It makes it so I can easily vendor third-party packages outside of the main `./src` tree.
## Who do I talk to? ## ## Who do I talk to? ##
Written by Stephen Waits. Please contact me at <mailto:steve@waits.net> with any questions. Written by Stephen Waits. Please contact me at <mailto:steve@waits.net> with any questions.