46 lines
1.6 KiB
Markdown
46 lines
1.6 KiB
Markdown
# Crawler Requirements
|
|
|
|
## Minimum Features
|
|
|
|
This is the assignment (directly) from DO.
|
|
|
|
* We do ask that you put a heavy emphasis on testing in the assignment.
|
|
* written in a modern language
|
|
* limited to one domain - so when crawling digitalocean.com it would crawl all pages within the digitalocean.com domain, but not follow the links to our Facebook or Instagram accounts or subdomains like cloud.digitalocean.com.
|
|
* It should be Given a URL
|
|
* it should output a site map
|
|
* it should show which static assets each page depends on, and the links between pages.
|
|
* Choose the most appropriate data structure to store & display this site map.
|
|
* Build this as you would build something for production - it's fine if you don't finish everything, but focus on code quality and write tests.
|
|
* We're interested in how you code and how you test your code.
|
|
|
|
## Desired
|
|
|
|
These are features I'd like to include.
|
|
|
|
* efficient, optimal
|
|
* handles base URL properly (for root & relative links)
|
|
* documented
|
|
* uses standard Go practices
|
|
* uses Go stdlib (only!)
|
|
* thoroughly tested
|
|
* configurable on command line (defaults are for company specs)
|
|
* variable output formats: json, dot
|
|
* adheres to URL RFC (as far as case sensitivity, acceptable character sets, etc.)
|
|
* go gettable
|
|
* go fmt'ed, vet'ed, lint'ed
|
|
* polite - robots.txt support
|
|
* robust - detect infinite loops
|
|
* throttling
|
|
* use Go context pattern
|
|
* supports http & https
|
|
|
|
## Wishlist
|
|
|
|
These are features that sound cool but are probably out of scope.
|
|
|
|
* distributed
|
|
* sitemap support https://en.wikipedia.org/wiki/Sitemaps
|
|
* noindex tag
|
|
* noidext http header
|