Crawler Requirements

Minimum Features

This is the assignment (directly) from DO.

We do ask that you put a heavy emphasis on testing in the assignment.
written in a modern language
limited to one domain - so when crawling digitalocean.com it would crawl all pages within the digitalocean.com domain, but not follow the links to our Facebook or Instagram accounts or subdomains like cloud.digitalocean.com.
It should be Given a URL
it should output a site map
it should show which static assets each page depends on, and the links between pages.
Choose the most appropriate data structure to store & display this site map.
Build this as you would build something for production - it's fine if you don't finish everything, but focus on code quality and write tests.
We're interested in how you code and how you test your code.

These are features I'd like to include.

efficient, optimal
handles base URL properly (for root & relative links)
documented
uses standard Go practices
uses Go stdlib (only!)
thoroughly tested
configurable on command line (defaults are for company specs)
variable output formats: json, dot
adheres to URL RFC (as far as case sensitivity, acceptable character sets, etc.)
go gettable
go fmt'ed, vet'ed, lint'ed
polite - robots.txt support
robust - detect infinite loops
throttling
use Go context pattern
supports http & https

These are features that sound cool but are probably out of scope.