warcprox

For best results while browsing through warcprox, you need to add the CA cert as a trusted cert in your browser. If you don't do that, you will get the warning when you visit each new site. But worse, any embedded https content on a different server will simply fail to load, because the browser will reject the certificate without telling you.

###Dependencies

Currently depends on tweaks branch of my fork of warctools. https://github.com/nlevitt/warctools/tree/tweaks Hopefully the changes in that branch, or something equivalent, will be incorporated into warctools mainline.

###Usage

usage: warcprox.py [-h] [-p PORT] [-b ADDRESS] [-c CACERT]
                   [--certs-dir CERTS_DIR] [-d DIRECTORY] [-z] [-n PREFIX]
                   [-s SIZE] [-v] [-q]

warcprox - WARC writing MITM HTTP/S proxy

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  port to listen on (default: 8080)
  -b ADDRESS, --address ADDRESS
                        address to listen on (default: localhost)
  -c CACERT, --cacert CACERT
                        CA certificate file; if file does not exist, it will
                        be created (default: ./warcprox-ca.pem)
  --certs-dir CERTS_DIR
                        where to store and load generated certificates
                        (default: ./warcprox-ca)
  -d DIRECTORY, --dir DIRECTORY
                        where to write warcs (default: ./warcs)
  -z, --gzip            write gzip-compressed warc records (default: False)
  -n PREFIX, --prefix PREFIX
                        WARC filename prefix (default: WARCPROX)
  -s SIZE, --size SIZE  WARC file rollover size threshold in bytes (default:
                        1000000000)
  -v, --verbose
  -q, --quiet

###To do

integration tests, unit tests
url-agnostic deduplication
unchunk and/or ungzip before storing payload, or alter request to discourage server from chunking/gzipping
check certs from proxied website, like browser does, and present browser-like warning if appropriate
keep statistics, produce reports
write cdx while crawling?
performance testing
base32 sha1 like heritrix?
configurable timeouts and stuff
evaluate ipv6 support
more explicit handling of connection closed exception during transfer? other error cases?
dns cache?? the system already does a fine job I'm thinking
keepalive with remote servers?
python3

To not do

The features below could also be part of warcprox. But maybe they don't belong here, since this is a proxy, not a crawler/robot. It can be used by a human with a browser, or by something automated, i.e. a robot. My feeling is that it's more appropriate to implement these in the robot.

politeness, i.e. throttle requests per server
fetch and obey robots.txt
alter user-agent, maybe insert something like "warcprox mitm archiving proxy; +http://archive.org/details/archive.org_bot"