mirror of
https://github.com/internetarchive/warcprox.git
synced 2025-01-18 13:22:09 +01:00
restore 80 column lines
This commit is contained in:
Noah Levitt
parent
fa6b98cf4e
commit
9da5e86b67
93
README.rst
93
README.rst
@ -3,9 +3,19 @@ Warcprox - WARC writing MITM HTTP/S proxy
|
||||
.. image:: https://travis-ci.org/internetarchive/warcprox.svg?branch=master
|
||||
:target: https://travis-ci.org/internetarchive/warcprox
|
||||
|
||||
Warcprox is an HTTP proxy designed for web archiving applications. When used in parallel with `brozzler <https://github.com/internetarchive/brozzler>`_ it supports a comprehensive, modern, and distributed archival web capture system. Warcprox stores its traffic to disk in the `Web ARChive (WARC) file format <https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/>`_, which may then be accessed with web archival replay software like `OpenWayback <https://github.com/iipc/openwayback>`_ and `pywb <https://github.com/webrecorder/pywb>`_. It captures encrypted HTTPS traffic by using the "man-in-the-middle" technique (see the `Man-in-the-middle`_ section for more info).
|
||||
Warcprox is an HTTP proxy designed for web archiving applications. When used in
|
||||
parallel with `brozzler <https://github.com/internetarchive/brozzler>`_ it
|
||||
supports a comprehensive, modern, and distributed archival web capture system.
|
||||
Warcprox stores its traffic to disk in the `Web ARChive (WARC) file format
|
||||
<https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/>`_,
|
||||
which may then be accessed with web archival replay software like `OpenWayback
|
||||
<https://github.com/iipc/openwayback>`_ and `pywb
|
||||
<https://github.com/webrecorder/pywb>`_. It captures encrypted HTTPS traffic by
|
||||
using the "man-in-the-middle" technique (see the `Man-in-the-middle`_ section
|
||||
for more info).
|
||||
|
||||
Warcprox was originally based on `pymiproxy <https://github.com/allfro/pymiproxy>`_ by Nadeem Douba.
|
||||
Warcprox was originally based on `pymiproxy
|
||||
<https://github.com/allfro/pymiproxy>`_ by Nadeem Douba.
|
||||
|
||||
.. contents::
|
||||
|
||||
@ -30,15 +40,31 @@ Try ``warcprox --help`` for documentation on command line options.
|
||||
|
||||
Man-in-the-middle
|
||||
=================
|
||||
Normally, HTTP proxies can't read encrypted HTTPS traffic. The browser uses the HTTP ``CONNECT`` method to establish a tunnel through the proxy, and the proxy merely routes raw bytes between the client and server. Since the bytes are encrypted, the proxy can't make sense of the information that it proxies. This nonsensical encrypted data is not typically useful for web archiving purposes.
|
||||
Normally, HTTP proxies can't read encrypted HTTPS traffic. The browser uses the
|
||||
HTTP ``CONNECT`` method to establish a tunnel through the proxy, and the proxy
|
||||
merely routes raw bytes between the client and server. Since the bytes are
|
||||
encrypted, the proxy can't make sense of the information that it proxies. This
|
||||
nonsensical encrypted data is not typically useful for web archiving purposes.
|
||||
|
||||
In order to capture HTTPS traffic, warcprox acts as a "man-in-the-middle" (MITM). When it receives a ``CONNECT`` directive from a client, it generates a public key certificate for the requested site, presents to the client, and proceeds to establish an encrypted connection with the client. It then makes a separate, normal HTTPS connection to the remote site. It decrypts, archives, and re-encrypts traffic in both directions.
|
||||
In order to capture HTTPS traffic, warcprox acts as a "man-in-the-middle"
|
||||
(MITM). When it receives a ``CONNECT`` directive from a client, it generates a
|
||||
public key certificate for the requested site, presents to the client, and
|
||||
proceeds to establish an encrypted connection with the client. It then makes a
|
||||
separate, normal HTTPS connection to the remote site. It decrypts, archives,
|
||||
and re-encrypts traffic in both directions.
|
||||
|
||||
Configuring a warcprox instance as a browser’s HTTP proxy will result in security certificate warnings because none of the certificates will be signed by trusted authorities. However, there is nothing malicious about warcprox functions. To use warcprox effectively, the client needs to disable certificate verification or add the CA certificate generated by warcprox as a trusted authority. When using the latter, remember to undo this change when finished using warcprox.
|
||||
Configuring a warcprox instance as a browser’s HTTP proxy will result in
|
||||
security certificate warnings because none of the certificates will be signed
|
||||
by trusted authorities. However, there is nothing malicious about warcprox
|
||||
functions. To use warcprox effectively, the client needs to disable certificate
|
||||
verification or add the CA certificate generated by warcprox as a trusted
|
||||
authority. When using the latter, remember to undo this change when finished
|
||||
using warcprox.
|
||||
|
||||
API
|
||||
===
|
||||
The warcprox API may be used to retrieve information from and interact with a running warcprox instance, including:
|
||||
The warcprox API may be used to retrieve information from and interact with a
|
||||
running warcprox instance, including:
|
||||
|
||||
* Retrieving status information via ``/status`` URL
|
||||
* Writing WARC records via ``WARCPROX_WRITE_RECORD`` HTTP method
|
||||
@ -48,37 +74,58 @@ For warcprox API documentation, see: `<api.rst>`_.
|
||||
|
||||
Deduplication
|
||||
=============
|
||||
Warcprox avoids archiving redundant content by "deduplicating" it. The process for deduplication works similarly to deduplication by `Heritrix <https://github.com/internetarchive/heritrix3>`_ and other web archiving tools:
|
||||
Warcprox avoids archiving redundant content by "deduplicating" it. The process
|
||||
for deduplication works similarly to deduplication by `Heritrix
|
||||
<https://github.com/internetarchive/heritrix3>`_ and other web archiving tools:
|
||||
|
||||
1. While fetching URL, calculate payload content digest (typically SHA1 checksum value)
|
||||
2. Look up digest in deduplication database (warcprox currently supports `sqlite <https://sqlite.org/>`_ by default, `rethinkdb <https://github.com/rethinkdb/rethinkdb>`_ with two different schemas, and `trough <https://github.com/internetarchive/trough>`_)
|
||||
3. If found, write warc ``revisit`` record referencing the url and capture time of the previous capture
|
||||
1. While fetching URL, calculate payload content digest (typically SHA1
|
||||
checksum value)
|
||||
2. Look up digest in deduplication database (warcprox currently supports
|
||||
`sqlite <https://sqlite.org/>`_ by default, `rethinkdb
|
||||
<https://github.com/rethinkdb/rethinkdb>`_ with two different schemas, and
|
||||
`trough <https://github.com/internetarchive/trough>`_)
|
||||
3. If found, write warc ``revisit`` record referencing the url and capture time
|
||||
of the previous capture
|
||||
4. If not found,
|
||||
|
||||
a. Write ``response`` record with full payload
|
||||
b. Store new entry in deduplication database
|
||||
|
||||
The deduplication database is partitioned into different "buckets". URLs are deduplicated only against other captures in the same bucket. If specified, the ``dedup-bucket`` field of the `Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>`_ determines the bucket. Otherwise, the default bucket is used.
|
||||
The deduplication database is partitioned into different "buckets". URLs are
|
||||
deduplicated only against other captures in the same bucket. If specified, the
|
||||
``dedup-bucket`` field of the `Warcprox-Meta HTTP request header
|
||||
<api.rst#warcprox-meta-http-request-header>`_ determines the bucket. Otherwise,
|
||||
the default bucket is used.
|
||||
|
||||
Deduplication can be disabled entirely by starting warcprox with the argument
|
||||
``--dedup-db-file=/dev/null``.
|
||||
|
||||
Statistics
|
||||
==========
|
||||
Warcprox stores some crawl statistics to sqlite or rethinkdb. These are consulted for enforcing ``limits`` and ``soft-limits`` (see `Warcprox-Meta fields <api.rst#warcprox-meta-fields>`_), and can also be consulted by other processes outside of warcprox, such as for crawl job reporting.
|
||||
Warcprox stores some crawl statistics to sqlite or rethinkdb. These are
|
||||
consulted for enforcing ``limits`` and ``soft-limits`` (see `Warcprox-Meta
|
||||
fields <api.rst#warcprox-meta-fields>`_), and can also be consulted by other
|
||||
processes outside of warcprox, such as for crawl job reporting.
|
||||
|
||||
Statistics are grouped by "bucket". Every capture is counted as part of the ``__all__`` bucket. Other buckets can be specified in the ``Warcprox-Meta`` request header. The fallback bucket in case none is specified is called ``__unspecified__``.
|
||||
Statistics are grouped by "bucket". Every capture is counted as part of the
|
||||
``__all__`` bucket. Other buckets can be specified in the ``Warcprox-Meta``
|
||||
request header. The fallback bucket in case none is specified is called
|
||||
``__unspecified__``.
|
||||
|
||||
Within each bucket are three sub-buckets:
|
||||
|
||||
* ``new`` - tallies captures for which a complete record (usually a ``response`` record) was written to a WARC file
|
||||
* ``revisit`` - tallies captures for which a ``revisit`` record was written to a WARC file
|
||||
* ``total`` - includes all URLs processed, even those not written to a WARC file, and so may be greater than the sum of new and revisit records
|
||||
* ``new`` - tallies captures for which a complete record (usually a
|
||||
``response`` record) was written to a WARC file
|
||||
* ``revisit`` - tallies captures for which a ``revisit`` record was written to
|
||||
a WARC file
|
||||
* ``total`` - includes all URLs processed, even those not written to a WARC
|
||||
file, and so may be greater than the sum of new and revisit records
|
||||
|
||||
Within each of these sub-buckets, warcprox generates two kinds of statistics:
|
||||
|
||||
* ``urls`` - simple count of URLs
|
||||
* ``wire_bytes`` - sum of bytes received over the wire from the remote server for each URL, including HTTP headers
|
||||
* ``wire_bytes`` - sum of bytes received over the wire from the remote server
|
||||
for each URL, including HTTP headers
|
||||
|
||||
For historical reasons, the default sqlite store keeps statistics as JSON blobs::
|
||||
|
||||
@ -90,9 +137,17 @@ For historical reasons, the default sqlite store keeps statistics as JSON blobs:
|
||||
|
||||
Plugins
|
||||
=======
|
||||
Warcprox supports a limited notion of plugins by way of the ``--plugin`` command line argument. Plugin classes are loaded from the regular python module search path. They are instantiated with one argument that contains the values of all command line arguments, ``warcprox.Options``. Legacy plugins with constructors that take no arguments are also supported. Plugins should either have a method ``notify(self, recorded_url, records)`` or should subclass ``warcprox.BasePostfetchProcessor``. More than one plugin can be configured by specifying ``--plugin`` multiples times.
|
||||
Warcprox supports a limited notion of plugins by way of the ``--plugin``
|
||||
command line argument. Plugin classes are loaded from the regular python module
|
||||
search path. They are instantiated with one argument that contains the values
|
||||
of all command line arguments, ``warcprox.Options``. Legacy plugins with
|
||||
constructors that take no arguments are also supported. Plugins should either
|
||||
have a method ``notify(self, recorded_url, records)`` or should subclass
|
||||
``warcprox.BasePostfetchProcessor``. More than one plugin can be configured by
|
||||
specifying ``--plugin`` multiples times.
|
||||
|
||||
See a minimal example `here <https://github.com/internetarchive/warcprox/blob/318405e795ac0ab8760988a1a482cf0a17697148/warcprox/__init__.py#L165>`__.
|
||||
See a minimal example `here
|
||||
<https://github.com/internetarchive/warcprox/blob/318405e795ac0ab8760988a1a482cf0a17697148/warcprox/__init__.py#L165>`__.
|
||||
|
||||
License
|
||||
=======
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user