backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Copy edits updated

Browse Source 
Edits for readability updated as per https://github.com/internetarchive/warcprox/pull/95#discussion_r200491731

@nlevitt please go ahead and apply your < 80 lines retroactively and I'll refrain from that in future PRs.
This commit is contained in:
Karl-Rainer Blumenthal 2018-07-06 11:33:34 -04:00 committed by Noah Levitt
parent b72192d3d0
commit fa6b98cf4e
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
README.rst
View File

@ -34,7 +34,7 @@ Normally, HTTP proxies can't read encrypted HTTPS traffic. The browser uses the
In order to capture HTTPS traffic, warcprox acts as a "man-in-the-middle" (MITM). When it receives a ``CONNECT`` directive from a client, it generates a public key certificate for the requested site, presents to the client, and proceeds to establish an encrypted connection with the client. It then makes a separate, normal HTTPS connection to the remote site. It decrypts, archives, and re-encrypts traffic in both directions.
Configuring a warcprox instance as a browser’s HTTP proxy will result in security certificate warnings because none of the certificates will be signed by trusted authorities. However, there is nothing malicious about warcprox functions. To use warcprox effectively, the client needs to disable certificate verification or add the CA certification generated by warcprox as a trusted authority. When using the latter, remember to undo this change when finished using warcprox.
Configuring a warcprox instance as a browser’s HTTP proxy will result in security certificate warnings because none of the certificates will be signed by trusted authorities. However, there is nothing malicious about warcprox functions. To use warcprox effectively, the client needs to disable certificate verification or add the CA certificate generated by warcprox as a trusted authority. When using the latter, remember to undo this change when finished using warcprox.
API
===
@ -42,7 +42,7 @@ The warcprox API may be used to retrieve information from and interact with a ru
* Retrieving status information via ``/status`` URL
* Writing WARC records via ``WARCPROX_WRITE_RECORD`` HTTP method
* Querying and editing ``Warcprox-Meta`` HTTP request header and response header
* Controlling warcprox settings via the ``Warcprox-Meta`` HTTP header
For warcprox API documentation, see: `<api.rst>`_.
@ -55,7 +55,7 @@ Warcprox avoids archiving redundant content by "deduplicating" it. The process f
3. If found, write warc ``revisit`` record referencing the url and capture time of the previous capture
4. If not found,
a. Write warc ``response`` record with full payload
a. Write ``response`` record with full payload
b. Store new entry in deduplication database
The deduplication database is partitioned into different "buckets". URLs are deduplicated only against other captures in the same bucket. If specified, the ``dedup-bucket`` field of the `Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>`_ determines the bucket. Otherwise, the default bucket is used.