Merge branch 'dedup-fixes' into qa

This commit is contained in:
Barbara Miller 2019-06-20 14:55:45 -07:00
commit fa7d5e9326

View File

@ -90,7 +90,7 @@ for deduplication works similarly to deduplication by `Heritrix
a. Write ``response`` record with full payload
b. Store new entry in deduplication database (can be disabled, see
`Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>`_
`Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>`_)
The deduplication database is partitioned into different "buckets". URLs are
deduplicated only against other captures in the same bucket. If specified, the