backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Merge branch 'dedup-fixes' into qa

Browse Source
This commit is contained in:
Barbara Miller 2019-06-20 14:52:46 -07:00
commit d495674e44
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.rst
View File

@ -90,7 +90,7 @@ for deduplication works similarly to deduplication by `Heritrix
a. Write ``response`` record with full payload a. Write ``response`` record with full payload
b. Store new entry in deduplication database (can be disabled, see b. Store new entry in deduplication database (can be disabled, see
`Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>` `Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>`_
The deduplication database is partitioned into different "buckets". URLs are The deduplication database is partitioned into different "buckets". URLs are
deduplicated only against other captures in the same bucket. If specified, the deduplicated only against other captures in the same bucket. If specified, the