backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

dedup-bucket is required in Warcprox-Meta to do dedup

Browse Source 
Modify `DedupableMixin.should_dedup` to check Warcprox-Meta for
`dedup-bucket` in order to perform dedup.
This commit is contained in:
Vangelis Banos 2018-05-04 14:27:42 +00:00
parent 9baa2e22d5
commit 432e42803c
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
warcprox/dedup.py
View File

@ -44,8 +44,12 @@ class DedupableMixin(object):
def should_dedup(self, recorded_url):
"""Check if we should try to run dedup on resource based on payload
size compared with min text/binary dedup size options. Return Boolean.
size compared with min text/binary dedup size options.
`dedup-bucket` is required in Warcprox-Meta to perform dedup.
Return Boolean.
"""
if "dedup-bucket" not in recorded_url.warcprox_meta:
return False
if recorded_url.is_text():
return recorded_url.response_recorder.payload_size() > self.min_text_size
else: