backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Expand comment with limit=-1 explanation

Browse Source
This commit is contained in:
Vangelis Banos 2017-10-25 20:28:56 +00:00
parent 4282032772
commit 6beb19dc16
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
warcprox/dedup.py
View File

@ -206,6 +206,10 @@ class CdxServerDedup(object):
computed on the original content, after decoding Content-Encoding and computed on the original content, after decoding Content-Encoding and
Transfer-Encoding, if any), if they match, write a revisit record. Transfer-Encoding, if any), if they match, write a revisit record.
Get only the last item (limit=-1) because Wayback Machine has special
performance optimisation to handle that. limit < 0 is very inefficient
in general. Maybe it could be configurable in the future.
:param digest_key: b'sha1:<KEY-VALUE>' (prefix is optional). :param digest_key: b'sha1:<KEY-VALUE>' (prefix is optional).
Example: b'sha1:B2LTWWPUOYAH7UIPQ7ZUPQ4VMBSVC36A' Example: b'sha1:B2LTWWPUOYAH7UIPQ7ZUPQ4VMBSVC36A'
:param url: Target URL string :param url: Target URL string