backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Merge pull request #173 from internetarchive/increase_batch_sec

Browse Source 
tune MIN_BATCH_SEC, MAX_BATCH_SEC for fewer dedup errors
This commit is contained in:
Barbara Miller 2022-06-24 11:13:18 -07:00 committed by GitHub
commit d253ea85c3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
warcprox/__init__.py
View File

@ -175,8 +175,10 @@ class BaseStandardPostfetchProcessor(BasePostfetchProcessor):
class BaseBatchPostfetchProcessor(BasePostfetchProcessor):
MAX_BATCH_SIZE = 500
MAX_BATCH_SEC = 30
MIN_BATCH_SEC = 10
MAX_BATCH_SEC = 60
MIN_BATCH_SEC = 30
# these updated batch seconds values have resulted in fewer reported dedup
# errors and otherwise have worked well in qa
def _get_process_put(self):
batch = []