Vangelis Banos 2c2c1d008a New --blackout-period option to skip writing redundant revisits to WARC ...

Add option `--blackout-period` (default=0)

When set and if the record is a duplicate (revisit record), check the
datetime of `dedup_info` and its inside the `blackout_period`, skip
writing the record to WARC.

Add some unit tests.

This is an improved implementation based on @nlevitt comments here:
https://github.com/internetarchive/warcprox/pull/92

2018-07-21 11:20:49 +00:00

..

__init__.py

move tests out of installed package dir

2016-01-26 18:47:08 -08:00

conftest.py

change rethinkdb-related command line options to use "rethinkdb urls" (parser just added to doublethink) to reduce the proliferation of rethinkdb options, and add --rethinkdb-trough-db-url option

2017-10-11 12:06:19 -07:00

Dockerfile

fix trough deployment in Dockerfile

2018-05-16 13:48:04 -07:00

run-tests.sh

only run tests in py3

2018-05-16 14:21:18 -07:00

run-trough.sh

fix trough deployment in Dockerfile

2018-05-16 13:48:04 -07:00

single-threaded-proxy.py

fix payload digest by pulling calculation up one level where content has already been transfer-decoded

2017-11-10 17:18:22 -08:00

test_dedup.py

Update CdxServerDedup unit test

2018-01-15 20:58:36 +00:00

test_ensure_rethinkdb_tables.py

fix warcprox-ensure-rethinkdb-tables and add tests

2017-11-28 10:38:38 -08:00

test_queue.py

add seconds_behind to service registry and status api, which is the length of time the next url to be written to warc has been waiting in the queue

2017-03-30 15:54:19 -07:00

test_warcprox.py

add another "wait" to fix failing test

2018-05-29 13:08:34 -07:00

test_writer.py

New --blackout-period option to skip writing redundant revisits to WARC

2018-07-21 11:20:49 +00:00