Save WARCs in subdirectories equal to the current value of Warcprox-Meta['warc-prefix'].
E.g. if warc-prefix=='spn2' and --dir=/warcs, save them in /warcs/spn2/.
New option `--logging-conf-file` to load `logging` conf from a JSON
file.
Prefer JSON over the `configparser` format supported by
`logging.config.fileConfig` because JSON format is much better (nesting
is supported) and its easier to detect errors.
`CdxServerDedupLoader` used `max_workers=400` by default.
We make it a CLI option `--cdxserver-dedup-max-threads` with a default
value of 400.
We need to be able to tweak this setting because it creates too many CDX
queries which cause problems with our production CDX servers.
New API to allow some code from outside of warcprox proper (in a
third-party plugin for example) can close open warcs promptly when it
knows they are finished.
Add option `--blackout-period` (default=0)
When set and if the record is a duplicate (revisit record), check the
datetime of `dedup_info` and its inside the `blackout_period`, skip
writing the record to WARC.
Add some unit tests.
This is an improved implementation based on @nlevitt comments here:
https://github.com/internetarchive/warcprox/pull/92
New `--dedup-min-text-size` and `--dedup-min-binary-size` cli options
with default value = `0`.
New `DedupableMixin` which can be used in any dedup class. It is
currently used only in CDX dedup. Instead of checking `payload_size() >
0`, we now use `.is_dedupable(recorded_url)`
New utility method `RecordedUrl.is_text`.
We use `tempfile.SpooledTemporaryFile(max_size=512*1024)` to keep
recorded data before writing them to WARC.
Data are kept in memory when they are smaller than `max_size`, else they
are written to disk.
We add option `--tmp-file-max-memory-size` to make this configurable.
A higher value means less /tmp disk I/O and higher overall performance but
also increased memory usage.
Add ``ProxyingRecordingHTTPResponse.truncated`` and
``RecordedUrl.truncated`` attributes.
Set ``truncated = b'length'` when max resource size limit applies.
Add `Warc-Truncated: length` header to WARC record.
Add hidden option ``--max-resource-size`` which indicates the max file size of
a target resource in bytes. If the size is over the limit, an exception is
raised.
It is necessary to pass cookies to the CDX Server we use for deduplication.
To do this, we add the optional CLI argument
``--cdxserver-dedup-cookies="cookie1=val1;cookie2=val2"`` and if it is
available, its used in the `Cookie` HTTP header in CDX server requests.
* master:
fix running_stats thing
Update CdxServerDedup unit test
Chec writer._fname in unit test
Configurable CdxServerDedup urllib3 connection pool size
Yet another unit test fix
Change the writer unit test
fix github problem with unit test
Another fix for the unit test
Fix writer unit test
Add WarcWriter warc_filename unit test
Fix warc_filename default value
Configurable WARC filenames
urllib3 pool has default ``maxsize=1``
http://urllib3.readthedocs.io/en/latest/advanced-usage.html.
We need to set a higher value because we get warnings like this:
```
2018-01-15 20:04:10,044 18436 WARNING WarcWriterThread030(tid=18502)
urllib3.connectionpool._put_conn(connectionpool.py:277) Connection pool
is full, discarding connection: wwwb-dedup
```
We set value: ```cdxserver_maxsize = args.writer_threads or 200```.
Note that the ideal would be to use this
https://github.com/internetarchive/warcprox/blob/master/warcprox/main.py#L284
but it is initialized after dedup, there is a dependency and we cannot
use it.
* master:
not gonna bother figuring out why pypy regex is not matching https://travis-ci.org/internetarchive/warcprox/jobs/299864258#L615
fix failing test just committed, which involves running "listeners" for all urls, including those not archived; make adjustments accordingly
make test_crawl_log expect HEAD request to be logged
fix crawl log handling of WARCPROX_WRITE_RECORD request
modify test_crawl_log to expect crawl log to honor --base32 setting and add tests of WARCPROX_WRITE_RECORD request and HEAD request (not written to warc)
bump dev version number
add --crawl-log-dir option to fix failing test
create crawl log dir at startup if it doesn't exist
make test pass with py27
fix crawl log test to avoid any dedup collisions
fix crawl log test
heritrix-style crawl log support
disallow slash and backslash in warc-prefix
can't see any reason to split the main() like this (anymore?)
add missing dependency warcio to tests_require
* master:
Update docstring
Move Warcprox-Meta header construction to warcproxy
Improve test_writer tests
Replace timestamp parameter with more generic request/response syntax
Return capture timestamp
Swap fcntl.flock with fcntl.lockf
Unit test fix for Python2 compatibility
Test WarcWriter file locking when no_warc_open_suffix=True
Rename writer var and add exception handling
Acquire and exclusive file lock when not using .open WARC suffix
Add hidden --no-warc-open-suffix CLI option
Fix missing dummy url param in bigtable lookup method
back to dev version number
version 2.2 for pypi to address https://github.com/internetarchive/warcprox/issues/42
Expand comment with limit=-1 explanation
Drop unnecessary split for newline in CDX results
fix benchmarks (update command line args)
Update CdxServerDedup lookup algorithm
Pass url instead of recorded_url obj to dedup lookup methods
Filter out warc/revisit records in CdxServerDedup
Improve CdxServerDedup implementation
Fix minor CdxServerDedup unit test
Fix bug with dedup_info date encoding
Add mock pkg to run-tests.sh
Add CdxServerDedup unit tests and improve its exception handling
Add CDX Server based deduplication
cryptography lib version 2.1.1 is causing problems
Revert changes to test_warcprox.py
Revert changes to bigtable and dedup
Revert warc to previous behavior
Update unit test
Replace invalid warcfilename variable in playback
Stop using WarcRecord.REFERS_TO header and use payload_digest instead
greatly simplify automated test setup by reusing initialization code from the command line executable; this also has the benefit of testing that initialization code
avoid TypeError: 'NoneType' object is not iterable exception at shutdown
wait for rethinkdb indexes to be ready
Remove deleted ``close`` method call from test.
bump dev version number after merging pull requests
Add missing "," in deps
Remove tox.ini, move warcio to test_requires
allow very long request header lines, to support large warcprox-meta header values
Remove redundant stop() & sync() dedup methods
Remove redundant close method from DedupDb and RethinkDedupDb
Remove unused imports
Add missing packages from setup.py, add tox config.
fix python2 tests
don't use http.client.HTTPResponse.getheader() to get the content-type header, because it can return a comma-delimited string
fix zero-indexing of warc_writer_threads so they can be disabled via empty list
* master:
Update docstring
Move Warcprox-Meta header construction to warcproxy
Improve test_writer tests
Replace timestamp parameter with more generic request/response syntax
Return capture timestamp
Swap fcntl.flock with fcntl.lockf
Unit test fix for Python2 compatibility
Test WarcWriter file locking when no_warc_open_suffix=True
Rename writer var and add exception handling
Acquire and exclusive file lock when not using .open WARC suffix
Add hidden --no-warc-open-suffix CLI option
Fix missing dummy url param in bigtable lookup method
back to dev version number
version 2.2 for pypi to address https://github.com/internetarchive/warcprox/issues/42
Expand comment with limit=-1 explanation
Drop unnecessary split for newline in CDX results
fix benchmarks (update command line args)
Update CdxServerDedup lookup algorithm
Pass url instead of recorded_url obj to dedup lookup methods
Filter out warc/revisit records in CdxServerDedup
Improve CdxServerDedup implementation
Fix minor CdxServerDedup unit test
Fix bug with dedup_info date encoding
Add mock pkg to run-tests.sh
Add CdxServerDedup unit tests and improve its exception handling
Add CDX Server based deduplication
cryptography lib version 2.1.1 is causing problems
Revert changes to test_warcprox.py
Revert changes to bigtable and dedup
Revert warc to previous behavior
Update unit test
Replace invalid warcfilename variable in playback
Stop using WarcRecord.REFERS_TO header and use payload_digest instead
