* master:
bump version after merge
include warcprox host and port in filenames
replace pencil drawing with nice diagram by James
fix bug
readable stack traces, thanks py.test
--quiet means NOTICE level logging
tweak max threads option handling
set socket timeout for tor .onion fetching
WARCPROX_WRITE_RECORD respect buffer size setting
--help-hidden for help on hidden args
half-baked readme section on warcprox architecture
bump dev version number after merge
restore 80 column lines
Copy edits updated
Copy edits
update cryptography dep version
use SpooledTemporaryFile for WARCPROX_WRITE_RECORD
Apply blackout on when dedup URL equals request URL
New --blackout-period option to skip writing redundant revisits to WARC
Add option `--blackout-period` (default=0)
When set and if the record is a duplicate (revisit record), check the
datetime of `dedup_info` and its inside the `blackout_period`, skip
writing the record to WARC.
Add some unit tests.
This is an improved implementation based on @nlevitt comments here:
https://github.com/internetarchive/warcprox/pull/92
* master:
bump dev version after pull request
dumb mistake
hopefully fix a trough dedup concurrency bug
some logging improvements
test should expose trough dedup concurrency bug
run trough with python 3.6 plus travis cleanup
record request method in crawl log if not GET
back to dev version number
2.4b2 for pypi
setuptools likes README.rst not readme.rst
docker image python:3 is now using 3.7 and building pyyaml < 3.13 fails
yaml/pyyaml#126
also filed pull request to update trough's pyyaml dependency spec
internetarchive/trough#20
* master:
log exception and continue 🤞 if schema reg fails
log stack trace in case batch postprocessor raises
bump dev version after merge
more edits
more little edits
explain a bit about mitm
little edits
describe the last two remaining fields
fixlets
more progress on documenting "limits"
add another "wait" to fix failing test
fix bug in limits enforcement
docs still in progress
new checks exposing bug in limits enforcement
working on "limits" and "soft-limits"
explain warcprox-meta "blocks"
starting to explain some warcprox-meta fields
short sectioni on stats
barely starting to flesh out warcprox-meta section
explain deduplication
starting to talk about warcprox-meta
fix failure message in test_return_capture_timestamp
double the backticks
stubby api docs
rename README.rst -> readme.rst
add some debug logging in BatchTroughLoader
just one should_dedup() for trough dedup
only run tests in py3
fix trough deployment in Dockerfile
fix test_dedup_min_text_size failure?
rewrite test_dedup_min_size() to account for
we want to save all captures to the big "captures"
default values for dedup_min_text_size et al
the fact that we always save a record to the big captures table,
partly by adding a new check that --dedup-min-*-size is respected even
if there is an entry in the dedup db for the sha1
* master:
support "captures-bucket" for backward compatibility
Add hidden CLI option --dedup-only-with-bucket
dedup-bucket is required in Warcprox-Meta to do dedup
Rename captures-bucket to dedup-bucket in Warcprox-Meta
bump dev version number after #86
Use DedupableMixin in RethinkCapturesDedup
Fix travis-ci unit test issue
Add unit tests
Remove method decorate_with_dedup_info
Use DedupableMixin in all dedup classes
default to 100 proxy threads, 1 warc writer thread
include warc writer worker threads in profiling
cap the number of urls queued for warc writing
oops! /status has been lying about queued urls
Configurable min dedupable size for text/binary resources
bump dev version number after PR
Fix Accept-Encoding request header
CDX dedup improvements
bump dev version number after PR
make test server multithreaded so tests will pass
always call socket.shutdown() to close connections
bump dev version number
close connection when truncating response
test another request after truncated response
close all remote connections at shutdown
tweak tests to make them pass now that keepalive
enable keepalive on test http server
more logging
remove some debug logging
this is some logging meant to debug the mysterious
work around odd problem (see comment in code)
`test_dedup_https` fails on travis-ci.
https://travis-ci.org/internetarchive/warcprox/jobs/370598950
We didn't touch that at all but worked on `test_dedup_min_size` which
runs just before that. We move `test_dedup_min_size` to the end of the
file hoping to resolve this.
Create two very small dummy responses (text, 2 bytes and binary, 4 bytes).
Use options --dedup-min-text-size=3 and --dedup-min-binary-size=5.
Ensure that due to the effects of these options, dedup is not happening.
Existing dedup unit tests are not affected at all.
As of fairly recently, warcprox does keepalive with the remote server
using the urllib3 connection pool. The test http server in
test_warcprox.py was acting as if it supported keepalive (sending
HTTP/1.1 and not sending "Connection: close"). But in fact it did not
support keepalive. It was closing the connection after each request.
Depending on the timing of what was happening in different threads,
sometimes the client thread would try to send another request on a
connection it still believed to be open for keepalive. Then the server
side would complete its request processing and close the connection.
This resulted in test failures with error messages like this (depending
on python version):
2018-04-03 21:20:06,555 12586 ERROR MainThread warcprox.mitmproxy.MitmProxyHandler.do_COMMAND(mitmproxy.py:389) error from remote server(?) None: BadStatusLine("''",)
2018-04-04 19:06:29,599 11632 ERROR MainThread warcprox.mitmproxy.MitmProxyHandler.do_COMMAND(mitmproxy.py:389) error from remote server(?) None: RemoteDisconnected('Remote end closed connection without response',)
For instance https://travis-ci.org/internetarchive/warcprox/jobs/362288603
