64 Commits

Author	SHA1	Message	Date
Noah Levitt	320df0565e	support "soft limits" which result in a different response code (430) than regular (hard) limits (which result in a 420)	2016-06-27 16:07:20 -05:00
Noah Levitt	fabd732b7f	couple of fixes for host limits	2016-06-24 21:58:37 -05:00
Noah Levitt	2fe0c2f25b	support for tallying substats of a configured bucket by host, and enforcing limits host limits using those stats, with tests	2016-06-24 20:04:27 -05:00
Noah Levitt	d48e2c462d	add a start() method to the two classes that save data to rethinkdb periodically in batches, instead of starting the timer in __init__	2016-06-16 00:04:59 +00:00
Noah Levitt	4bb3556709	implement enforcement of Warcprox-Meta header block rules; includes automated tests	2016-05-10 23:11:47 +00:00
Noah Levitt	6f10e2708d	disable tor test to give travis build a chance to pass tests (waiting on https://github.com/travis-ci/apt-package-whitelist/issues/1753)	2016-04-06 19:39:28 -07:00
Noah Levitt	2c65ff89fa	add license headers	2016-04-06 19:37:55 -07:00
Noah Levitt	42a81d8f8f	fix bug where two warc-payload-digest headers were written to revisit records	2016-03-15 06:27:21 +00:00
Noah Levitt	4bb7e043d4	wait longer for stats to be updated in test_limits(), now that rethinkdb stats are pushed only every 2.0 seconds	2016-01-26 18:47:08 -08:00
Noah Levitt	00dc9eed84	new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites	2016-01-26 18:47:08 -08:00
Noah Levitt	18cc818cf0	more timing tweaks to make sure tests pass, improved logging etc	2016-01-26 18:47:08 -08:00
Noah Levitt	e67c7be5bc	service registry init	2016-01-26 18:47:08 -08:00
Noah Levitt	95e611a5d0	update stats in RethinkDb asynchronously, since profiling shows this to be a bottleneck in WarcWriterThread (which in turn makes it a bottleneck for the whole app)	2016-01-26 18:47:08 -08:00
Noah Levitt	4c380dcc41	move tests out of installed package dir	2016-01-26 18:47:08 -08:00