Noah Levitt
|
e5f2c348e2
|
fix dockerized automated tests now that phusion/baseimage is ubuntu xenial
|
2016-11-15 12:09:09 -08:00 |
|
Noah Levitt
|
3b167459e3
|
change tested idns to valid idna2008 now that requests 2.12.0 enforces that (for better or worse, see https://github.com/kennethreitz/requests/issues/3687)
|
2016-11-15 12:08:07 -08:00 |
|
Noah Levitt
|
fa1e8d3af4
|
allow travis-ci failures for python-nightly and also test 3.6-dev (but allow failures);
enable the onion site tor test because apparently travis-ci is allowing me to
install tor now, see https://travis-ci.org/internetarchive/warcprox/jobs/169101744
although https://github.com/travis-ci/apt-package-whitelist/issues/1753 is still open
|
2016-10-19 18:24:25 -07:00 |
|
Noah Levitt
|
719380e612
|
refactor some general mitm proxy stuff into mitmproxy.py
|
2016-10-19 15:32:58 -07:00 |
|
Noah Levitt
|
314be33707
|
new test that reveals connection hang on https urls missing a content-length http response header (not chunked and server leaves connection open) -- reported by Alex Osborne
|
2016-10-19 13:43:44 -07:00 |
|
Noah Levitt
|
46c24833ff
|
emoji idn fails with python 2.7, so test with a BMP unicode character
|
2016-06-29 17:16:50 -05:00 |
|
Noah Levitt
|
33775d360a
|
comment out segfaulting test
|
2016-06-29 16:47:54 -05:00 |
|
Noah Levitt
|
a59871e17b
|
idn support, at least for domain limits (getting a segfault in tests on mac however, let's see what happens on travis-ci)
|
2016-06-29 15:54:40 -05:00 |
|
Noah Levitt
|
c9e403585b
|
switching from host limits to domain limits, which apply in aggregate to the host and subdomains
|
2016-06-29 14:56:14 -05:00 |
|
Noah Levitt
|
2c8b194090
|
really only apply host limits to the host
|
2016-06-28 15:53:29 -05:00 |
|
Noah Levitt
|
04c4b63f03
|
renaming scope rule "host" to "domain" to make it a less confusing, since rules apply to subdomains as well
|
2016-06-28 15:35:02 -05:00 |
|
Noah Levitt
|
320df0565e
|
support "soft limits" which result in a different response code (430) than regular (hard) limits (which result in a 420)
|
2016-06-27 16:07:20 -05:00 |
|
Noah Levitt
|
fabd732b7f
|
couple of fixes for host limits
|
2016-06-24 21:58:37 -05:00 |
|
Noah Levitt
|
2fe0c2f25b
|
support for tallying substats of a configured bucket by host, and enforcing limits host limits using those stats, with tests
|
2016-06-24 20:04:27 -05:00 |
|
Noah Levitt
|
d48e2c462d
|
add a start() method to the two classes that save data to rethinkdb periodically in batches, instead of starting the timer in __init__
|
2016-06-16 00:04:59 +00:00 |
|
Noah Levitt
|
4bb3556709
|
implement enforcement of Warcprox-Meta header block rules; includes automated tests
|
2016-05-10 23:11:47 +00:00 |
|
Noah Levitt
|
6f10e2708d
|
disable tor test to give travis build a chance to pass tests (waiting on https://github.com/travis-ci/apt-package-whitelist/issues/1753)
|
2016-04-06 19:39:28 -07:00 |
|
Noah Levitt
|
2c65ff89fa
|
add license headers
|
2016-04-06 19:37:55 -07:00 |
|
Noah Levitt
|
42a81d8f8f
|
fix bug where two warc-payload-digest headers were written to revisit records
|
2016-03-15 06:27:21 +00:00 |
|
Noah Levitt
|
df31068c80
|
improve test running script
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
4bb7e043d4
|
wait longer for stats to be updated in test_limits(), now that rethinkdb stats are pushed only every 2.0 seconds
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
c9f5b72fd7
|
really run tor in docker container for tests
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
00dc9eed84
|
new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
d8f97ad472
|
single threaded warcprox (that doesn't write warcs), useful for debugging
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
f38ce708bf
|
set PYTHONDONTWRITEBYTECODE in one place
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
18cc818cf0
|
more timing tweaks to make sure tests pass, improved logging etc
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
e67c7be5bc
|
service registry init
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
2ecd2facd9
|
surt 0.3b2 is in pypi now, no need for devpi
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
4930cc2d24
|
try to avoid conflicts with *.pyc files from outside of the docker tests
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
03c506dade
|
stop after first failing test, use py.test -s
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
95e611a5d0
|
update stats in RethinkDb asynchronously, since profiling shows this to be a bottleneck in WarcWriterThread (which in turn makes it a bottleneck for the whole app)
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
a41c426b0a
|
giving up on using git revision in version number :( latest issue is when installing a package that calls git to compute a version number, but cwd is some other git project, you get the wrong thing
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
28d213fb18
|
spin up rethinkdb in docker, run tests in there
|
2016-01-26 18:47:08 -08:00 |
|
Noah Levitt
|
4c380dcc41
|
move tests out of installed package dir
|
2016-01-26 18:47:08 -08:00 |
|