backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity
935 Commits 22 Branches 10 Tags
Commit Graph

184 Commits

Author SHA1 Message Date
Noah Levitt
e5f2c348e2 fix dockerized automated tests now that phusion/baseimage is ubuntu xenial 2016-11-15 12:09:09 -08:00
Noah Levitt
3b167459e3 change tested idns to valid idna2008 now that requests 2.12.0 enforces that (for better or worse, see https://github.com/kennethreitz/requests/issues/3687) 2016-11-15 12:08:07 -08:00
Noah Levitt
fa1e8d3af4 allow travis-ci failures for python-nightly and also test 3.6-dev (but allow failures); 
enable the onion site tor test because apparently travis-ci is allowing me to
install tor now, see https://travis-ci.org/internetarchive/warcprox/jobs/169101744
although https://github.com/travis-ci/apt-package-whitelist/issues/1753 is still open
 2016-10-19 18:24:25 -07:00
Noah Levitt
719380e612 refactor some general mitm proxy stuff into mitmproxy.py 2016-10-19 15:32:58 -07:00
Noah Levitt
314be33707 new test that reveals connection hang on https urls missing a content-length http response header (not chunked and server leaves connection open) -- reported by Alex Osborne 2016-10-19 13:43:44 -07:00
Noah Levitt
46c24833ff emoji idn fails with python 2.7, so test with a BMP unicode character 2016-06-29 17:16:50 -05:00
Noah Levitt
33775d360a comment out segfaulting test 2016-06-29 16:47:54 -05:00
Noah Levitt
a59871e17b idn support, at least for domain limits (getting a segfault in tests on mac however, let's see what happens on travis-ci) 2016-06-29 15:54:40 -05:00
Noah Levitt
c9e403585b switching from host limits to domain limits, which apply in aggregate to the host and subdomains 2016-06-29 14:56:14 -05:00
Noah Levitt
2c8b194090 really only apply host limits to the host 2016-06-28 15:53:29 -05:00
Noah Levitt
04c4b63f03 renaming scope rule "host" to "domain" to make it a less confusing, since rules apply to subdomains as well 2016-06-28 15:35:02 -05:00
Noah Levitt
320df0565e support "soft limits" which result in a different response code (430) than regular (hard) limits (which result in a 420) 2016-06-27 16:07:20 -05:00
Noah Levitt
fabd732b7f couple of fixes for host limits 2016-06-24 21:58:37 -05:00
Noah Levitt
2fe0c2f25b support for tallying substats of a configured bucket by host, and enforcing limits host limits using those stats, with tests 2016-06-24 20:04:27 -05:00
Noah Levitt
d48e2c462d add a start() method to the two classes that save data to rethinkdb periodically in batches, instead of starting the timer in __init__ 2016-06-16 00:04:59 +00:00
Noah Levitt
4bb3556709 implement enforcement of Warcprox-Meta header block rules; includes automated tests 2016-05-10 23:11:47 +00:00
Noah Levitt
6f10e2708d disable tor test to give travis build a chance to pass tests (waiting on https://github.com/travis-ci/apt-package-whitelist/issues/1753) 2016-04-06 19:39:28 -07:00
Noah Levitt
2c65ff89fa add license headers 2016-04-06 19:37:55 -07:00
Noah Levitt
42a81d8f8f fix bug where two warc-payload-digest headers were written to revisit records 2016-03-15 06:27:21 +00:00
Noah Levitt
df31068c80 improve test running script 2016-01-26 18:47:08 -08:00
Noah Levitt
4bb7e043d4 wait longer for stats to be updated in test_limits(), now that rethinkdb stats are pushed only every 2.0 seconds 2016-01-26 18:47:08 -08:00
Noah Levitt
c9f5b72fd7 really run tor in docker container for tests 2016-01-26 18:47:08 -08:00
Noah Levitt
00dc9eed84 new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites 2016-01-26 18:47:08 -08:00
Noah Levitt
d8f97ad472 single threaded warcprox (that doesn't write warcs), useful for debugging 2016-01-26 18:47:08 -08:00
Noah Levitt
f38ce708bf set PYTHONDONTWRITEBYTECODE in one place 2016-01-26 18:47:08 -08:00
Noah Levitt
18cc818cf0 more timing tweaks to make sure tests pass, improved logging etc 2016-01-26 18:47:08 -08:00
Noah Levitt
e67c7be5bc service registry init 2016-01-26 18:47:08 -08:00
Noah Levitt
2ecd2facd9 surt 0.3b2 is in pypi now, no need for devpi 2016-01-26 18:47:08 -08:00
Noah Levitt
4930cc2d24 try to avoid conflicts with *.pyc files from outside of the docker tests 2016-01-26 18:47:08 -08:00
Noah Levitt
03c506dade stop after first failing test, use py.test -s 2016-01-26 18:47:08 -08:00
Noah Levitt
95e611a5d0 update stats in RethinkDb asynchronously, since profiling shows this to be a bottleneck in WarcWriterThread (which in turn makes it a bottleneck for the whole app) 2016-01-26 18:47:08 -08:00
Noah Levitt
a41c426b0a giving up on using git revision in version number :( latest issue is when installing a package that calls git to compute a version number, but cwd is some other git project, you get the wrong thing 2016-01-26 18:47:08 -08:00
Noah Levitt
28d213fb18 spin up rethinkdb in docker, run tests in there 2016-01-26 18:47:08 -08:00
Noah Levitt
4c380dcc41 move tests out of installed package dir 2016-01-26 18:47:08 -08:00