backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity
296 Commits 22 Branches 10 Tags
Commit Graph

296 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt
4fd17be339 started adding some docstrings, and moved some of the more generally man-in-the-middle recording proxy code from warcproxy.py into mitmproxy.py 2016-05-10 01:11:17 -07:00
Noah Levitt
0809c78486 add Strict-Transport-Security to list of http response headers to swallow, to avoid some problems with HSTS when browsing through warcprox (doesn't solve the case of preloaded HSTS though) 2016-04-08 23:26:20 -07:00
Noah Levitt
6f10e2708d disable tor test to give travis build a chance to pass tests (waiting on https://github.com/travis-ci/apt-package-whitelist/issues/1753) 2016-04-06 19:39:28 -07:00
Noah Levitt
2c65ff89fa add license headers 2016-04-06 19:37:55 -07:00
Noah Levitt
ad661dcead expand license info, update usage section, remove crufty todo section 2016-04-06 19:37:16 -07:00
Noah Levitt
6490583dd0 this brozzler branch will be warcprox 2.0, today it's 2.0.dev4 2016-03-18 02:07:29 +00:00
Noah Levitt
6b6c0b3bac make sure batch insert timer thread survives rethinkdb outages 2016-03-18 02:06:07 +00:00
Noah Levitt
42a81d8f8f fix bug where two warc-payload-digest headers were written to revisit records 2016-03-15 06:27:21 +00:00
Noah Levitt
2c91eb03d3 support new Warcprox-Meta json field captures-table-extra-fields, extra fields to include in the rethinkdb captures table entry 2016-03-13 07:46:33 +00:00
Noah Levitt
910cd062ee bump version number 2016-03-08 22:55:42 +00:00
Noah Levitt
2bec9db7df handle old dedup entries missing "warc_id" 2016-03-08 22:52:02 +00:00
Noah Levitt
422672408a fix this error 
File "/home/nlevitt/workspace/warcprox/warcprox/warcproxy.py", line 256, in _proxy_request
    return recorded_url
UnboundLocalError: local variable 'recorded_url' referenced before assignment
 2016-03-04 21:02:47 +00:00
Noah Levitt
918fdd3e9b heuristic to set size of thread pool based on open files limit, to hopefully fix problem where warcprox got stuck because it ran out of file handles 2016-03-04 20:59:11 +00:00
Noah Levitt
46887f7594 better handle exceptions from listeners 2016-03-03 18:59:13 +00:00
Noah Levitt
89f965d1d3 use kafka-python 1.0 recommended api; use kafka capture feed specified in warcprox-meta header, if any 2016-03-03 18:58:52 +00:00
Noah Levitt
ee3ee5d621 call this 1.5.0.dev1 for now 2016-02-25 01:36:36 +00:00
Noah Levitt
1e0a3f0135 import dbm only if used 2016-01-27 21:18:02 +00:00
Noah Levitt
2cb1454302 s/abbr_canon_surt_timesamp/abbr_canon_surt_timestamp/ 2016-01-26 18:47:08 -08:00
Noah Levitt
927419645b use rethinkdb native time type for captures table timestamp 2016-01-26 18:47:08 -08:00
Noah Levitt
df31068c80 improve test running script 2016-01-26 18:47:08 -08:00
Noah Levitt
4bb7e043d4 wait longer for stats to be updated in test_limits(), now that rethinkdb stats are pushed only every 2.0 seconds 2016-01-26 18:47:08 -08:00
Noah Levitt
c9f5b72fd7 really run tor in docker container for tests 2016-01-26 18:47:08 -08:00
Noah Levitt
00dc9eed84 new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites 2016-01-26 18:47:08 -08:00
Noah Levitt
fb58244c4f update stats in rethinkdb only every 2.0 seconds instead of every 0.5 2016-01-26 18:47:08 -08:00
Noah Levitt
734b2f5396 limit max number of threads to 500; make sure connection with proxy client has a timeout; log errors from connection with proxy client 2016-01-26 18:47:08 -08:00
Noah Levitt
d8f97ad472 single threaded warcprox (that doesn't write warcs), useful for debugging 2016-01-26 18:47:08 -08:00
Noah Levitt
fe4d7a2769 tid="n/a" if not available 2016-01-26 18:47:08 -08:00
Noah Levitt
e3a5717446 hidden --profile option to enable profiling of warc writer thread and periodic logging of memory usage info; at shutdown, close stats db and unregister from service registry; logging improvements 2016-01-26 18:47:08 -08:00
Noah Levitt
7eb82ab8a2 adding missing import, remove unused method, logging tweaks, avoid exception at shutdown joining unstarted timer thread 2016-01-26 18:47:08 -08:00
Noah Levitt
f38ce708bf set PYTHONDONTWRITEBYTECODE in one place 2016-01-26 18:47:08 -08:00
Noah Levitt
18cc818cf0 more timing tweaks to make sure tests pass, improved logging etc 2016-01-26 18:47:08 -08:00
Noah Levitt
fcaaa7b09b include tid in thread name for more threads (linux only) for correlation with top -H 2016-01-26 18:47:08 -08:00
Noah Levitt
a9fc550453 oops, argparse.SUPPRESS isn't supposed to be in quotes 2016-01-26 18:47:08 -08:00
Noah Levitt
95ef8b80b0 make sure load score for service registry is a float; comment out memory debugging call; close dedup db after warc writer thread finishes 2016-01-26 18:47:08 -08:00
Noah Levitt
9af17ba7c3 update stats batch every 0.5 seconds, since rethinkdb updates were falling way behind sometimes 2016-01-26 18:47:08 -08:00
Noah Levitt
783e730e52 insert captures entries in batch every 0.5 seconds, since rethinkdb updates were falling way behind sometimes 2016-01-26 18:47:08 -08:00
Noah Levitt
afdb6cf557 log status in close() 2016-01-26 18:47:08 -08:00
Noah Levitt
93a2e4ff85 .travis.yml - disable pypy (not working because of cryptography library), require docker service 2016-01-26 18:47:08 -08:00
Noah Levitt
9f84c20274 test with rethinkdb flags too 2016-01-26 18:47:08 -08:00
Noah Levitt
3e2696525b make sure svcreg is set 2016-01-26 18:47:08 -08:00
Noah Levitt
248d110f81 add port to service registry, fix bug with service hearbeat 2016-01-26 18:47:08 -08:00
Noah Levitt
e67c7be5bc service registry init 2016-01-26 18:47:08 -08:00
Noah Levitt
2ecd2facd9 surt 0.3b2 is in pypi now, no need for devpi 2016-01-26 18:47:08 -08:00
Noah Levitt
4dcaedb5d9 py.test the right thing 2016-01-26 18:47:08 -08:00
Noah Levitt
d7d992731c register self for service discovery 2016-01-26 18:47:08 -08:00
Noah Levitt
7e731d40bc try new travis docker-based infrastructure, more versions of python 2016-01-26 18:47:08 -08:00
Noah Levitt
465cf1ef45 ./tests/run-tests.sh is better than tox 2016-01-26 18:47:08 -08:00
Noah Levitt
ca4c62fc6d don't load dedup info for empty payload 2016-01-26 18:47:08 -08:00
Noah Levitt
3363b2ec95 continue after unexpected error 2016-01-26 18:47:08 -08:00
Noah Levitt
fd847f01cd log error but don't give up if there is >1 record with same digest 2016-01-26 18:47:08 -08:00