Noah Levitt 6b6c0b3bac make sure batch insert timer thread survives rethinkdb outages

2016-03-18 02:06:07 +00:00

..

__init__.py

tid="n/a" if not available

2016-01-26 18:47:08 -08:00

bigtable.py

make sure batch insert timer thread survives rethinkdb outages

2016-03-18 02:06:07 +00:00

controller.py

hidden --profile option to enable profiling of warc writer thread and periodic logging of memory usage info; at shutdown, close stats db and unregister from service registry; logging improvements

2016-01-26 18:47:08 -08:00

dedup.py

import dbm only if used

2016-01-27 21:18:02 +00:00

kafkafeed.py

use kafka-python 1.0 recommended api; use kafka capture feed specified in warcprox-meta header, if any

2016-03-03 18:58:52 +00:00

main.py

heuristic to set size of thread pool based on open files limit, to hopefully fix problem where warcprox got stuck because it ran out of file handles

2016-03-04 20:59:11 +00:00

mitmproxy.py

new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites

2016-01-26 18:47:08 -08:00

playback.py

import dbm only if used

2016-01-27 21:18:02 +00:00

stats.py

import dbm only if used

2016-01-27 21:18:02 +00:00

warc.py

fix bug where two warc-payload-digest headers were written to revisit records

2016-03-15 06:27:21 +00:00

warcproxy.py

fix this error

2016-03-04 21:02:47 +00:00

writer.py

add length field to each record in big captures table (size in bytes of compressed warc record) because pywayback needs it

2016-01-26 18:47:08 -08:00

writerthread.py

better handle exceptions from listeners

2016-03-03 18:59:13 +00:00