mirror of
https://github.com/internetarchive/warcprox.git
synced 2025-01-18 13:22:09 +01:00
7afa92f102
* master: support "captures-bucket" for backward compatibility Add hidden CLI option --dedup-only-with-bucket dedup-bucket is required in Warcprox-Meta to do dedup Rename captures-bucket to dedup-bucket in Warcprox-Meta bump dev version number after #86 Use DedupableMixin in RethinkCapturesDedup Fix travis-ci unit test issue Add unit tests Remove method decorate_with_dedup_info Use DedupableMixin in all dedup classes default to 100 proxy threads, 1 warc writer thread include warc writer worker threads in profiling cap the number of urls queued for warc writing oops! /status has been lying about queued urls Configurable min dedupable size for text/binary resources bump dev version number after PR Fix Accept-Encoding request header CDX dedup improvements bump dev version number after PR make test server multithreaded so tests will pass always call socket.shutdown() to close connections bump dev version number close connection when truncating response test another request after truncated response close all remote connections at shutdown tweak tests to make them pass now that keepalive enable keepalive on test http server more logging remove some debug logging this is some logging meant to debug the mysterious work around odd problem (see comment in code)