756 Commits

Author SHA1 Message Date
Noah Levitt
7afa92f102 Merge branch 'master' into qa
* master:
  support "captures-bucket" for backward compatibility
  Add hidden CLI option --dedup-only-with-bucket
  dedup-bucket is required in Warcprox-Meta to do dedup
  Rename captures-bucket to dedup-bucket in Warcprox-Meta
  bump dev version number after #86
  Use DedupableMixin in RethinkCapturesDedup
  Fix travis-ci unit test issue
  Add unit tests
  Remove method decorate_with_dedup_info
  Use DedupableMixin in all dedup classes
  default to 100 proxy threads, 1 warc writer thread
  include warc writer worker threads in profiling
  cap the number of urls queued for warc writing
  oops! /status has been lying about queued urls
  Configurable min dedupable size for text/binary resources
  bump dev version number after PR
  Fix Accept-Encoding request header
  CDX dedup improvements
  bump dev version number after PR
  make test server multithreaded so tests will pass
  always call socket.shutdown() to close connections
  bump dev version number
  close connection when truncating response
  test another request after truncated response
  close all remote connections at shutdown
  tweak tests to make them pass now that keepalive
  enable keepalive on test http server
  more logging
  remove some debug logging
  this is some logging meant to debug the mysterious
  work around odd problem (see comment in code)
2018-05-09 15:43:52 -07:00
Noah Levitt
15830fc5a2 support "captures-bucket" for backward compatibility 2018-05-09 15:43:39 -07:00
Noah Levitt
5fa1f8f61c
Merge pull request #90 from vbanos/dedup-bucket
Require dedup-bucket in Warcprox-Meta to perform dedup
2018-05-08 11:06:32 -07:00
Vangelis Banos
abb54e42d1 Add hidden CLI option --dedup-only-with-bucket
When we use `--dedup-only-with-bucket`, dedup will be done only when a
request has key `dedup-bucket` in `Warcprox-Meta`.
2018-05-04 20:50:54 +00:00
Vangelis Banos
432e42803c dedup-bucket is required in Warcprox-Meta to do dedup
Modify `DedupableMixin.should_dedup` to check Warcprox-Meta for
`dedup-bucket` in order to perform dedup.
2018-05-04 14:27:42 +00:00
Vangelis Banos
9baa2e22d5 Rename captures-bucket to dedup-bucket in Warcprox-Meta 2018-05-04 13:26:38 +00:00
Noah Levitt
6f6a88fc0b bump dev version number after #86 2018-05-03 12:36:16 -07:00
Noah Levitt
f76b43f2a3
Merge pull request #86 from vbanos/configurable-dedup-size-limits
Configurable min dedupable size for text/binary resources
2018-05-03 12:35:43 -07:00
Vangelis Banos
255d359ad4 Use DedupableMixin in RethinkCapturesDedup
I note that we didn't do any payload_size check at all here.
2018-04-24 17:06:56 +00:00
Vangelis Banos
9dac806ca1 Fix travis-ci unit test issue
`test_dedup_https` fails on travis-ci.
https://travis-ci.org/internetarchive/warcprox/jobs/370598950

We didn't touch that at all but worked on `test_dedup_min_size` which
runs just before that. We move `test_dedup_min_size` to the end of the
file hoping to resolve this.
2018-04-24 16:31:37 +00:00
Vangelis Banos
944c9a1e11 Add unit tests
Create two very small dummy responses (text, 2 bytes and binary, 4 bytes).
Use options --dedup-min-text-size=3 and --dedup-min-binary-size=5.
Ensure that due to the effects of these options, dedup is not happening.

Existing dedup unit tests are not affected at all.
2018-04-24 12:18:20 +00:00
Vangelis Banos
6dce8cc644 Remove method decorate_with_dedup_info
Method `warcprox.dedup.decorate_with_dedup_info` is only used in
`DedupLoader._process_url` and nowhere else.

The problem is that `decorate_with_dedup_info` cannot get warcprox cli
options. Thus we cannot pass the custom min size limits.
2018-04-24 10:58:13 +00:00
Vangelis Banos
9057fbdf36 Use DedupableMixin in all dedup classes
Rename `DedupableMixin.is_dedupable` to `should_dedup`.
2018-04-24 10:29:35 +00:00
Noah Levitt
a1930495af default to 100 proxy threads, 1 warc writer thread
see https://github.com/internetarchive/warcprox/wiki/benchmarking-number-of-threads
2018-04-12 12:31:04 -07:00
Noah Levitt
ea4fc0f10a include warc writer worker threads in profiling 2018-04-11 22:35:37 +00:00
Noah Levitt
cc8fb4c608 cap the number of urls queued for warc writing 2018-04-11 22:29:50 +00:00
Noah Levitt
cb0dea3739 oops! /status has been lying about queued urls 2018-04-11 22:05:31 +00:00
Vangelis Banos
d32bf743bd Configurable min dedupable size for text/binary resources
New `--dedup-min-text-size` and `--dedup-min-binary-size` cli options
with default value = `0`.

New `DedupableMixin` which can be used in any dedup class. It is
currently used only in CDX dedup. Instead of checking `payload_size() >
0`, we now use `.is_dedupable(recorded_url)`

New utility method `RecordedUrl.is_text`.
2018-04-09 15:52:44 +00:00
Noah Levitt
ebf5453c2f bump dev version number after PR 2018-04-06 13:26:56 -07:00
Noah Levitt
797e33b91d
Merge pull request #81 from vbanos/cdxdedup-improvements2
CDX dedup improvements
2018-04-06 13:26:28 -07:00
Vangelis Banos
cce0c705fb Fix Accept-Encoding request header 2018-04-06 19:55:19 +00:00
Vangelis Banos
7c5c5da9b7 CDX dedup improvements
Check for not empty captured content (`payload_size() > 0`) before
creating a new thread and running a CDX dedup request.
Most dedup modules perform the same check to avoid unnecessary dedup
requests.

Increase CDX dedup max workers from 200 to 400 in order to handle more
load.

Set `user-agent: warcprox` for HTTP requests we send to CDX server. Its
useful to identify and monitor `warcprox` requests.

Pass HTTP headers to connection pool on init and not on each request.
2018-04-06 19:55:19 +00:00
Noah Levitt
cff8423bef bump dev version number after PR 2018-04-06 12:09:33 -07:00
Noah Levitt
ac3e7a433d
Merge pull request #84 from nlevitt/multithread-test-server
make test server multithreaded so tests will pass

Merging this one (multithreaded server, multithreaded proxy) rather than #85 (single-threaded server, single-threaded proxy). The single-threaded option is nice because sometimes it reveals bugs that rarely or never come up when everything is multithreaded, and it's easier to reason about the behavior of the system and debug problems. Nevertheless, I'm choosing this one because it's more similar to a realistic workload. (Maybe we should do both? But the tests already take a long time to run...)
2018-04-06 10:16:50 -07:00
Noah Levitt
38e2a87f31 make test server multithreaded so tests will pass 2018-04-05 17:59:10 -07:00
Noah Levitt
385014c322 always call socket.shutdown() to close connections 2018-04-04 17:49:08 -07:00
Noah Levitt
ab52e81019 bump dev version number 2018-04-04 15:45:50 -07:00
Noah Levitt
7ef0612fa6 close connection when truncating response 2018-04-04 15:45:32 -07:00
Noah Levitt
595e819961 test another request after truncated response
to check for hangs or timeouts
2018-04-04 15:45:13 -07:00
Noah Levitt
7c814d71ba close all remote connections at shutdown
to avoid hang
2018-04-04 15:42:45 -07:00
Noah Levitt
3f9ecbacac tweak tests to make them pass now that keepalive
is enabled on the test server
2018-04-04 15:41:54 -07:00
Noah Levitt
8ac0420cb2 enable keepalive on test http server
As of fairly recently, warcprox does keepalive with the remote server
using the urllib3 connection pool. The test http server in
test_warcprox.py was acting as if it supported keepalive (sending
HTTP/1.1 and not sending "Connection: close"). But in fact it did not
support keepalive. It was closing the connection after each request.
Depending on the timing of what was happening in different threads,
sometimes the client thread would try to send another request on a
connection it still believed to be open for keepalive. Then the server
side would complete its request processing and close the connection.
This resulted in test failures with error messages like this (depending
on python version):
2018-04-03 21:20:06,555 12586 ERROR MainThread warcprox.mitmproxy.MitmProxyHandler.do_COMMAND(mitmproxy.py:389) error from remote server(?) None: BadStatusLine("''",)
2018-04-04 19:06:29,599 11632 ERROR MainThread warcprox.mitmproxy.MitmProxyHandler.do_COMMAND(mitmproxy.py:389) error from remote server(?) None: RemoteDisconnected('Remote end closed connection without response',)
For instance https://travis-ci.org/internetarchive/warcprox/jobs/362288603
2018-04-04 15:38:48 -07:00
Noah Levitt
2fa0f232b7 more logging 2018-04-04 15:36:46 -07:00
Noah Levitt
c2b2a844d9 remove some debug logging 2018-04-04 10:22:02 -07:00
Noah Levitt
08aada3ca9 this is some logging meant to debug the mysterious
test failure we've been seeing
which so far has made the problem go away(!?!?)
😀😞 ¯\_(ツ)_/¯ 😞😀 ¯\_(ツ)_/¯ 😀😞 ¯\_(ツ)_/¯ 😞😀
here is the last time the failure happened:
https://travis-ci.org/internetarchive/warcprox/jobs/361409280
2018-04-03 11:15:48 -07:00
Noah Levitt
e989b2f667 work around odd problem (see comment in code) 2018-04-03 11:12:25 -07:00
Noah Levitt
623fb24bee Merge branch 'master' into qa
* master:
  stop swallowing exception on _proxy_request()
  logging tweaks
2018-03-28 18:05:13 -07:00
Noah Levitt
7f1c7f532e stop swallowing exception on _proxy_request() 2018-03-28 18:04:54 -07:00
Noah Levitt
41486f5f82 logging tweaks 2018-03-27 12:51:37 -07:00
Noah Levitt
1ee901c6fa Merge branch 'master' into qa
* master:
  bump version number after PR #72
  Fix SOCKS connection error
  Improve Connection Pool
  Reduce the PoolManager num_pools size and fix bugs
  bump dev version after PR #75
  bump dev version number
  Fix ListenerPostfetchProcessor typo
  Configurable tmp file max memory size
  Address unit test failure in Python 3.4
  a minimal example
  Extra connection evaluation before putting it back to the pool
  Fix typo
  Remove whitespace
  Remote server connection pool
2018-03-20 10:53:21 -07:00
Noah Levitt
c79b89108a bump version number after PR #72 2018-03-20 10:53:04 -07:00
Noah Levitt
3ece9cbe6f
Merge pull request #72 from vbanos/remote-conn-pool
Remote server connection pooling
2018-03-20 10:52:14 -07:00
Vangelis Banos
0404ad239f Fix SOCKS connection error 2018-03-20 07:35:49 +00:00
Vangelis Banos
0002d29f0d Improve Connection Pool
Set connection pool maxsize to 6 (borrowing from browser behavior).

Set num_pools to `max_threads / 6` but set a minimum of 200 for the cases
that we use a very low number of `max_threads`.

Remove `connection_is_fine` variable from connection code.

Fix http headers bug introduced in the previous commit.
2018-03-16 21:06:34 +00:00
Vangelis Banos
1d5692dd13 Reduce the PoolManager num_pools size and fix bugs
Define PoolManager num_pools size as `max(max_threads, 500)` and reduce
each pool size from 100 to 30. The aim is to limit the total number of
open connections.

Fix remote SOCKS connection typo.

Now that we reuse remote connections, its better NOT to remove the
`keep-alive` request header. We need to send it to the remote host to make it
keep the connection open if possible.
2018-03-16 13:10:29 +00:00
Noah Levitt
9bb2018fd2 bump dev version after PR #75 2018-03-12 11:22:05 -07:00
Noah Levitt
4d578c5541
Merge pull request #75 from vbanos/tmp-file-max-memory-size
Configurable SpooledTemporaryFile max memory size
2018-03-12 11:21:19 -07:00
Noah Levitt
45c06eab58 bump dev version number 2018-03-08 16:35:25 -08:00
Noah Levitt
6c2fbfab78
Merge pull request #76 from vbanos/ListenerPostfetchProcessor-typo
Fix ListenerPostfetchProcessor typo
2018-03-08 16:34:08 -08:00
Vangelis Banos
2f84fa8dbf Fix ListenerPostfetchProcessor typo
Use `self.listener` instead of `listener`.
2018-03-08 08:01:54 +00:00