backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity
935 Commits 22 Branches 10 Tags
Commit Graph

301 Commits

Author SHA1 Message Date
Barbara Miller
ae11daedc1
bump version 2020-08-18 09:29:57 -07:00
Christian Clauss
c649355285
setup.py: Add Python 3.8 2020-08-06 17:58:00 +02:00
Noah Levitt
de9219e646 require more recent urllib3 
to avoid this error: https://github.com/internetarchive/warcprox/issues/148

2020-01-28 14:42:44,851 2023 ERROR MitmProxyHandler(tid=2037,started=2020-01-28T20:42:44.834551,client=127.0.0.1:49100) warcprox.warcprox.WarcProxyHandler.do_COMMAND(mitmproxy.py:442) problem processing request 'GET / HTTP/1.1': TypeError("connection_from_host() got an unexpected keyword argument 'pool_kwargs'",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/warcprox/mitmproxy.py", line 413, in do_COMMAND
    self._connect_to_remote_server()
  File "/usr/local/lib/python3.5/dist-packages/warcprox/warcproxy.py", line 189, in _connect_to_remote_server
    return warcprox.mitmproxy.MitmProxyHandler._connect_to_remote_server(self)
  File "/usr/local/lib/python3.5/dist-packages/warcprox/mitmproxy.py", line 277, in _connect_to_remote_server
    pool_kwargs={'maxsize': 12, 'timeout': self._socket_timeout})
TypeError: connection_from_host() got an unexpected keyword argument 'pool_kwargs'
 2020-02-06 10:10:53 -08:00
Noah Levitt
90fba01514 make trough dependency optional 2020-01-08 13:37:01 -08:00
Noah Levitt
a8cd53bfe4 bump version, trough dep version 2020-01-08 13:24:00 -08:00
Noah Levitt
469b41773a fix logging config which trough interfered with 2020-01-07 15:19:03 -08:00
Noah Levitt
91fcc054c4 bump version after merge 2020-01-07 14:42:40 -08:00
Noah Levitt
f54e1b37c7 bump version after merge 2020-01-07 14:40:58 -08:00
Noah Levitt
fe19bb268f use trough.client instead of warcprox.trough 
less redundant code!
trough.client was based off of warcprox.trough but has been improved
since then
 2019-11-19 11:45:14 -08:00
Noah Levitt
f77c152037 bump version after merge 2019-09-26 11:49:07 -07:00
Noah Levitt
1f852f5f36 bump version after merges 2019-09-23 11:55:00 -07:00
Noah Levitt
88a7f79a7e bump version 2019-09-13 10:58:16 -07:00
Noah Levitt
1aa6b0c5d6 log remote host/ip/port on SSLError 2019-08-16 18:31:35 +00:00
Noah Levitt
fce1c3d722 requests/urllib3 version conflict from april must 
be obsolete by now...
 2019-07-26 14:03:36 -07:00
Noah Levitt
932001c921 bump version after merge 2019-06-20 14:57:36 -07:00
Noah Levitt
81a945e840 bump version after a few small PRs 2019-06-11 10:58:52 -07:00
Noah Levitt
8c31ec2916 bigger connection pool, for Vangelis 2019-05-15 16:06:42 -07:00
Noah Levitt
bbf3fad1dc avoid using warcproxy.py stuff in mitmproxy.py 2019-05-15 15:58:47 -07:00
Noah Levitt
f51f2ec225 some tweaks to error responses 
use 502, 504 when appropriate, and don't send `str(e)` as in the http
status line, because that is often an ugly jumble
 2019-05-14 15:51:11 -07:00
Noah Levitt
2772b80fab bump version after merge 2019-05-14 15:50:59 -07:00
Vangelis Banos
89d987a181 Cache bad target hostname:port to avoid reconnection attempts 
If connection to a hostname:port fails, add it to a `TTLCache` with
60 sec expiration time. Subsequent requests to the same hostname:port
return really quickly as we check the cache and avoid trying a new
network connection.

The short expiration time guarantees that if a host becomes OK again,
we'll be able to connect to it quickly.

Adding `cachetools` dependency was necessary as there isn't any other
way to have an expiring in-memory cache using stdlib. The library
doesn't have any other dependencies, it has good test coverage and seems
maintained. It also supports Python 3.7.
 2019-05-09 10:03:16 +00:00
Noah Levitt
41d7f0be53 bump version after merges 2019-05-06 16:49:35 -07:00
Noah Levitt
dfc081fff8 do not write incorrect warc-payload-digest to... 
... request records

see https://github.com/webrecorder/warcio/issues/74#issuecomment-487816378
 2019-05-02 14:25:29 -07:00
Noah Levitt
38d6e4337d handle graceful shutdown failure 
print stack trace and kill myself -9
 2019-04-24 13:14:12 -07:00
Noah Levitt
de01d498cb requests/urllib3 version conflict 2019-04-24 12:11:20 -07:00
Noah Levitt
3298128e0c deal with bad content-type header 
we had bad stuff get into a crawl log because of a url that returned a
Content-Type header value with spaces in it (but no semicolon)
 2019-04-24 10:40:22 -07:00
Noah Levitt
f207e32f50 followup on IncompleteRead 2019-04-15 00:17:50 -07:00
Noah Levitt
5de2569430 bump version after #124 merge 2019-04-13 18:11:02 -07:00
Noah Levitt
98b3c1f80b bump version after merge 2019-04-09 21:52:31 +00:00
Noah Levitt
2ca84ae023 bump version after merge 2019-04-08 11:50:27 -07:00
Noah Levitt
794cc29c80 bump version 2019-03-21 16:04:05 -07:00
Noah Levitt
cb2a07bff2 account for surt fix in urlcanon 0.3.0 2019-03-21 12:59:32 -07:00
Noah Levitt
1e0a0ca63a every change is a point release now 2019-03-21 12:38:29 -07:00
Vangelis Banos
436a27b19e Upgrade PyYAML to >=5.1 2019-03-21 19:34:52 +00:00
Vangelis Banos
878ab0977f Use YAML instead of JSON 
Add PyYAML<=3.13 dependency.
 2019-03-21 19:18:55 +00:00
Noah Levitt
c70bf2e2b9 debugging a shutdown issue 2019-02-27 12:36:35 -08:00
Noah Levitt
adca46427d back to dev version number 2019-02-12 15:04:22 -08:00
Noah Levitt
5a7a4ff710 pypi release 2019-02-12 15:00:22 -08:00
Noah Levitt
cb72af015a fix idle rollover 2019-01-21 10:37:09 -08:00
Noah Levitt
a780f1774c back to dev version number 2019-01-17 17:15:33 -08:00
Noah Levitt
e07ee3630e 2.4b3 for pypi 2019-01-09 15:15:37 -08:00
Noah Levitt
3ea5c36e7f add idna as dep with acceptable to other deps 
because
<kenji> my understanding is that pip cannot fully resolve version
constraints for indirect dependencies.
 2019-01-09 15:10:37 -08:00
Noah Levitt
1ea8a06a69 3 hour hard timeout on urls without content-length 
so that indefinite streams like icecast radio stations don't hang
forever
 2018-11-12 15:57:37 -08:00
Noah Levitt
bb50a6c7ff use predictable id in service registry 
so that when warcprox restarts it replaces the obsolete entry
 2018-11-12 15:11:23 -08:00
Noah Levitt
4f836e9179 bump version number 2018-11-06 11:29:33 -08:00
Noah Levitt
1460040789 bump version after merge 2018-10-31 16:23:00 -07:00
Noah Levitt
52f2ac0f4e send nice 503s and avoid scary stack traces... 
... at shutdown
 2018-10-26 15:26:27 -07:00
Noah Levitt
89212e782d fix failing test 2018-10-26 13:44:27 -07:00
Noah Levitt
e993b0c28c fix shutdown 
at shutdown, abort active connections, but allow completed fetches to
finish processing

this should fix race condition issue at shutdown, where postfetch
processor B would shut down, then postfetch processor A would try to
enqueue more urls, filling up the queue to the point where it blocks
forever, since B is no longer pulling urls off the queue
 2018-10-26 13:21:15 -07:00
Noah Levitt
4f01772782 enforce limits on WARCPROX_WRITE_RECORD requests 
should make test from previous commit pass
 2018-10-10 18:24:54 -07:00