backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity
1,019 Commits 22 Branches 10 Tags
Commit Graph

334 Commits

Author SHA1 Message Date
Noah Levitt
6f6a88fc0b bump dev version number after #86 2018-05-03 12:36:16 -07:00
Noah Levitt
a1930495af default to 100 proxy threads, 1 warc writer thread 
see https://github.com/internetarchive/warcprox/wiki/benchmarking-number-of-threads
 2018-04-12 12:31:04 -07:00
Noah Levitt
ea4fc0f10a include warc writer worker threads in profiling 2018-04-11 22:35:37 +00:00
Noah Levitt
cc8fb4c608 cap the number of urls queued for warc writing 2018-04-11 22:29:50 +00:00
Noah Levitt
cb0dea3739 oops! /status has been lying about queued urls 2018-04-11 22:05:31 +00:00
Noah Levitt
ebf5453c2f bump dev version number after PR 2018-04-06 13:26:56 -07:00
Noah Levitt
cff8423bef bump dev version number after PR 2018-04-06 12:09:33 -07:00
Noah Levitt
385014c322 always call socket.shutdown() to close connections 2018-04-04 17:49:08 -07:00
Noah Levitt
ab52e81019 bump dev version number 2018-04-04 15:45:50 -07:00
Noah Levitt
e989b2f667 work around odd problem (see comment in code) 2018-04-03 11:12:25 -07:00
Noah Levitt
7f1c7f532e stop swallowing exception on _proxy_request() 2018-03-28 18:04:54 -07:00
Noah Levitt
41486f5f82 logging tweaks 2018-03-27 12:51:37 -07:00
Noah Levitt
c79b89108a bump version number after PR #72 2018-03-20 10:53:04 -07:00
Noah Levitt
9bb2018fd2 bump dev version after PR #75 2018-03-12 11:22:05 -07:00
Noah Levitt
45c06eab58 bump dev version number 2018-03-08 16:35:25 -08:00
Noah Levitt
c2172c6b5b make sure to roll over idle warcs 
even when warcprox is idle itself
 2018-02-28 13:02:03 -08:00
Noah Levitt
8a7ed0cf57 bump dev version number after merge 2018-02-28 11:45:10 -08:00
Noah Levitt
d316569196 bump dev version after revert 2018-02-27 17:28:44 -08:00
Noah Levitt
d29a367db6 bump dev version number after PR merge 2018-02-27 10:33:02 -08:00
Noah Levitt
f3e270b796 make test_method_filter() pass by waiting 
in test_limit_large_resource() for url processing to finish, to prevent
stats from affecting the subsequent test
 2018-02-20 14:54:58 -08:00
Noah Levitt
6d6f2c9aa0 fix sqlite3 string escaping 2018-02-12 11:42:35 -08:00
Noah Levitt
b2a1f15bf6 clean up test infrastructure 
- fix crufty, broken test in setup.py
- include tests in sdist tarball for pypi
 2018-02-07 16:06:46 -08:00
Noah Levitt
688e53d889 bump version number after pull request 2018-02-07 15:49:35 -08:00
Noah Levitt
e68be9354d back to dev version number 2018-02-07 15:48:42 -08:00
Noah Levitt
2ceedd3fd2 2.4b1 for pypi 2018-02-07 15:48:42 -08:00
Noah Levitt
322512dab6 bump version number after latest pull request 2018-02-07 15:48:42 -08:00
Noah Levitt
824c194142 make plugin api more flexible 2018-01-24 16:07:45 -08:00
Noah Levitt
5b414102ba respect CA-related command line options 2018-01-24 10:27:40 -08:00
Noah Levitt
1cfb4d46c6 bump version number after pull request 2018-01-22 12:50:16 -08:00
Noah Levitt
41b531e398 use trick to avoid dns looking up local ip 2018-01-21 19:47:15 -08:00
Noah Levitt
de327450ea close open warcs at shutdown 2018-01-21 19:46:31 -08:00
Noah Levitt
4b53c10132 bump minor version after these big changes 2018-01-19 14:37:53 -08:00
Noah Levitt
b43ab751f0 fix running_stats thing 2018-01-15 17:28:20 -08:00
Noah Levitt
c459812c93 roll over idle warcs on time 2018-01-12 11:46:44 -08:00
Noah Levitt
7fef2336e6 fix logging.notice/trace methods which were masking file/line/function of log message 2017-12-29 16:28:48 -08:00
Noah Levitt
f401b21958 update test_svcreg_status to expect new fields 2017-12-29 13:03:45 -08:00
Noah Levitt
5347cc92c3 change where RunningStats is initialized and fix tests 2017-12-29 11:06:46 -08:00
Noah Levitt
c966f7f6e8 more stats available from /status (and in rethindkb services table) 2017-12-28 17:07:02 -08:00
Noah Levitt
a85c665ce9 timeouts for trough requests to prevent hanging 2017-12-27 16:32:54 -08:00
Noah Levitt
eacf070a2a dropping claim of support for python 2.7 (not worth hacking around tempfile.TemporaryDirectory to make tests pass) 2017-12-21 15:45:39 -08:00
Noah Levitt
500ffad7e4 implementation of special prefix "-" which means "do not archive" 2017-12-21 14:33:30 -08:00
Noah Levitt
9784c91459 test for special warc prefix "-" which means "do not archive" 2017-12-21 14:31:54 -08:00
Noah Levitt
399853dea0 if --profile is enabled, dump results every ten minutes, as well as at shutdown 2017-12-21 11:13:37 -08:00
Noah Levitt
af6e5ea112 fix error logging in case of failure promoting trough segment 2017-12-20 12:24:28 -08:00
Noah Levitt
0e324eaecf avoid unexpected error KeyError: ... 2017-12-20 12:07:14 -08:00
Noah Levitt
6b67f49da4 back to dev version number 2017-12-15 16:44:34 -08:00
Noah Levitt
0e46dd466c 2.3 for pypi 2017-12-15 16:43:08 -08:00
Noah Levitt
995a11f444 bump dev version number after big merge 2017-11-30 16:15:55 -08:00
jkafader
e5a3dd8b3e
Merge pull request #37 from nlevitt/trough-dedup 
WIP: trough for deduplication initial proof-of-concept-ish code
 2017-11-30 16:14:43 -08:00
Noah Levitt
330635c0a8 fix test in py<=3.4 2017-11-22 13:55:44 -08:00