Ilya Kreymer
aa80cd6881
recorder: add simple recorder config indexing to redis
2016-03-21 11:50:01 -07:00
Ilya Kreymer
d38bb5a1fd
filters: add extensible 'skip filters', with default filters to accept certain collections, filter out
...
recording of range requests. Opportunity to skip recording at request or response time
RespWrapper handles reading stream fully on close() (no need for old ReadFullyStream),
skips recording if read was interrupted/incomplete
writer: avoiding writing duplicate content-length/content-type headers
2016-03-21 11:47:12 -07:00
Ilya Kreymer
cbe7d1c981
webagg: add tests for RedisPathResolver and errors on missing warc, missing warc keys
2016-03-21 11:44:32 -07:00
Ilya Kreymer
22ead52604
webagg: convert StreamIter to generate, remove unused ReadFullyStream
...
loaders: add support for RedisResolver as well as PathPrefixResolver
inputreq: reconstruct_request() skips host header if already present
improve test app to include replay
2016-03-21 11:04:52 -07:00
Ilya Kreymer
b8a1cf5081
bump version to 0.11.4
2016-03-20 23:36:42 -07:00
Ilya Kreymer
33f60efb96
tweak CHANGES
2016-03-20 23:35:58 -07:00
Ilya Kreymer
f014c32650
tweak CHANGES
2016-03-20 23:33:57 -07:00
Ilya Kreymer
dfda481b19
Merge branch '0.11.4' into develop
2016-03-20 23:19:16 -07:00
Ilya Kreymer
1a458d6b26
gevent: set env var GEVENT_MONKEY_PATCH=1 in uwsgi.ini to enable monkey-patching explicitly in pywb.apps.wayback
2016-03-20 23:18:11 -07:00
Ilya Kreymer
fff94bb526
Merge branch '0.11.4' into develop
2016-03-20 22:54:44 -07:00
Ilya Kreymer
63f44cbc26
wombat: bump version to 2.11
2016-03-20 22:54:27 -07:00
Ilya Kreymer
8302353209
wombat: fix typo in window.Crypto check!
2016-03-20 22:52:19 -07:00
Ilya Kreymer
4b8bbb7b40
wombat: fix typo in window.Crypto check!
2016-03-20 22:51:39 -07:00
Ilya Kreymer
ad66f9e11a
Merge branch '0.11.4' into develop
2016-03-20 22:44:55 -07:00
Ilya Kreymer
c570f76ae4
update changelist for 0.11.4
2016-03-20 22:43:44 -07:00
Ilya Kreymer
d2390ee809
wombat: add check for window.crypto, generate 32-bit values
2016-03-20 22:43:05 -07:00
Ilya Kreymer
4b37d13c34
tests/gevent: skip gevent monkey patch when running tests
2016-03-20 22:23:07 -07:00
Ilya Kreymer
aa0113147a
Merge branch '0.11.4' into develop
2016-03-20 18:51:23 -07:00
Ilya Kreymer
08cb02c644
wombat: override window.crypto.getRandomValues() with seeded random
2016-03-20 18:50:45 -07:00
Ilya Kreymer
8e90ea6576
gevent/uwsgi config: explicitly monkey patch in pywb.apps.wayback as early monkey patch
...
not available until uwsgi 2.1 is released!
2016-03-20 18:49:59 -07:00
Ilya Kreymer
4cf935abd1
directory agg: add CacheDirectoryAggregator to cache file listing, rescan dir only if changed
2016-03-19 20:34:09 -07:00
Ilya Kreymer
f5ee3c7bca
inputreq: add reconstruct_request() to return a bytestring of the request, add test for inputreq
2016-03-19 20:32:37 -07:00
Ilya Kreymer
c96e419341
recorder: ensure filename is also tracked by the indexer, add tests
...
for redis file mapping
2016-03-19 10:24:28 -07:00
Ilya Kreymer
3452cf39e0
recorder: use more general MultiFileWARCWriter, supporting both keeping file open
...
and one-warc-per record use cases
2016-03-18 21:40:41 -07:00
Ilya Kreymer
e81457df5f
rename WARCRecorder -> WARCWriter, add optional max_size to single warc recorder
...
per-record recorder combines http response/req into single file
2016-03-18 19:49:14 -07:00
Ilya Kreymer
b64be0dff1
recorder: add tests for single file writer, including file locking
...
dedup policy: support customizable dedup/skip/write policy plugins and add tests
2016-03-18 15:28:24 -07:00
Ilya Kreymer
cba8e4ee3a
filters: more functional filter impl for header exclusion
2016-03-17 18:22:26 -07:00
Ilya Kreymer
58e8c709aa
docker: add initial docker-compose, webagg Dockerfile
2016-03-16 18:42:15 -07:00
Ilya Kreymer
fe0f8ed1d8
Merge branch '0.11.3' into develop
2016-03-16 14:38:49 -07:00
Ilya Kreymer
cd65ddafbc
bump version to 0.11.3
2016-03-16 14:29:42 -07:00
Ilya Kreymer
f962418c1f
html rewrite typo: ensure rw_mod is set for meta content rewrite
2016-03-16 14:27:55 -07:00
Ilya Kreymer
c26660e20f
cookies: use httplib headers pair list instead of requests headers dict to avoid 'set-cookie' headers being concatenated, as that messes up parsing in 3.5.1
2016-03-16 09:47:55 -07:00
Ilya Kreymer
ef5860901f
warc record loader: if no content-length is specified on WARC record (as opposed to error or invalid), leave stream alone, don't force size of 0 and 204
2016-03-13 17:56:37 -07:00
Ilya Kreymer
8dc59ef6bd
webagg: add test for live server config
2016-03-13 16:53:39 -07:00
Ilya Kreymer
06978bd8d2
recorder: check for empty input stream (support for direct proxy?)
2016-03-13 11:17:52 -07:00
Ilya Kreymer
709d2b1ea2
reorg: move StreamIter to utils
2016-03-12 23:29:23 -08:00
Ilya Kreymer
7a828017d1
recorder: clean up logging, ReadFullyStream moves to utils, get_request_uri to inputreq
2016-03-12 22:18:01 -08:00
Ilya Kreymer
49b6ae78a8
live loader: remove liverec (doesn't work well with gevent), use regular requests
...
instead of overriden version.
reconstruct header block from httplib header pairs list
move ReadFullyStream to utils
2016-03-12 22:15:24 -08:00
Ilya Kreymer
9adb8da3b7
recorder: add support for filtering collections to record by regex (default: .*)
...
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
2016-03-11 11:12:25 -08:00
Ilya Kreymer
2003925b75
setup: fix pywb py3 version to 0.30.0, add coverage for recorder
2016-03-11 11:11:43 -08:00
Ilya Kreymer
3b3e190cf4
testing: use test mixins for class-scope temp directory, live server creation
...
use processes instead of threads for live server
2016-03-11 11:10:22 -08:00
Ilya Kreymer
2051785e6b
statusandheaders: add to_str() method with 'exclude_list' to support converting to str with certain headers
...
excluded. also supported by to_bytes()
2016-03-11 11:02:13 -08:00
Ilya Kreymer
46d013ab19
test redis: minor tweak to use @patch for fakeredis mock
2016-03-10 21:35:01 -08:00
Ilya Kreymer
c309637a3a
tests: webagg test tweaks, create TempDirTests for sharing tests that require a temp dir
2016-03-10 16:04:27 -08:00
Ilya Kreymer
7b847311d5
dir agg: include filename in dir source name
2016-03-10 15:51:01 -08:00
Ilya Kreymer
3f734e1c98
tests: remove 3.2, fix auto_index test assert
2016-03-10 13:07:57 -08:00
Ilya Kreymer
42aa12f9ae
test py3.2 also
2016-03-10 12:55:36 -08:00
Ilya Kreymer
34cc3ccacb
versions and readme: update version to 0.30.0, update README with python 2 and 3 support
2016-03-10 12:51:14 -08:00
Ilya Kreymer
0f6e3da127
cdx: tests: add tests for comparison ops
2016-03-10 12:47:36 -08:00
Ilya Kreymer
e5ca9bf601
Merge branch 'master' into py3
2016-03-10 10:53:30 -08:00
