1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

1656 Commits

Author SHA1 Message Date
Ilya Kreymer
b8a1cf5081 bump version to 0.11.4 2016-03-20 23:36:42 -07:00
Ilya Kreymer
33f60efb96 tweak CHANGES 2016-03-20 23:35:58 -07:00
Ilya Kreymer
f014c32650 tweak CHANGES 2016-03-20 23:33:57 -07:00
Ilya Kreymer
dfda481b19 Merge branch '0.11.4' into develop 2016-03-20 23:19:16 -07:00
Ilya Kreymer
1a458d6b26 gevent: set env var GEVENT_MONKEY_PATCH=1 in uwsgi.ini to enable monkey-patching explicitly in pywb.apps.wayback 2016-03-20 23:18:11 -07:00
Ilya Kreymer
fff94bb526 Merge branch '0.11.4' into develop 2016-03-20 22:54:44 -07:00
Ilya Kreymer
63f44cbc26 wombat: bump version to 2.11 2016-03-20 22:54:27 -07:00
Ilya Kreymer
8302353209 wombat: fix typo in window.Crypto check! 2016-03-20 22:52:19 -07:00
Ilya Kreymer
4b8bbb7b40 wombat: fix typo in window.Crypto check! 2016-03-20 22:51:39 -07:00
Ilya Kreymer
ad66f9e11a Merge branch '0.11.4' into develop 2016-03-20 22:44:55 -07:00
Ilya Kreymer
c570f76ae4 update changelist for 0.11.4 2016-03-20 22:43:44 -07:00
Ilya Kreymer
d2390ee809 wombat: add check for window.crypto, generate 32-bit values 2016-03-20 22:43:05 -07:00
Ilya Kreymer
4b37d13c34 tests/gevent: skip gevent monkey patch when running tests 2016-03-20 22:23:07 -07:00
Ilya Kreymer
aa0113147a Merge branch '0.11.4' into develop 2016-03-20 18:51:23 -07:00
Ilya Kreymer
08cb02c644 wombat: override window.crypto.getRandomValues() with seeded random 2016-03-20 18:50:45 -07:00
Ilya Kreymer
8e90ea6576 gevent/uwsgi config: explicitly monkey patch in pywb.apps.wayback as early monkey patch
not available until uwsgi 2.1 is released!
2016-03-20 18:49:59 -07:00
Ilya Kreymer
4cf935abd1 directory agg: add CacheDirectoryAggregator to cache file listing, rescan dir only if changed 2016-03-19 20:34:09 -07:00
Ilya Kreymer
f5ee3c7bca inputreq: add reconstruct_request() to return a bytestring of the request, add test for inputreq 2016-03-19 20:32:37 -07:00
Ilya Kreymer
c96e419341 recorder: ensure filename is also tracked by the indexer, add tests
for redis file mapping
2016-03-19 10:24:28 -07:00
Ilya Kreymer
3452cf39e0 recorder: use more general MultiFileWARCWriter, supporting both keeping file open
and one-warc-per record use cases
2016-03-18 21:40:41 -07:00
Ilya Kreymer
e81457df5f rename WARCRecorder -> WARCWriter, add optional max_size to single warc recorder
per-record recorder combines http response/req into single file
2016-03-18 19:49:14 -07:00
Ilya Kreymer
b64be0dff1 recorder: add tests for single file writer, including file locking
dedup policy: support customizable dedup/skip/write policy plugins and add tests
2016-03-18 15:28:24 -07:00
Ilya Kreymer
cba8e4ee3a filters: more functional filter impl for header exclusion 2016-03-17 18:22:26 -07:00
Ilya Kreymer
58e8c709aa docker: add initial docker-compose, webagg Dockerfile 2016-03-16 18:42:15 -07:00
Ilya Kreymer
fe0f8ed1d8 Merge branch '0.11.3' into develop 2016-03-16 14:38:49 -07:00
Ilya Kreymer
cd65ddafbc bump version to 0.11.3 2016-03-16 14:29:42 -07:00
Ilya Kreymer
f962418c1f html rewrite typo: ensure rw_mod is set for meta content rewrite 2016-03-16 14:27:55 -07:00
Ilya Kreymer
c26660e20f cookies: use httplib headers pair list instead of requests headers dict to avoid 'set-cookie' headers being concatenated, as that messes up parsing in 3.5.1 2016-03-16 09:47:55 -07:00
Ilya Kreymer
ef5860901f warc record loader: if no content-length is specified on WARC record (as opposed to error or invalid), leave stream alone, don't force size of 0 and 204 2016-03-13 17:56:37 -07:00
Ilya Kreymer
8dc59ef6bd webagg: add test for live server config 2016-03-13 16:53:39 -07:00
Ilya Kreymer
06978bd8d2 recorder: check for empty input stream (support for direct proxy?) 2016-03-13 11:17:52 -07:00
Ilya Kreymer
709d2b1ea2 reorg: move StreamIter to utils 2016-03-12 23:29:23 -08:00
Ilya Kreymer
7a828017d1 recorder: clean up logging, ReadFullyStream moves to utils, get_request_uri to inputreq 2016-03-12 22:18:01 -08:00
Ilya Kreymer
49b6ae78a8 live loader: remove liverec (doesn't work well with gevent), use regular requests
instead of overriden version.
reconstruct header block from httplib header pairs list
move ReadFullyStream to utils
2016-03-12 22:15:24 -08:00
Ilya Kreymer
9adb8da3b7 recorder: add support for filtering collections to record by regex (default: .*)
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
2016-03-11 11:12:25 -08:00
Ilya Kreymer
2003925b75 setup: fix pywb py3 version to 0.30.0, add coverage for recorder 2016-03-11 11:11:43 -08:00
Ilya Kreymer
3b3e190cf4 testing: use test mixins for class-scope temp directory, live server creation
use processes instead of threads for live server
2016-03-11 11:10:22 -08:00
Ilya Kreymer
2051785e6b statusandheaders: add to_str() method with 'exclude_list' to support converting to str with certain headers
excluded. also supported by to_bytes()
2016-03-11 11:02:13 -08:00
Ilya Kreymer
46d013ab19 test redis: minor tweak to use @patch for fakeredis mock 2016-03-10 21:35:01 -08:00
Ilya Kreymer
c309637a3a tests: webagg test tweaks, create TempDirTests for sharing tests that require a temp dir 2016-03-10 16:04:27 -08:00
Ilya Kreymer
7b847311d5 dir agg: include filename in dir source name 2016-03-10 15:51:01 -08:00
Ilya Kreymer
3f734e1c98 tests: remove 3.2, fix auto_index test assert 2016-03-10 13:07:57 -08:00
Ilya Kreymer
42aa12f9ae test py3.2 also 2016-03-10 12:55:36 -08:00
Ilya Kreymer
34cc3ccacb versions and readme: update version to 0.30.0, update README with python 2 and 3 support 2016-03-10 12:51:14 -08:00
Ilya Kreymer
0f6e3da127 cdx: tests: add tests for comparison ops 2016-03-10 12:47:36 -08:00
Ilya Kreymer
e5ca9bf601 Merge branch 'master' into py3 2016-03-10 10:53:30 -08:00
Ilya Kreymer
effd618bb3 tests: add parse_comment test for html_rewriter 2016-03-10 10:10:51 -08:00
Ilya Kreymer
8ae692d630 Merge pull request #172 from machawk1/patch-3
Spelling: Quick comment fix
2016-03-10 09:52:25 -08:00
Ilya Kreymer
a25bb5e238 Merge pull request #166 from machawk1/patch-2
Fixed misspelling
2016-03-10 09:51:56 -08:00
Ilya Kreymer
12ecb29a01 tweak CHANGES 2016-03-10 09:48:28 -08:00