backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
1,626 Commits 56 Branches 63 Tags
Commit Graph

1626 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
06978bd8d2 recorder: check for empty input stream (support for direct proxy?) 2016-03-13 11:17:52 -07:00
Ilya Kreymer
709d2b1ea2 reorg: move StreamIter to utils 2016-03-12 23:29:23 -08:00
Ilya Kreymer
7a828017d1 recorder: clean up logging, ReadFullyStream moves to utils, get_request_uri to inputreq 2016-03-12 22:18:01 -08:00
Ilya Kreymer
49b6ae78a8 live loader: remove liverec (doesn't work well with gevent), use regular requests 
instead of overriden version.
reconstruct header block from httplib header pairs list
move ReadFullyStream to utils
 2016-03-12 22:15:24 -08:00
Ilya Kreymer
9adb8da3b7 recorder: add support for filtering collections to record by regex (default: .*) 
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
 2016-03-11 11:12:25 -08:00
Ilya Kreymer
2003925b75 setup: fix pywb py3 version to 0.30.0, add coverage for recorder 2016-03-11 11:11:43 -08:00
Ilya Kreymer
3b3e190cf4 testing: use test mixins for class-scope temp directory, live server creation 
use processes instead of threads for live server
 2016-03-11 11:10:22 -08:00
Ilya Kreymer
2051785e6b statusandheaders: add to_str() method with 'exclude_list' to support converting to str with certain headers 
excluded. also supported by to_bytes()
 2016-03-11 11:02:13 -08:00
Ilya Kreymer
46d013ab19 test redis: minor tweak to use @patch for fakeredis mock 2016-03-10 21:35:01 -08:00
Ilya Kreymer
c309637a3a tests: webagg test tweaks, create TempDirTests for sharing tests that require a temp dir 2016-03-10 16:04:27 -08:00
Ilya Kreymer
7b847311d5 dir agg: include filename in dir source name 2016-03-10 15:51:01 -08:00
Ilya Kreymer
3f734e1c98 tests: remove 3.2, fix auto_index test assert 2016-03-10 13:07:57 -08:00
Ilya Kreymer
42aa12f9ae test py3.2 also 2016-03-10 12:55:36 -08:00
Ilya Kreymer
34cc3ccacb versions and readme: update version to 0.30.0, update README with python 2 and 3 support 2016-03-10 12:51:14 -08:00
Ilya Kreymer
0f6e3da127 cdx: tests: add tests for comparison ops 2016-03-10 12:47:36 -08:00
Ilya Kreymer
e5ca9bf601 Merge branch 'master' into py3 2016-03-10 10:53:30 -08:00
Ilya Kreymer
effd618bb3 tests: add parse_comment test for html_rewriter 2016-03-10 10:10:51 -08:00
Ilya Kreymer
8ae692d630 Merge pull request #172 from machawk1/patch-3 
Spelling: Quick comment fix
 2016-03-10 09:52:25 -08:00
Ilya Kreymer
a25bb5e238 Merge pull request #166 from machawk1/patch-2 
Fixed misspelling
 2016-03-10 09:51:56 -08:00
Ilya Kreymer
12ecb29a01 tweak CHANGES 2016-03-10 09:48:28 -08:00
Ilya Kreymer
2f67b78023 Update CHANGES for 0.11.2 2016-03-10 09:47:12 -08:00
Ilya Kreymer
c1bdeac92b redis: fix redis key lookup, add tests for zrangebylex with new fakeredis 2016-03-09 18:33:04 -08:00
Ilya Kreymer
31fb2f926f add recorder app, initial pass! 2016-03-09 14:33:36 -08:00
Ilya Kreymer
1499f0e611 add shared README.rst and coverage 2016-03-09 14:33:11 -08:00
Ilya Kreymer
34386578a5 shared setup: move webagg test to webagg/test 2016-03-09 14:29:14 -08:00
Ilya Kreymer
0198bf1213 Merge branch 'develop' into py3 2016-03-09 07:29:54 -08:00
Ilya Kreymer
bc84c2fda0 indexing: declare 'record' and bail if no record was loaded, add test for empty file indexing, fixes #168 2016-03-09 07:27:03 -08:00
Ilya Kreymer
3477cb0bb5 drop process/thread mixin support (doesn't work as well on py2) could readd processes only if need arises, but for now focusing on gevent 
rename header Source-Coll -> WebAgg-Source-Coll
 2016-03-08 10:56:03 -08:00
Ilya Kreymer
348fb133e0 add upstream/proxy tests 2016-03-08 10:29:59 -08:00
Ilya Kreymer
107ba9aabc add ProxyLiveIndexSource for proxying upstream conn directly w/o a second index query 
liveloader: if 'memento_url' key is set, then memento-datetime header must be present or its an error response
liveindexsource: add option to specify custom live path (eg. prefix for cacheing)
fix test cases changed due to ia (todo: mock up all external data!)
 2016-03-08 10:27:13 -08:00
Ilya Kreymer
b3372f64c3 timeutils: add datetime_to_iso_date 2016-03-08 08:39:45 -08:00
Ilya Kreymer
3af2979cf1 cdx: skip any fields starting with '_' when serializing 2016-03-08 08:38:55 -08:00
Ilya Kreymer
5ad01f7d64 statusandheaders: add a to_bytes() func for serializing header 2016-03-08 08:26:51 -08:00
Ilya Kreymer
c1895ae70f loaders: return full WARC record in response, no need for upstream response handler 
add UpstreamAggIndexSource to simplify upstream aggregator config, add test for upstream config
bottle app: wrap in a ResAppAgg, allow multiple bottle apps
py2: non-gevent concurrency not supported
 2016-03-06 23:12:14 -08:00
Ilya Kreymer
0823ff4bd0 added 'upstream' handler for connecting to another webagg when 'upstream_url' is set 
output 'is_live' as string in live index
 2016-03-06 09:10:17 -08:00
Ilya Kreymer
20ebccc13e handlers: return out_headers directly instead of setting bottle response, contains bottle dependency to app.py (to allow alternate impl not using bottle) 
param parsing: instead of setting custom _src_params and _all_params, use a custom ParamFormatter which will check param dict for params with prefix and custom name
 2016-03-05 16:49:26 -08:00
Ilya Kreymer
648e567805 statusandheaders: add __str__ func to reconstruct statusline + headers text 2016-03-04 12:48:36 -08:00
Mat Kelly
96da397456 Quick comment fix 2016-03-04 11:17:35 -05:00
Ilya Kreymer
bb806d7f26 Merge branch 'develop' into py3 2016-03-03 14:09:00 -08:00
Ilya Kreymer
5ddc843094 resolvingloader: use string_types instead of str for compat 2016-03-03 14:05:14 -08:00
Ilya Kreymer
bdda1b8c03 minor fixes for py2 support 2016-03-03 13:58:09 -08:00
Ilya Kreymer
a6dc57cf4a post query: ensure post query optional buffer is a byte not string buffer 
exceptions: move LiveRequestException to wbexceptions
cdx query: support for 'alt_url' which, if set, is used to create start_key and end_key
 2016-03-03 13:13:44 -08:00
Ilya Kreymer
896f81fd1c Add README.rst 2016-03-03 12:09:17 -08:00
Ilya Kreymer
ed1d3555c3 rename rezag -> webagg 
rename aggindexsource -> aggregator
 2016-03-03 11:55:43 -08:00
Ilya Kreymer
98830147b5 add memento headers to all response loaders, use BaseLoader base class, update tests 
for memento headers
 2016-03-03 11:04:28 -08:00
Ilya Kreymer
65e969a492 errors and timeouts reported back to the user via ResErrors header 
add new /index, /resource access point system
 2016-03-02 18:13:13 -08:00
Ilya Kreymer
1f3763d02c misc fixes: add route listing, more not found tests, timemap use file:// with ranges 2016-03-01 14:46:05 -08:00
Ilya Kreymer
008e5284b1 seperate iter_sources from list_sources api 
all errors returned as json block with error msg
tests for not found, invalid errors
 2016-02-29 12:34:06 -08:00
Ilya Kreymer
68090d00c1 add routing setup via app.py 
add full test suite for handlers and responseloaders, as well as timeouts
 2016-02-28 14:33:08 -08:00
Ilya Kreymer
c88c5f4cca add new package setup! 
add tests and testdata, splitting mem and dir agg tests
 2016-02-26 18:25:10 -08:00