Ilya Kreymer
06978bd8d2
recorder: check for empty input stream (support for direct proxy?)
2016-03-13 11:17:52 -07:00
Ilya Kreymer
709d2b1ea2
reorg: move StreamIter to utils
2016-03-12 23:29:23 -08:00
Ilya Kreymer
7a828017d1
recorder: clean up logging, ReadFullyStream moves to utils, get_request_uri to inputreq
2016-03-12 22:18:01 -08:00
Ilya Kreymer
49b6ae78a8
live loader: remove liverec (doesn't work well with gevent), use regular requests
...
instead of overriden version.
reconstruct header block from httplib header pairs list
move ReadFullyStream to utils
2016-03-12 22:15:24 -08:00
Ilya Kreymer
9adb8da3b7
recorder: add support for filtering collections to record by regex (default: .*)
...
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
2016-03-11 11:12:25 -08:00
Ilya Kreymer
2003925b75
setup: fix pywb py3 version to 0.30.0, add coverage for recorder
2016-03-11 11:11:43 -08:00
Ilya Kreymer
3b3e190cf4
testing: use test mixins for class-scope temp directory, live server creation
...
use processes instead of threads for live server
2016-03-11 11:10:22 -08:00
Ilya Kreymer
2051785e6b
statusandheaders: add to_str() method with 'exclude_list' to support converting to str with certain headers
...
excluded. also supported by to_bytes()
2016-03-11 11:02:13 -08:00
Ilya Kreymer
46d013ab19
test redis: minor tweak to use @patch for fakeredis mock
2016-03-10 21:35:01 -08:00
Ilya Kreymer
c309637a3a
tests: webagg test tweaks, create TempDirTests for sharing tests that require a temp dir
2016-03-10 16:04:27 -08:00
Ilya Kreymer
7b847311d5
dir agg: include filename in dir source name
2016-03-10 15:51:01 -08:00
Ilya Kreymer
3f734e1c98
tests: remove 3.2, fix auto_index test assert
2016-03-10 13:07:57 -08:00
Ilya Kreymer
42aa12f9ae
test py3.2 also
2016-03-10 12:55:36 -08:00
Ilya Kreymer
34cc3ccacb
versions and readme: update version to 0.30.0, update README with python 2 and 3 support
2016-03-10 12:51:14 -08:00
Ilya Kreymer
0f6e3da127
cdx: tests: add tests for comparison ops
2016-03-10 12:47:36 -08:00
Ilya Kreymer
e5ca9bf601
Merge branch 'master' into py3
2016-03-10 10:53:30 -08:00
Ilya Kreymer
effd618bb3
tests: add parse_comment test for html_rewriter
2016-03-10 10:10:51 -08:00
Ilya Kreymer
8ae692d630
Merge pull request #172 from machawk1/patch-3
...
Spelling: Quick comment fix
2016-03-10 09:52:25 -08:00
Ilya Kreymer
a25bb5e238
Merge pull request #166 from machawk1/patch-2
...
Fixed misspelling
2016-03-10 09:51:56 -08:00
Ilya Kreymer
12ecb29a01
tweak CHANGES
2016-03-10 09:48:28 -08:00
Ilya Kreymer
2f67b78023
Update CHANGES for 0.11.2
2016-03-10 09:47:12 -08:00
Ilya Kreymer
c1bdeac92b
redis: fix redis key lookup, add tests for zrangebylex with new fakeredis
2016-03-09 18:33:04 -08:00
Ilya Kreymer
31fb2f926f
add recorder app, initial pass!
2016-03-09 14:33:36 -08:00
Ilya Kreymer
1499f0e611
add shared README.rst and coverage
2016-03-09 14:33:11 -08:00
Ilya Kreymer
34386578a5
shared setup: move webagg test to webagg/test
2016-03-09 14:29:14 -08:00
Ilya Kreymer
0198bf1213
Merge branch 'develop' into py3
2016-03-09 07:29:54 -08:00
Ilya Kreymer
bc84c2fda0
indexing: declare 'record' and bail if no record was loaded, add test for empty file indexing, fixes #168
2016-03-09 07:27:03 -08:00
Ilya Kreymer
3477cb0bb5
drop process/thread mixin support (doesn't work as well on py2) could readd processes only if need arises, but for now focusing on gevent
...
rename header Source-Coll -> WebAgg-Source-Coll
2016-03-08 10:56:03 -08:00
Ilya Kreymer
348fb133e0
add upstream/proxy tests
2016-03-08 10:29:59 -08:00
Ilya Kreymer
107ba9aabc
add ProxyLiveIndexSource for proxying upstream conn directly w/o a second index query
...
liveloader: if 'memento_url' key is set, then memento-datetime header must be present or its an error response
liveindexsource: add option to specify custom live path (eg. prefix for cacheing)
fix test cases changed due to ia (todo: mock up all external data!)
2016-03-08 10:27:13 -08:00
Ilya Kreymer
b3372f64c3
timeutils: add datetime_to_iso_date
2016-03-08 08:39:45 -08:00
Ilya Kreymer
3af2979cf1
cdx: skip any fields starting with '_' when serializing
2016-03-08 08:38:55 -08:00
Ilya Kreymer
5ad01f7d64
statusandheaders: add a to_bytes() func for serializing header
2016-03-08 08:26:51 -08:00
Ilya Kreymer
c1895ae70f
loaders: return full WARC record in response, no need for upstream response handler
...
add UpstreamAggIndexSource to simplify upstream aggregator config, add test for upstream config
bottle app: wrap in a ResAppAgg, allow multiple bottle apps
py2: non-gevent concurrency not supported
2016-03-06 23:12:14 -08:00
Ilya Kreymer
0823ff4bd0
added 'upstream' handler for connecting to another webagg when 'upstream_url' is set
...
output 'is_live' as string in live index
2016-03-06 09:10:17 -08:00
Ilya Kreymer
20ebccc13e
handlers: return out_headers directly instead of setting bottle response, contains bottle dependency to app.py (to allow alternate impl not using bottle)
...
param parsing: instead of setting custom _src_params and _all_params, use a custom ParamFormatter which will check param dict for params with prefix and custom name
2016-03-05 16:49:26 -08:00
Ilya Kreymer
648e567805
statusandheaders: add __str__ func to reconstruct statusline + headers text
2016-03-04 12:48:36 -08:00
Mat Kelly
96da397456
Quick comment fix
2016-03-04 11:17:35 -05:00
Ilya Kreymer
bb806d7f26
Merge branch 'develop' into py3
2016-03-03 14:09:00 -08:00
Ilya Kreymer
5ddc843094
resolvingloader: use string_types instead of str for compat
2016-03-03 14:05:14 -08:00
Ilya Kreymer
bdda1b8c03
minor fixes for py2 support
2016-03-03 13:58:09 -08:00
Ilya Kreymer
a6dc57cf4a
post query: ensure post query optional buffer is a byte not string buffer
...
exceptions: move LiveRequestException to wbexceptions
cdx query: support for 'alt_url' which, if set, is used to create start_key and end_key
2016-03-03 13:13:44 -08:00
Ilya Kreymer
896f81fd1c
Add README.rst
2016-03-03 12:09:17 -08:00
Ilya Kreymer
ed1d3555c3
rename rezag -> webagg
...
rename aggindexsource -> aggregator
2016-03-03 11:55:43 -08:00
Ilya Kreymer
98830147b5
add memento headers to all response loaders, use BaseLoader base class, update tests
...
for memento headers
2016-03-03 11:04:28 -08:00
Ilya Kreymer
65e969a492
errors and timeouts reported back to the user via ResErrors header
...
add new /index, /resource access point system
2016-03-02 18:13:13 -08:00
Ilya Kreymer
1f3763d02c
misc fixes: add route listing, more not found tests, timemap use file:// with ranges
2016-03-01 14:46:05 -08:00
Ilya Kreymer
008e5284b1
seperate iter_sources from list_sources api
...
all errors returned as json block with error msg
tests for not found, invalid errors
2016-02-29 12:34:06 -08:00
Ilya Kreymer
68090d00c1
add routing setup via app.py
...
add full test suite for handlers and responseloaders, as well as timeouts
2016-02-28 14:33:08 -08:00
Ilya Kreymer
c88c5f4cca
add new package setup!
...
add tests and testdata, splitting mem and dir agg tests
2016-02-26 18:25:10 -08:00