1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

1709 Commits

Author SHA1 Message Date
Ilya Kreymer
8ae692d630 Merge pull request #172 from machawk1/patch-3
Spelling: Quick comment fix
2016-03-10 09:52:25 -08:00
Ilya Kreymer
a25bb5e238 Merge pull request #166 from machawk1/patch-2
Fixed misspelling
2016-03-10 09:51:56 -08:00
Ilya Kreymer
12ecb29a01 tweak CHANGES 2016-03-10 09:48:28 -08:00
Ilya Kreymer
2f67b78023 Update CHANGES for 0.11.2 2016-03-10 09:47:12 -08:00
Ilya Kreymer
c1bdeac92b redis: fix redis key lookup, add tests for zrangebylex with new fakeredis 2016-03-09 18:33:04 -08:00
Ilya Kreymer
31fb2f926f add recorder app, initial pass! 2016-03-09 14:33:36 -08:00
Ilya Kreymer
1499f0e611 add shared README.rst and coverage 2016-03-09 14:33:11 -08:00
Ilya Kreymer
34386578a5 shared setup: move webagg test to webagg/test 2016-03-09 14:29:14 -08:00
Ilya Kreymer
0198bf1213 Merge branch 'develop' into py3 2016-03-09 07:29:54 -08:00
Ilya Kreymer
bc84c2fda0 indexing: declare 'record' and bail if no record was loaded, add test for empty file indexing, fixes #168 2016-03-09 07:27:03 -08:00
Ilya Kreymer
3477cb0bb5 drop process/thread mixin support (doesn't work as well on py2) could readd processes only if need arises, but for now focusing on gevent
rename header Source-Coll -> WebAgg-Source-Coll
2016-03-08 10:56:03 -08:00
Ilya Kreymer
348fb133e0 add upstream/proxy tests 2016-03-08 10:29:59 -08:00
Ilya Kreymer
107ba9aabc add ProxyLiveIndexSource for proxying upstream conn directly w/o a second index query
liveloader: if 'memento_url' key is set, then memento-datetime header must be present or its an error response
liveindexsource: add option to specify custom live path (eg. prefix for cacheing)
fix test cases changed due to ia (todo: mock up all external data!)
2016-03-08 10:27:13 -08:00
Ilya Kreymer
b3372f64c3 timeutils: add datetime_to_iso_date 2016-03-08 08:39:45 -08:00
Ilya Kreymer
3af2979cf1 cdx: skip any fields starting with '_' when serializing 2016-03-08 08:38:55 -08:00
Ilya Kreymer
5ad01f7d64 statusandheaders: add a to_bytes() func for serializing header 2016-03-08 08:26:51 -08:00
Ilya Kreymer
c1895ae70f loaders: return full WARC record in response, no need for upstream response handler
add UpstreamAggIndexSource to simplify upstream aggregator config, add test for upstream config
bottle app: wrap in a ResAppAgg, allow multiple bottle apps
py2: non-gevent concurrency not supported
2016-03-06 23:12:14 -08:00
Ilya Kreymer
0823ff4bd0 added 'upstream' handler for connecting to another webagg when 'upstream_url' is set
output 'is_live' as string in live index
2016-03-06 09:10:17 -08:00
Ilya Kreymer
20ebccc13e handlers: return out_headers directly instead of setting bottle response, contains bottle dependency to app.py (to allow alternate impl not using bottle)
param parsing: instead of setting custom _src_params and _all_params, use a custom ParamFormatter which will check param dict for params with prefix and custom name
2016-03-05 16:49:26 -08:00
Ilya Kreymer
648e567805 statusandheaders: add __str__ func to reconstruct statusline + headers text 2016-03-04 12:48:36 -08:00
Mat Kelly
96da397456 Quick comment fix 2016-03-04 11:17:35 -05:00
Ilya Kreymer
bb806d7f26 Merge branch 'develop' into py3 2016-03-03 14:09:00 -08:00
Ilya Kreymer
5ddc843094 resolvingloader: use string_types instead of str for compat 2016-03-03 14:05:14 -08:00
Ilya Kreymer
bdda1b8c03 minor fixes for py2 support 2016-03-03 13:58:09 -08:00
Ilya Kreymer
a6dc57cf4a post query: ensure post query optional buffer is a byte not string buffer
exceptions: move LiveRequestException to wbexceptions
cdx query: support for 'alt_url' which, if set, is used to create start_key and end_key
2016-03-03 13:13:44 -08:00
Ilya Kreymer
896f81fd1c Add README.rst 2016-03-03 12:09:17 -08:00
Ilya Kreymer
ed1d3555c3 rename rezag -> webagg
rename aggindexsource -> aggregator
2016-03-03 11:55:43 -08:00
Ilya Kreymer
98830147b5 add memento headers to all response loaders, use BaseLoader base class, update tests
for memento headers
2016-03-03 11:04:28 -08:00
Ilya Kreymer
65e969a492 errors and timeouts reported back to the user via ResErrors header
add new /index, /resource access point system
2016-03-02 18:13:13 -08:00
Ilya Kreymer
1f3763d02c misc fixes: add route listing, more not found tests, timemap use file:// with ranges 2016-03-01 14:46:05 -08:00
Ilya Kreymer
008e5284b1 seperate iter_sources from list_sources api
all errors returned as json block with error msg
tests for not found, invalid errors
2016-02-29 12:34:06 -08:00
Ilya Kreymer
68090d00c1 add routing setup via app.py
add full test suite for handlers and responseloaders, as well as timeouts
2016-02-28 14:33:08 -08:00
Ilya Kreymer
c88c5f4cca add new package setup!
add tests and testdata, splitting mem and dir agg tests
2016-02-26 18:25:10 -08:00
Ilya Kreymer
fc5d7cc7cd rewrite: add rewriting of <meta> content="" attribute if it is a url 2016-02-25 18:49:31 -08:00
Ilya Kreymer
8fc789cc8f rewrite: leave out charset in top-frame and don't modify it in replay frame
to allow browser to detect best charset, as it would on original page if it is absent)
see #170 for details
2016-02-25 18:25:53 -08:00
Ilya Kreymer
c76aa17b78 wb.js: pad timestamp to 14 digits 2016-02-25 18:25:28 -08:00
Ilya Kreymer
e6361c58ac bump version to 0.11.2 2016-02-25 18:15:29 -08:00
Ilya Kreymer
398e8f1a77 inputrequest: add input request handling (direct wsgi headers) or as a prepared post request
add timemap link output
rename source_name -> source
2016-02-24 14:22:29 -08:00
Ilya Kreymer
20bd9d118b travis: remove --use-mirrors 2016-02-23 18:39:27 -08:00
Ilya Kreymer
1d5b23413f proxy: ensure proxy cert download sets content length
proxy options: 'use_default_coll' must specify exact default coll
(otherwise a random coll is chosen, as ordering is not defined)
travis: add py3.4, py3.5!
2016-02-23 18:09:09 -08:00
Ilya Kreymer
cebd6b6239 rewrite: fix rewriting encoding -- for best rewriting, keep strategy of encoding
insert to match page, then using latin-1 for rewriting. support for non-ascii
based encoding still needed
2016-02-23 18:07:34 -08:00
Ilya Kreymer
3a584a1ec3 py3: all tests pass, at last!
but not yet py2... need to resolve encoding in rewriting issues
2016-02-23 13:26:53 -08:00
Ilya Kreymer
0dff388e4e cdx: CDXQuery takes params dict not **params
CDXObject comparison using to_json()
2016-02-23 01:36:39 -08:00
Ilya Kreymer
57991fd0cf cdx: ensure url param required check is performed on init 2016-02-22 13:59:07 -08:00
Ilya Kreymer
af7c876263 cdx: ensure CDXQuery computes key and end_key automatically
key and end_key encoded as utf-8 by default
2016-02-22 13:39:47 -08:00
Ilya Kreymer
1a0b2fba17 add aggregate index source and tests! 2016-02-22 13:30:12 -08:00
Ilya Kreymer
7513011cac path resolvers: add PathResolverMapper for converting paths to resolvers,
ResolvingLoader takes a list of resolvers, not paths (to allow for custom overrides)
ResolvingLoader and ArcWarcRecordLoader support 'no_record_parse' on load to not parse http headers from stream
2016-02-19 22:33:38 -08:00
Ilya Kreymer
37198767ed add utils, responseloader and liverec 2016-02-19 17:27:19 -08:00
Ilya Kreymer
baa02add69 add indexloader and tests, including file, redis, remote cdx, memento, and live sources 2016-02-19 17:25:54 -08:00
Ilya Kreymer
bd841b91a9 more python 3 support work -- pywb.cdx, pywb.warc tests succeed
most relative imports replaced with absolute
2016-02-18 21:26:40 -08:00