Ilya Kreymer
66ca8d8b26
http block loader: raise exception for 4xx, 5xx responses
...
tests: add tests for limitreader posting, fix charset for frame test
2016-07-31 12:56:00 -04:00
Ilya Kreymer
c8c0cecda3
rewrite improvements: if content-type is text/plain but mod is js_ or cs_, treat as js or css ( #31 )
...
header rewriter: ensure removed content-length and content-encoding are added back if no rewriting performed on response body
2016-07-27 21:34:58 -04:00
Ilya Kreymer
658303caad
rewrite headers: undo not rewriting x- headers, needs more research and exclusions (eg. x-frame-options)
2016-04-26 13:11:08 -07:00
Ilya Kreymer
4a60e15577
cookie rewrite improvements: #177
...
- don't remove max-age and expires if in 'live' rewrite mode (flag set on urlrewriter)
- remove secure only if replay prefix is not https
- fix expires UTC->GMT as cookie parsing chokes on UTC
- other rewriting: don't append rewrite prefix to x- headers
tests: add more cookie rewriting tests
2016-04-26 09:45:23 -07:00
Ilya Kreymer
c5a166f601
tests: use httpbin.org instead of example.com/ for range-request test
2016-03-26 22:28:04 -04:00
Ilya Kreymer
3f734e1c98
tests: remove 3.2, fix auto_index test assert
2016-03-10 13:07:57 -08:00
Ilya Kreymer
1d5b23413f
proxy: ensure proxy cert download sets content length
...
proxy options: 'use_default_coll' must specify exact default coll
(otherwise a random coll is chosen, as ordering is not defined)
travis: add py3.4, py3.5!
2016-02-23 18:09:09 -08:00
Ilya Kreymer
3a584a1ec3
py3: all tests pass, at last!
...
but not yet py2... need to resolve encoding in rewriting issues
2016-02-23 13:26:53 -08:00
Ilya Kreymer
1e54f8c8fa
proxy: add tests for proxy-mode 'Pywb-Rewrite-Prefix' header which adds optional prefix to proxy mode rewrites.. ensures such rewrites always absolute to include the prefix
2015-12-29 16:10:23 -08:00
Ilya Kreymer
a25096968a
proxy: ip resolver: show 500 error if incorrect coll preconfigured for ip-based settings (todo: make it configurable?)
2015-12-29 14:53:50 -08:00
Ilya Kreymer
381f350917
proxy: switching not available for ip resolver either
...
tests: update tests for auth and ip resolver to check that proxy magic is not set
2015-12-12 22:59:32 -08:00
Ilya Kreymer
7a0680fb35
memento: for not found timemap query, return empty timemap, instead of html query error page, closes #158
2015-11-30 09:40:07 -08:00
Ilya Kreymer
d98c1f6cf7
memento/api: add a new /collinfo.json end-point, enabled with 'enable_coll_info' config setting, which returns
...
the value fo collinfo.json template. Default template returns an entry for each handler route,
including the route path (id), title (name) and memento timegate and timemap paths, to be used with
an aggregator. Using a custom 'info_json' template can specify a different collinfo template, alternative to #69 (local aggregation)
Closes #146
2015-11-04 15:36:44 -08:00
Ilya Kreymer
bd2b5181a0
tests: add new tests for redis-based cache, #145
2015-10-30 13:18:58 -07:00
Ilya Kreymer
3132bfa7f4
cache: add a simple RedisCache implementation (alongside local and uwsgi)
...
proxy_ip_resolver: add option to use RedisCache if redis_cache_key set in config
proxy_ip_resolver: add 'delete' option to delete ip from cache, closes #145
2015-10-30 13:15:07 -07:00
Ilya Kreymer
eeb35ea3b4
proxy: add ProxyRouter wrapper to check for content-length and, if missing, perform full buffering (http1.0) or chunked encoding (http1.1) (separate from replay view buffering)
...
add tests for buffering and chunked encoding, fixes #143 , also tests no banner url-rewrite only proxy related to #142
2015-10-25 18:02:51 -07:00
Ilya Kreymer
979fcaeda3
tests: fix mock YoutubeDLWrapper after refactor, #141
2015-10-23 12:19:15 -07:00
Ilya Kreymer
39e824cb3a
live rewite proxy: decouple having http/https proxy from recording,
...
move youtubedl wrapper calls, metadata add calls to live rewrite proxy class for easier extension
closes #141 also improves #136
2015-10-23 11:57:12 -07:00
Ilya Kreymer
c7224ecceb
tests: use proxy str directly (imrpove test cov)
2015-10-23 11:54:16 -07:00
Ilya Kreymer
4ba4521b56
tests: use random port instead of 8080 for cli test to avoid conflicts with running services
2015-10-23 11:53:28 -07:00
Ilya Kreymer
e37636de84
cdxindexer: if latest ujson (with forward slash not-escaping) is available, use that when indexing, closes #140
...
tests: update indexer CDXJ tests to be order-independent
travis: install ujson for testing
2015-10-22 17:46:05 -07:00
Ilya Kreymer
e249f300e3
tests refactor! init pywb once per module, instead of once per test
...
refactor common init pattern to server_mock for now (can add fixtures also)
2015-10-14 20:34:46 -07:00
Ilya Kreymer
b612c584de
tests: test fixes for windows
2015-10-13 21:36:27 -07:00
Ilya Kreymer
6f7bd8c291
proxy resolvers: add tests for ip-based resolver
...
cache: default cache returns empty instead of raise KeyError on invalid key, to be consistent with uwsgi
2015-10-11 17:46:12 -07:00
Ilya Kreymer
31912b3bf7
proxy: update tests for new use_banner, use_client_rewrite options, #107
2015-09-09 13:22:32 -07:00
Ilya Kreymer
e1a9334a54
tests: update test to match cdx-convert
2015-08-25 23:06:00 +03:00
Ilya Kreymer
63c6efc851
autocolls test: patch wsgiref not waitress as it is default
2015-07-31 09:26:48 -07:00
Ilya Kreymer
f2a2c86552
tests: proxy check to ensure content-length header is always present in proxy mode
2015-07-30 11:06:44 -07:00
Ilya Kreymer
c2f99d6cfd
replay/memento: always include 'Content-Location' for in no-redir mode replay (not just for memento timegate), #122
2015-07-19 00:11:25 -07:00
Ilya Kreymer
66f5ad62b3
memento: when redir_to_exact is false, don't redirect latest replay/timegate to current timestamp, but return directly latest capture.
...
when memento enabled, the timegate now follows memento pattern 2.2 (http://tools.ietf.org/html/rfc7089#section-4.2.2 )
also return content-location instead of location, update memento no-redirect tests to match new behavior. closes #122
2015-07-18 23:30:31 -07:00
Ilya Kreymer
080587516b
youtube-dl tests: use mock youtube-dl info for tests
2015-06-27 20:46:55 -07:00
Ilya Kreymer
f0359877f0
youtube-dl: remove from dependency, installation is optional. Return 404 if attempting live
...
proxy of videos and youtube-dl is not available (the only use case).
HTTPParser wrapping logic no longer needed in latest versions
Modify tests to only run if youtube-dl is installed in cases where it is not available #118
2015-06-27 16:11:59 -07:00
Ilya Kreymer
06fcc89de6
readers: support 'content-encoding: deflate' using different zlib decompression options
...
support default and alt settings for attempting to decompress deflate stream
tests: add tests with httpbin.org/deflate Fixes #115
2015-06-24 13:11:33 -07:00
Ilya Kreymer
7bf8b97cb0
tests: add tests for root collection access, and also a custom handler passed to pywb_init
...
(a simple redirect handler)
2015-04-17 11:48:50 -07:00
Ilya Kreymer
307809bbe9
live-rewrite-server: switch to 'inverse' frame mode by default,
...
switch from /rewrite/ to /live/ path, update tests
2015-04-13 13:00:06 -07:00
Ilya Kreymer
f9bd2ba55a
jinja template: use shared template in J2Template, init on first use
2015-04-03 10:43:39 -07:00
Ilya Kreymer
4a85869427
cli refactor: use classes in cli to allow custom options
...
get rid of custom init for live_rewrite_handler, just use create_wb_router()
with custom config for consistent init
2015-04-03 10:43:39 -07:00
Ilya Kreymer
a34607764e
manager: validate name on collection init: must start with wordchar and can contain wordchar or -
2015-04-03 01:18:35 -07:00
Ilya Kreymer
8bd6787595
'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay
...
add additional test for inverse framed replay, #92
fix framed replay url replace slash
2015-04-01 16:21:44 -07:00
Ilya Kreymer
dd30e3f2a7
refactor: fixes for compat with latest certauth>=1.1.0
2015-03-30 09:38:42 -07:00
Ilya Kreymer
90eee03cdb
fixes for windows:
...
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
2015-03-25 10:56:53 -07:00
Ilya Kreymer
ec7a29a3ba
static paths: ensure consistent renaming of static/default -> static/__pywb for bundled static path
2015-03-23 16:15:37 -07:00
Ilya Kreymer
da7532a1f8
wb-manager: rename 'migrate' to 'cdx-convert' for clarity
2015-03-23 11:05:02 -07:00
Ilya Kreymer
ae363ad368
autoindex and cli: add autoindex to cli with 'wayback -a' option, #81
2015-03-22 23:03:39 -07:00
Ilya Kreymer
e8db31d066
cli: improve wayback cli to take optional port, threads and working dir arguments
...
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
733642551d
manager: support autoindexing! ( #91 ) wb-manager autoindex will use watchdog library to detect creation/updates
...
to any warc/arc in specified collection or across all and update autoindex cdx
cdx indexing: add --dir-root option to specify custom relative root dir for filenames used in cdx
2015-03-22 17:55:38 -07:00
Ilya Kreymer
b43a7f94f3
manager: add cdx -> cdxj migration tool #80 , which will convert all cdxs in a directory to cdxj, removing original files
...
migration will also recanonicalize the urlkey to surt form
add migration test using non-surt, 9-field cdx (created from samples)
cdxindexer: fix multi warc->multi cdx indexing options
2015-03-19 20:57:33 -07:00
Ilya Kreymer
c5b5c8ee4b
manager: fix index path to index.cdxj
2015-03-19 13:41:48 -07:00
Ilya Kreymer
ea460bb0f0
cdxj: support cdx json output from cdx server with output='json' (not yet default)
...
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
fe1c32c8f7
cdxj: support loading cdxj ( #76 )
...
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75 ) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
2015-03-19 12:36:49 -07:00
