Ilya Kreymer
e060ea7b56
frontendapp: encapsulate, don't extend rewriterapp
...
rewriterapp: add 'Content-Location' if fuzzy match, or if using memento
tests: fix test to check for Content-Location for fuzzy match instead of redirect
2017-04-21 15:37:21 -07:00
Ilya Kreymer
f593b5f80f
trailing slash fix: add trailing slash, preserving query, if no slash present after hostname ( #211 )
2017-04-04 18:10:49 -07:00
Ilya Kreymer
a82cfc1ab2
rewriter: add rewrite_dash for rewriting DASH and HLS manifests!
...
rewriter: refactor to use mixins to extend base rewriter (todo: more refactoring)
fuzzy-matcher: support for additional 'match_filters' to filter fuzzy results via optional regexes by mime type,
eg. allow more lenient fuzzy matching on DASH manifests than other resources (for now)
fuzzy-matching: add WebAgg-Fuzzy-Match response header if response is fuzzy matched, redirect to exact match in rewriterapp
2017-03-20 14:41:12 -07:00
Ilya Kreymer
0f0c20a03a
fuzzy matching: new, clean fuzzy matcher implementation for webagg
...
rules: default rule: fuzzy match urls ignoring prefix match (needs more testing)
tests: update tests for new broad fuzzy match rule
2017-03-14 11:44:15 -07:00
Ilya Kreymer
8fe2c1b5bd
apps & cli: remove old apps, keep:
...
- webagg-server
- wayback
- live-rewrite-server
support adding custom settings to AutoApp
support for --live flag that automatically adds live-web source at '/live'
tests: disable cdx_server tests as old cdx_server removed
2017-03-12 12:21:54 -07:00
Ilya Kreymer
a4b770d34e
new-pywb refactor!
...
frontendapp compatibility
- add support for separate not found page for 404s (not_found.html)
- support for exception handling with error template (error.html)
- support for home page (index.html)
- add memento headers for replay
- add referrer fallback check
- tests: port integration tests for front-end replay, cdx server
- not included: proxy mode, exact redirect mode, non-framed replay
- move unused tests to tests_disabled
- cli: add optional werkzeug profiler with --profile flag
2017-02-27 19:07:51 -08:00
Ilya Kreymer
50a3353da3
wsgi server: default to gevent-based wsgi server for all cmd line server apps, add -s command for specifying server #201
...
cli: add 'webagg-server' cli command for running new webagg system
tests: fix cli test for gevent server
2016-12-09 16:46:33 -08:00
Ilya Kreymer
ab77c1b6d9
refactor autoindex: switch to gevent-based simple polling, as watchdog doesn't work with gevent #200
2016-11-11 10:31:48 -08:00
Ilya Kreymer
66ca8d8b26
http block loader: raise exception for 4xx, 5xx responses
...
tests: add tests for limitreader posting, fix charset for frame test
2016-07-31 12:56:00 -04:00
Ilya Kreymer
c8c0cecda3
rewrite improvements: if content-type is text/plain but mod is js_ or cs_, treat as js or css ( #31 )
...
header rewriter: ensure removed content-length and content-encoding are added back if no rewriting performed on response body
2016-07-27 21:34:58 -04:00
Ilya Kreymer
658303caad
rewrite headers: undo not rewriting x- headers, needs more research and exclusions (eg. x-frame-options)
2016-04-26 13:11:08 -07:00
Ilya Kreymer
4a60e15577
cookie rewrite improvements: #177
...
- don't remove max-age and expires if in 'live' rewrite mode (flag set on urlrewriter)
- remove secure only if replay prefix is not https
- fix expires UTC->GMT as cookie parsing chokes on UTC
- other rewriting: don't append rewrite prefix to x- headers
tests: add more cookie rewriting tests
2016-04-26 09:45:23 -07:00
Ilya Kreymer
c5a166f601
tests: use httpbin.org instead of example.com/ for range-request test
2016-03-26 22:28:04 -04:00
Ilya Kreymer
3f734e1c98
tests: remove 3.2, fix auto_index test assert
2016-03-10 13:07:57 -08:00
Ilya Kreymer
1d5b23413f
proxy: ensure proxy cert download sets content length
...
proxy options: 'use_default_coll' must specify exact default coll
(otherwise a random coll is chosen, as ordering is not defined)
travis: add py3.4, py3.5!
2016-02-23 18:09:09 -08:00
Ilya Kreymer
3a584a1ec3
py3: all tests pass, at last!
...
but not yet py2... need to resolve encoding in rewriting issues
2016-02-23 13:26:53 -08:00
Ilya Kreymer
1e54f8c8fa
proxy: add tests for proxy-mode 'Pywb-Rewrite-Prefix' header which adds optional prefix to proxy mode rewrites.. ensures such rewrites always absolute to include the prefix
2015-12-29 16:10:23 -08:00
Ilya Kreymer
a25096968a
proxy: ip resolver: show 500 error if incorrect coll preconfigured for ip-based settings (todo: make it configurable?)
2015-12-29 14:53:50 -08:00
Ilya Kreymer
381f350917
proxy: switching not available for ip resolver either
...
tests: update tests for auth and ip resolver to check that proxy magic is not set
2015-12-12 22:59:32 -08:00
Ilya Kreymer
7a0680fb35
memento: for not found timemap query, return empty timemap, instead of html query error page, closes #158
2015-11-30 09:40:07 -08:00
Ilya Kreymer
d98c1f6cf7
memento/api: add a new /collinfo.json end-point, enabled with 'enable_coll_info' config setting, which returns
...
the value fo collinfo.json template. Default template returns an entry for each handler route,
including the route path (id), title (name) and memento timegate and timemap paths, to be used with
an aggregator. Using a custom 'info_json' template can specify a different collinfo template, alternative to #69 (local aggregation)
Closes #146
2015-11-04 15:36:44 -08:00
Ilya Kreymer
bd2b5181a0
tests: add new tests for redis-based cache, #145
2015-10-30 13:18:58 -07:00
Ilya Kreymer
3132bfa7f4
cache: add a simple RedisCache implementation (alongside local and uwsgi)
...
proxy_ip_resolver: add option to use RedisCache if redis_cache_key set in config
proxy_ip_resolver: add 'delete' option to delete ip from cache, closes #145
2015-10-30 13:15:07 -07:00
Ilya Kreymer
eeb35ea3b4
proxy: add ProxyRouter wrapper to check for content-length and, if missing, perform full buffering (http1.0) or chunked encoding (http1.1) (separate from replay view buffering)
...
add tests for buffering and chunked encoding, fixes #143 , also tests no banner url-rewrite only proxy related to #142
2015-10-25 18:02:51 -07:00
Ilya Kreymer
979fcaeda3
tests: fix mock YoutubeDLWrapper after refactor, #141
2015-10-23 12:19:15 -07:00
Ilya Kreymer
39e824cb3a
live rewite proxy: decouple having http/https proxy from recording,
...
move youtubedl wrapper calls, metadata add calls to live rewrite proxy class for easier extension
closes #141 also improves #136
2015-10-23 11:57:12 -07:00
Ilya Kreymer
c7224ecceb
tests: use proxy str directly (imrpove test cov)
2015-10-23 11:54:16 -07:00
Ilya Kreymer
4ba4521b56
tests: use random port instead of 8080 for cli test to avoid conflicts with running services
2015-10-23 11:53:28 -07:00
Ilya Kreymer
e37636de84
cdxindexer: if latest ujson (with forward slash not-escaping) is available, use that when indexing, closes #140
...
tests: update indexer CDXJ tests to be order-independent
travis: install ujson for testing
2015-10-22 17:46:05 -07:00
Ilya Kreymer
e249f300e3
tests refactor! init pywb once per module, instead of once per test
...
refactor common init pattern to server_mock for now (can add fixtures also)
2015-10-14 20:34:46 -07:00
Ilya Kreymer
b612c584de
tests: test fixes for windows
2015-10-13 21:36:27 -07:00
Ilya Kreymer
6f7bd8c291
proxy resolvers: add tests for ip-based resolver
...
cache: default cache returns empty instead of raise KeyError on invalid key, to be consistent with uwsgi
2015-10-11 17:46:12 -07:00
Ilya Kreymer
31912b3bf7
proxy: update tests for new use_banner, use_client_rewrite options, #107
2015-09-09 13:22:32 -07:00
Ilya Kreymer
e1a9334a54
tests: update test to match cdx-convert
2015-08-25 23:06:00 +03:00
Ilya Kreymer
63c6efc851
autocolls test: patch wsgiref not waitress as it is default
2015-07-31 09:26:48 -07:00
Ilya Kreymer
f2a2c86552
tests: proxy check to ensure content-length header is always present in proxy mode
2015-07-30 11:06:44 -07:00
Ilya Kreymer
c2f99d6cfd
replay/memento: always include 'Content-Location' for in no-redir mode replay (not just for memento timegate), #122
2015-07-19 00:11:25 -07:00
Ilya Kreymer
66f5ad62b3
memento: when redir_to_exact
is false, don't redirect latest replay/timegate to current timestamp, but return directly latest capture.
...
when memento enabled, the timegate now follows memento pattern 2.2 (http://tools.ietf.org/html/rfc7089#section-4.2.2 )
also return content-location instead of location, update memento no-redirect tests to match new behavior. closes #122
2015-07-18 23:30:31 -07:00
Ilya Kreymer
080587516b
youtube-dl tests: use mock youtube-dl info for tests
2015-06-27 20:46:55 -07:00
Ilya Kreymer
f0359877f0
youtube-dl: remove from dependency, installation is optional. Return 404 if attempting live
...
proxy of videos and youtube-dl is not available (the only use case).
HTTPParser wrapping logic no longer needed in latest versions
Modify tests to only run if youtube-dl is installed in cases where it is not available #118
2015-06-27 16:11:59 -07:00
Ilya Kreymer
06fcc89de6
readers: support 'content-encoding: deflate' using different zlib decompression options
...
support default and alt settings for attempting to decompress deflate stream
tests: add tests with httpbin.org/deflate Fixes #115
2015-06-24 13:11:33 -07:00
Ilya Kreymer
7bf8b97cb0
tests: add tests for root collection access, and also a custom handler passed to pywb_init
...
(a simple redirect handler)
2015-04-17 11:48:50 -07:00
Ilya Kreymer
307809bbe9
live-rewrite-server: switch to 'inverse' frame mode by default,
...
switch from /rewrite/ to /live/ path, update tests
2015-04-13 13:00:06 -07:00
Ilya Kreymer
f9bd2ba55a
jinja template: use shared template in J2Template, init on first use
2015-04-03 10:43:39 -07:00
Ilya Kreymer
4a85869427
cli refactor: use classes in cli to allow custom options
...
get rid of custom init for live_rewrite_handler, just use create_wb_router()
with custom config for consistent init
2015-04-03 10:43:39 -07:00
Ilya Kreymer
a34607764e
manager: validate name on collection init: must start with wordchar and can contain wordchar or -
2015-04-03 01:18:35 -07:00
Ilya Kreymer
8bd6787595
'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay
...
add additional test for inverse framed replay, #92
fix framed replay url replace slash
2015-04-01 16:21:44 -07:00
Ilya Kreymer
dd30e3f2a7
refactor: fixes for compat with latest certauth>=1.1.0
2015-03-30 09:38:42 -07:00
Ilya Kreymer
90eee03cdb
fixes for windows:
...
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
2015-03-25 10:56:53 -07:00
Ilya Kreymer
ec7a29a3ba
static paths: ensure consistent renaming of static/default -> static/__pywb for bundled static path
2015-03-23 16:15:37 -07:00