1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-04-02 12:06:14 +02:00

140 Commits

Author SHA1 Message Date
Ilya Kreymer
0f2c96879c refactor: split out optional cached replay components into cached_replay,
toggleable via 'enable_cache' in config -- regular replayview does not
need any cache info
move add_range() components to statusandheaders from wbrequestresponse
add x-pywb-noredirect' header which disables date related redirect
video replay works w/o cache if supported by frontend (nginx)
2014-12-19 18:40:45 -08:00
Ilya Kreymer
07e46f4b6c request/response: add range extraction and settings functions to
wbrequest/response (with tests)
2014-12-18 13:19:38 -08:00
Ilya Kreymer
7e36ad29e7 Merge branch 'develop' 0.6.6 into video 2014-12-06 19:19:12 -08:00
Ilya Kreymer
0495423e86 rewrite: add per-collection rewrite options, settable in 'rewrite_opts'
block in each collection. Added rewrite_base to disable rewriting <base>
tag and rewrite_rel_canon to disable rewriting link rel=canon.

Disabling <base> tag rewrite fixex #51 and new system addresses #50 as
well.
2014-12-06 17:16:35 -08:00
Ilya Kreymer
f21f4fb1ba Merge branch 'develop' into video 2014-12-01 09:10:08 -08:00
Ilya Kreymer
c996e70a6e wburl: detect and decode partially encoded schemes in url, such as http%3A//,
https%A2F2F// before handling further
add additional tests for wburl
2014-11-29 11:13:57 -08:00
Ilya Kreymer
1aac5a9f15 cache: move cache wrappers to seperate cache.py in framework from
proxy_resolvers
range cache: and buffering cache for serving range requests, intended
for videos but not only. full response cached in temp file and range
requests served from cache, still experimental
need to add deletion.
youtube_dl: wrap youtube-dl import due to youtube-dl HTMLParser regex
bug
tests: add test for vi_ handler
2014-11-01 15:41:01 -07:00
Ilya Kreymer
c9273ee5ed rewrite: add 'deprefix' support to remove wburl prefix from any query
params
2014-10-26 12:12:37 -07:00
Ilya Kreymer
037cf35eb8 wsgi_wrapper: check for str before decoding err msg 2014-10-25 11:42:44 -07:00
Ilya Kreymer
e8d3965269 pep8 style fixes, remove unused methods 2014-10-21 19:06:16 -07:00
Ilya Kreymer
d99f7f996c urlrewriter refactor: replace get_abs_url and get_timestamp_url with
get_new_ur() which just calls wburl.to_str and applies rewriter prefix
allows creating a new wburl with any component(s) changed
2014-10-19 00:24:00 -07:00
Ilya Kreymer
4a1cc46fa3 framed replay: invert framed replay paradigm, replay always uses
canonical, no-modifier archival url (instead of mp_).
When using frames, the page redirects to a 'tf_' page, which then uses
replaceHistory() to change url back to canonical form.
memento: support for framed replay, include memento headers in top frame
bump version to 0.6.2
2014-10-18 11:21:07 -07:00
Ilya Kreymer
50bf7d2634 rewrite: move extract_client_cookie to utils for access at rewrite
root cookie_rewriter: keep max-age
add csrf token copying (experimental)
update tests
2014-10-12 03:07:54 -07:00
Ilya Kreymer
498a864441 rewriting: support setting cookie_scope at collection level
js rewriting: add custom url rewrite option to per-url rewrite rules
2014-10-06 10:14:45 -07:00
Ilya Kreymer
f1b3f8c76f cookie rewriter work: ability to set a custom 'root scope' rewriter,
which sets the path of all cookies to pywb root.
Option to enable per url-prefix in rules, still more testing, other
options needed
2014-09-30 12:42:11 -07:00
Ilya Kreymer
7ac98fbfe2 cookie rewriter: use relative path for cookie path rewriting, pass
relative path to urlrewriter
rules: add more rules
2014-09-21 13:23:19 -07:00
Ilya Kreymer
5a536d75e4 proxy tests: refactor tests and add cookie resolver http-only test
rename tests to test_proxy_https_cookie, test_proxy_http_cookie, test_proxy_http_auth
to allow for testing, http proxy, cookie resolver, proxy auth and https support in a more
orthogonal way
rename configs to test_config_proxy_http and test_config_proxy_https
2014-09-07 11:02:43 -07:00
Ilya Kreymer
aeb247939d errors: ensure err msgs are properly encoded for jinja2 or raw text response 2014-09-07 01:07:58 -07:00
Ilya Kreymer
45628f7963 wsgi_wrapper: ensure error handling doesn't throw decoding errors 2014-09-06 18:35:30 -07:00
Ilya Kreymer
751084b097 update CHANGES, config.yaml docs for proxy mode
ensure proxy_options match defaults in config.yaml
default cookie_resolver to true
2014-09-06 17:03:04 -07:00
Ilya Kreymer
da7e6f31ac tests: pep8 and coverage pass, getting ready for release 2014-09-06 15:19:28 -07:00
Ilya Kreymer
f82afbc977 openssl: make openssl install optional, remove from dependency
https proxy support contingest on openssl installation
if not installed and https use is enabled, a warning is printed
tests: updated tests to skip if openssl not installed
(make https post to only run in 2.7, as there are issues with in 2.6)
2014-09-06 14:43:48 -07:00
Ilya Kreymer
c7228bf887 test coverage: proxy certauth: add unit tests for certauth cert creation (though not verifying validity yet)
add https proxy post test
2014-09-06 13:31:10 -07:00
Ilya Kreymer
eaaefbfd24 * config cleanup: remove 'hostpaths' setting entirely, avoiding the need to specify host on which pywb
will run (this was cumbersome to maintain and not really useful)
ReferRedirect just checks that the current request host header, if present, matches that of the referrer
and checks that the coll and script name match.
* removed proxy_pac as it was also unneeded/unused and required use of the hostpaths
* added test for invalid CONNECT usage (405 response)
2014-08-20 02:02:47 -04:00
Ilya Kreymer
6b476d83de proxy: extensive https and cookie resolver testing
move extract_cookie utility to wbrequest
fix head_insert 'wbinfo.proxy_magic' entry
2014-08-19 23:25:43 -04:00
Ilya Kreymer
5381c00c2a proxy: remove ciphers param to support 2.6 2014-08-19 03:30:08 -04:00
Ilya Kreymer
7ab398eba5 proxy tests: test download page 2014-08-19 03:15:42 -04:00
Ilya Kreymer
822a1c03e4 proxy tests: verify ssl, add a no-wildcard option (issues with requests and wildcard cert) 2014-08-18 20:57:21 -07:00
Ilya Kreymer
bfaad224fa proxy tests: add test for https proxy, using requests and wsgiref server!
testing one https replay and static https handling
fix bug in certauth params
2014-08-18 19:58:53 -07:00
Ilya Kreymer
ba1e276e2f misc fixes: ensure buffered response is an iterator (no need to explicitly check, check doesn't work in jython)
query_handler: include check for '-' status code for revisits
2014-08-15 14:23:25 -07:00
Ilya Kreymer
1cd82c1bc4 proxy: move test to seperate file
cert: create seperate get_wildcard_cert for clarity
2014-08-06 12:39:06 -07:00
Ilya Kreymer
a2d86fa495 Merge branch 'develop' into https-proxy 2014-08-04 22:01:16 -07:00
Ilya Kreymer
e1e8f679b2 rewrite/testing: add additional test for live rewrite post, invalid post
htmlrewrite: annotate untestable sections (unimplemented, 2.6 only exceptions)
2014-08-04 21:59:46 -07:00
Ilya Kreymer
92726309fc proxy: add 'extra_headers' to be added to proxy responses, customizable via proxy_options
defaults include no-cache and p3p policy (needed for IE default settings)
fix link generation for proxy_select page, better exception handling of ssl errors
2014-08-02 04:27:51 -07:00
Ilya Kreymer
4efd2d514c proxy: add proxy_cert download page at root http://pywb.proxy, serving .pem and .p12 (windows only) certs for auto installation 2014-08-01 17:15:49 -07:00
Ilya Kreymer
aeb246466b proxy: SSL version is 0-based not 1-based, set_version(2) for version 3! 2014-08-01 12:35:19 -07:00
Ilya Kreymer
f5c27d7b06 rewrite: fix header rewrite test
proxy_pac: use http host header if available for proxy host
2014-07-31 17:33:43 -07:00
Ilya Kreymer
407da7528b proxy/rewrite: don't rewrite headers banner_only 2014-07-31 17:02:26 -07:00
Ilya Kreymer
cfe11a5ad3 fix typo param ordering 2014-07-31 11:56:43 -07:00
Ilya Kreymer
522ea87637 proxy: timestamp selection support!
certauth: wildcard support, use *.host wildcard for proxy certs whenever possible
ui: add coll info/switch and calendar links to banner
2014-07-31 11:12:50 -07:00
Ilya Kreymer
eff5a74ec7 proxy: path fixup, using default magic name of 'pywb.proxy', default
selector 'select.pywb.proxy' stored in env['pywb_select'] and accessible
to head insert and error handler
2014-07-30 13:06:18 -07:00
Ilya Kreymer
527ad0d60e proxy mode: make banner replay and cookie resolver configurable via proxy_options 2014-07-30 10:47:24 -07:00
Ilya Kreymer
96d9f4dcad proxy mode: cookie based selector using session to coll
ui: add proxy_selector html, add switch link to error and banner
2014-07-30 10:38:13 -07:00
Ilya Kreymer
607ea1ccf0 proxy resolver: cookie resolver uses session cookies
proxy static handler: handled via proxy to support http/https
use 'pywb.proxy' prefix for custom env settings
2014-07-29 12:23:41 -07:00
Ilya Kreymer
b92eda77f6 rewrite: add 'bn_' banner only rewrite
cleanup rewrite_content/fetch_request api to take a full wb_url
add content-length to responses whenever possible (WbResponse) and static files
bump version to 0.5.2
2014-07-29 12:20:22 -07:00
Ilya Kreymer
9c96026904 proxy-cert-auth: add cli hook for 'proxy-cert-auth' for creating root certs, tweak help 2014-07-28 16:06:01 -07:00
Ilya Kreymer
ba61f23e40 proxy_resolvers: move resolvers to seperate file, default to ProxyAuthResolver (CookieResolver still work-in-progress) 2014-07-28 15:22:22 -07:00
Ilya Kreymer
6234d795dc proxy improvements: refactor coll selector into BaseCollSelector,
supporting either proxy auth or cookie-based selection (in progress)
https proxy: support POST requests, properly read http header and wrap remainder
in wsgi.input
https proxy: properly update wsgi for wrapped request
wbrequestresponse: add content-length 0 to redir_response
2014-07-28 14:28:00 -07:00
Ilya Kreymer
b6fb0e510e certauth: clean up CertificatAuthority, add cli interface for creating root cert and host certs
CertificateAuthority instance creates per-host certs, assume root cert exists
static method generate_ca_root() used to create root cert once
add proxy_options to enable https support
2014-07-27 19:35:16 -07:00
Ilya Kreymer
2a9197137e certauth: some cleanup for pep8, 2.6 compat 2014-07-26 21:06:28 -07:00