backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-20 18:59:11 +01:00
Code Issues Projects Releases Activity
505 Commits 56 Branches 63 Tags
Commit Graph

68 Commits

Author SHA1 Message Date
Ilya Kreymer
c7228bf887 test coverage: proxy certauth: add unit tests for certauth cert creation (though not verifying validity yet) 
add https proxy post test
 2014-09-06 13:31:10 -07:00
Ilya Kreymer
eaaefbfd24 * config cleanup: remove 'hostpaths' setting entirely, avoiding the need to specify host on which pywb 
will run (this was cumbersome to maintain and not really useful)
ReferRedirect just checks that the current request host header, if present, matches that of the referrer
and checks that the coll and script name match.
* removed proxy_pac as it was also unneeded/unused and required use of the hostpaths
* added test for invalid CONNECT usage (405 response)
 2014-08-20 02:02:47 -04:00
Ilya Kreymer
6b476d83de proxy: extensive https and cookie resolver testing 
move extract_cookie utility to wbrequest
fix head_insert 'wbinfo.proxy_magic' entry
 2014-08-19 23:25:43 -04:00
Ilya Kreymer
5381c00c2a proxy: remove ciphers param to support 2.6 2014-08-19 03:30:08 -04:00
Ilya Kreymer
7ab398eba5 proxy tests: test download page 2014-08-19 03:15:42 -04:00
Ilya Kreymer
822a1c03e4 proxy tests: verify ssl, add a no-wildcard option (issues with requests and wildcard cert) 2014-08-18 20:57:21 -07:00
Ilya Kreymer
bfaad224fa proxy tests: add test for https proxy, using requests and wsgiref server! 
testing one https replay and static https handling
fix bug in certauth params
 2014-08-18 19:58:53 -07:00
Ilya Kreymer
ba1e276e2f misc fixes: ensure buffered response is an iterator (no need to explicitly check, check doesn't work in jython) 
query_handler: include check for '-' status code for revisits
 2014-08-15 14:23:25 -07:00
Ilya Kreymer
1cd82c1bc4 proxy: move test to seperate file 
cert: create seperate get_wildcard_cert for clarity
 2014-08-06 12:39:06 -07:00
Ilya Kreymer
a2d86fa495 Merge branch 'develop' into https-proxy 2014-08-04 22:01:16 -07:00
Ilya Kreymer
e1e8f679b2 rewrite/testing: add additional test for live rewrite post, invalid post 
htmlrewrite: annotate untestable sections (unimplemented, 2.6 only exceptions)
 2014-08-04 21:59:46 -07:00
Ilya Kreymer
92726309fc proxy: add 'extra_headers' to be added to proxy responses, customizable via proxy_options 
defaults include no-cache and p3p policy (needed for IE default settings)
fix link generation for proxy_select page, better exception handling of ssl errors
 2014-08-02 04:27:51 -07:00
Ilya Kreymer
4efd2d514c proxy: add proxy_cert download page at root http://pywb.proxy, serving .pem and .p12 (windows only) certs for auto installation 2014-08-01 17:15:49 -07:00
Ilya Kreymer
aeb246466b proxy: SSL version is 0-based not 1-based, set_version(2) for version 3! 2014-08-01 12:35:19 -07:00
Ilya Kreymer
f5c27d7b06 rewrite: fix header rewrite test 
proxy_pac: use http host header if available for proxy host
 2014-07-31 17:33:43 -07:00
Ilya Kreymer
407da7528b proxy/rewrite: don't rewrite headers banner_only 2014-07-31 17:02:26 -07:00
Ilya Kreymer
cfe11a5ad3 fix typo param ordering 2014-07-31 11:56:43 -07:00
Ilya Kreymer
522ea87637 proxy: timestamp selection support! 
certauth: wildcard support, use *.host wildcard for proxy certs whenever possible
ui: add coll info/switch and calendar links to banner
 2014-07-31 11:12:50 -07:00
Ilya Kreymer
eff5a74ec7 proxy: path fixup, using default magic name of 'pywb.proxy', default 
selector 'select.pywb.proxy' stored in env['pywb_select'] and accessible
to head insert and error handler
 2014-07-30 13:06:18 -07:00
Ilya Kreymer
527ad0d60e proxy mode: make banner replay and cookie resolver configurable via proxy_options 2014-07-30 10:47:24 -07:00
Ilya Kreymer
96d9f4dcad proxy mode: cookie based selector using session to coll 
ui: add proxy_selector html, add switch link to error and banner
 2014-07-30 10:38:13 -07:00
Ilya Kreymer
607ea1ccf0 proxy resolver: cookie resolver uses session cookies 
proxy static handler: handled via proxy to support http/https
use 'pywb.proxy' prefix for custom env settings
 2014-07-29 12:23:41 -07:00
Ilya Kreymer
b92eda77f6 rewrite: add 'bn_' banner only rewrite 
cleanup rewrite_content/fetch_request api to take a full wb_url
add content-length to responses whenever possible (WbResponse) and static files
bump version to 0.5.2
 2014-07-29 12:20:22 -07:00
Ilya Kreymer
9c96026904 proxy-cert-auth: add cli hook for 'proxy-cert-auth' for creating root certs, tweak help 2014-07-28 16:06:01 -07:00
Ilya Kreymer
ba61f23e40 proxy_resolvers: move resolvers to seperate file, default to ProxyAuthResolver (CookieResolver still work-in-progress) 2014-07-28 15:22:22 -07:00
Ilya Kreymer
6234d795dc proxy improvements: refactor coll selector into BaseCollSelector, 
supporting either proxy auth or cookie-based selection (in progress)
https proxy: support POST requests, properly read http header and wrap remainder
in wsgi.input
https proxy: properly update wsgi for wrapped request
wbrequestresponse: add content-length 0 to redir_response
 2014-07-28 14:28:00 -07:00
Ilya Kreymer
b6fb0e510e certauth: clean up CertificatAuthority, add cli interface for creating root cert and host certs 
CertificateAuthority instance creates per-host certs, assume root cert exists
static method generate_ca_root() used to create root cert once
add proxy_options to enable https support
 2014-07-27 19:35:16 -07:00
Ilya Kreymer
2a9197137e certauth: some cleanup for pep8, 2.6 compat 2014-07-26 21:06:28 -07:00
Ilya Kreymer
ae35d92dde fix typo in certauth 2014-07-26 15:27:02 -07:00
Ilya Kreymer
5beb831ae9 wbrequestresponse: update doctest 2014-07-26 14:27:31 -07:00
Ilya Kreymer
1464e89c41 wbresponse: always include Content-Length for text_response 2014-07-26 14:24:28 -07:00
Ilya Kreymer
eca3cf5fbf https proxy: add ca generator! 
support uwsgi, gunicorn and ref
better handling of 407, other error responses in response to CONNECT
 2014-07-26 13:24:53 -07:00
Ilya Kreymer
739f23da9e https proxy support, CONNECT verb handling (uwsgi only) 2014-07-26 09:48:44 -07:00
Ilya Kreymer
4d31c17d4c archivalrouter: make SCRIPT_NAME key lookup default to '', addresses #39 2014-07-23 15:31:30 -07:00
Ilya Kreymer
7c57345363 proxy: add 'unaltered_replay' option to proxy_options to replay 
all content unaltered (no rewriting html, no banner, no wombat)
use 'proxy_options' instead of 'routing_options', add additional
tests for proxy mode
 2014-07-21 16:42:14 -07:00
Ilya Kreymer
a2973b04e7 wbrequest: add options dictionary to store misc request options 2014-07-21 14:02:31 -07:00
Ilya Kreymer
fa813bdd19 pep8 cleanup pass 2014-07-20 18:26:16 -07:00
Ilya Kreymer
b785cd6f08 memento: use mp_ modifier to support memento with frame or non-frame replay 
change memento test to use frame replay
 2014-07-20 15:43:39 -07:00
Ilya Kreymer
96fcaab521 live-rewrite-server: add ability to specify http/https proxy for live fetching 
(for example, for use with a recording proxy)
 2014-07-19 14:43:28 -07:00
Ilya Kreymer
1b1a1f8115 proxy: add 'proxy_coll_select' config which will require a proxy-auth to select a collection for proxy mode. 
Otherwise, defaults to first available collection, though proxy-auth can still be sent to specify different collection
 2014-07-14 19:12:30 -07:00
Ilya Kreymer
1317b2b10f route selection via proxy auth! 
refactor poute request parsing to happen in the actual router class instead of in the route
in proxy mode, add support for picking a route via proxy-auth
improve test for 'top' rewriting
 2014-07-10 21:54:23 -07:00
Ilya Kreymer
88d3e94b36 fixes for pep8, name fixes 2014-06-15 11:57:48 -07:00
Ilya Kreymer
0d3f663ef1 rewrite: disable refer-redirect in case of POST, handle request w/o redirect 
(can't use 307 because of FF)
 2014-06-13 16:23:11 -07:00
Ilya Kreymer
e2349a74e2 replay: better POST support via post query append! 
record_loader can optionally parse 'request' records
archiveindexer has -a flag to write all records ('request' included),
-p flag to append post query
post-test.warc.gz and cdx
POST redirects using 307
 2014-06-10 19:21:46 -07:00
Ilya Kreymer
52040127b3 update wombat.js to latest 
rewrite live: add another rewrite live header,
use 307 for archival referer based redirects
 2014-05-30 11:03:22 -07:00
Ilya Kreymer
89da165467 exceptions: add optional url param to WbException, move handler_exception() 
into WSGIApp for easier customization
 2014-05-13 01:54:12 -07:00
Ilya Kreymer
53f0cb540f url rewriter: add optional 'full prefix', check and don't rewrite urls 
if starting with prefix or full prefix
wbrequest: if no scheme present (shouldn't happen with wsgi) default to http
 2014-04-24 10:44:08 -07:00
Ilya Kreymer
85593696fa remove rfc3987 validation, was rejecting valid urls 
add extract_referer_wburl_str() to extract WbUrl str, if any,
from the referrer. Use that for live_rewrite_handler to override
default referrer
 2014-04-15 16:38:53 -07:00
Ilya Kreymer
19f2df4717 refactor: 
- move is_identity(), is_embed() to wburl from wbrequest
- add is_mainpage() predicate
- add create_template() to each J2TemplateView to create itself
- add HeadInsertView to create a reusable head insert for
RewriteContent
- add 'mp_' as modifier for frames mode to be used as possible
  modifier with HTMLRewriter
 2014-04-09 15:49:55 -07:00
Ilya Kreymer
80f2da9548 refactor: move configs/config.yaml to root again 
remove cdx-server specific config, instead make cdx server api-only
path configurable from regular config
 2014-04-02 21:26:53 -07:00