backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-16 00:24:48 +01:00
Code Issues Projects Releases Activity
1,002 Commits 56 Branches 63 Tags
b7e27ba1a8
Commit Graph

1002 Commits

Author SHA1 Message Date
Ilya Kreymer
307809bbe9 live-rewrite-server: switch to 'inverse' frame mode by default, 
switch from /rewrite/ to /live/ path, update tests
 2015-04-13 13:00:06 -07:00
Ilya Kreymer
756b80cac4 rewrite: remove urlrewriter optimization which uses wburl.original_url, as its 
not realiable (%-encoding, no trailing slash issues) and not really needed.. rename original_url -> _original_url, should not be used
directly.
add test for rewriting when base url has no trailing slash
 2015-04-13 12:51:38 -07:00
Ilya Kreymer
9f1ae42d1c rewrite: security: by default, disable file:// and local access unless explicitly allowed! 
enable for tests with remote_only=False param
 2015-04-13 09:51:12 -07:00
Ilya Kreymer
0aae4264f7 pywb_init: use custom route class for cdx api as well as main handler 2015-04-13 09:48:47 -07:00
Ilya Kreymer
5aee4a193b cdx fuzzy match: fix when fuzzy replace string is >1 char, keep full replace string (to be examined further) 2015-04-13 09:45:02 -07:00
Ilya Kreymer
b21e288063 wburl to_uri: when parsing scheme, also for first '?' in addition to '/' to catch any irregular urls (to be ignored) 2015-04-11 10:21:52 -07:00
Ilya Kreymer
d39f26872d error handling: pass in wbrequest to not_found error handler 2015-04-11 00:25:59 -07:00
Ilya Kreymer
4469754a5a routing: improved support for root collection via empty route. If '' route present, 
add it last to avoid conflicting w/ other routes ()
templates: pass in custom params to jinja2 template via env 'pywb.template_params' dictionary.
If present, dictionary contents will be added directly to jinja context for the request ()
 2015-04-10 14:47:02 -07:00
Ilya Kreymer
4bf1dd4cad urlrewriter: simply startswith check by using tuple 2015-04-10 14:46:46 -07:00
Ilya Kreymer
8513575dea cdx: small refactor to CDXFile and RedisCDXSource to facilitate better extensions, 
move generic methods to statics, add overridable params
 2015-04-10 14:36:19 -07:00
Ilya Kreymer
ddfb48ba53 rangecache: if digest is not set, use the urlkey for cache key 2015-04-06 20:29:59 -07:00
Ilya Kreymer
088a316b44 wb.js fix: check for possible null __orig_parent 2015-04-06 20:29:15 -07:00
Ilya Kreymer
f12af3410c framed replay fix for chrome: set -webkit-transform: none in framed mode on all elements, as otherwise it messes up iframe scrolling 2015-04-06 20:14:45 -07:00
Ilya Kreymer
ac06a49d4a fixes: add correct path for frame_mod and replay_mod 
wombat: copy Date prototype to appeace 'instanceof Date' check
 2015-04-04 17:31:51 -07:00
Ilya Kreymer
1286e9da4c wburl test: better test for bad domain 2015-04-04 17:31:38 -07:00
Ilya Kreymer
97b4081d89 cdx redis: for empty, use iter instead of list for consistency 2015-04-04 12:56:15 -07:00
Ilya Kreymer
c089ba35bc proxy init: instead of using first route, find first valid route (eg. not static) 
move static paths to be checked first
 2015-04-04 12:54:32 -07:00
Ilya Kreymer
10d9a6ac9a jinja templates: add 'templates' as default lookup dir, allow specifying 
custom dir via config. when specifiying custom paths, need not use full dir
as per usual paradigm
 2015-04-04 12:53:07 -07:00
Ilya Kreymer
1844597889 wburl to_uri: catch error on idna encode with very invalid urls 2015-04-04 12:51:49 -07:00
Ilya Kreymer
f9bd2ba55a jinja template: use shared template in J2Template, init on first use 2015-04-03 10:43:39 -07:00
Ilya Kreymer
4a85869427 cli refactor: use classes in cli to allow custom options 
get rid of custom init for live_rewrite_handler, just use create_wb_router()
with custom config for consistent init
 2015-04-03 10:43:39 -07:00
Ilya Kreymer
6ba5163e72 jinja template: refactor jinja template setup, use a shared jinja env instead of a new env for every template 
can pass in an existing env via the config
 2015-04-03 10:03:12 -07:00
Ilya Kreymer
fcb6e94736 framework refactor: move rel_request_uri() call down to the routers, for easier reuse 
each router now calls ensure_rel_uri_set() to ensure that REL_REQUEST_URI field is set before
use. allows router to be called directly without setup.
add optional fallback_app to allow acting as middleware
 2015-04-03 08:45:18 -07:00
Ilya Kreymer
a34607764e manager: validate name on collection init: must start with wordchar and can contain wordchar or - 2015-04-03 01:18:35 -07:00
Ilya Kreymer
134b90eca5 bump version to 0.9.4-dev 2015-04-03 00:45:12 -07:00
Ilya Kreymer
b379d15066 changelist fixes 2015-04-01 17:07:27 -07:00
Ilya Kreymer
6064b45bd4 set version to 0.9.3 2015-04-01 17:05:29 -07:00
Ilya Kreymer
b0773ca8b8 pywb_init: ability to override DirectoryCollsLoader with custom class 2015-04-01 17:03:36 -07:00
Ilya Kreymer
7a8a0e5244 update .gitignore 2015-04-01 16:32:49 -07:00
Ilya Kreymer
4d74d65b61 update CHANGES for 0.9.3 2015-04-01 16:32:13 -07:00
Ilya Kreymer
8bd6787595 'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay 
add additional test for inverse framed replay, 
fix framed replay url replace slash
 2015-04-01 16:21:44 -07:00
Ilya Kreymer
bd21480db9 framed replay: add supporting for 'inverting' frame and replay modifiers, 
setting default mod to be top-frame and inner frame to be 'mp_' 
can enable this mode by setting framed_replay: inverse instead of true
modifiers passed to client side script via wbinfo as well
 2015-04-01 10:13:56 -07:00
Ilya Kreymer
546cd8ac3a frame redirect: only attempt redirect if in 'framed' mode (add flag to wbinfo) 
ensure both uris are decoded before comparing for top-frame redirect
 2015-04-01 09:13:55 -07:00
Ilya Kreymer
c378cb5188 rewrite: check for closed before any use of readline() (2.6 may throw if closed), 
only use readline() if line alignment needed (non-html), related to  work
 2015-04-01 07:54:17 -07:00
Ilya Kreymer
e806a33289 add unclosed script sample 2015-04-01 07:13:51 -07:00
Ilya Kreymer
8e60a6464c chunkeddatareader: read(): catch ValueError when attempting to read again in case stream is already closed 2015-03-31 23:31:49 -07:00
Ilya Kreymer
990af5ee79 rewrite: add extra test for rewriting html with <script> tag that's never closed 2015-03-31 23:30:56 -07:00
Ilya Kreymer
c137dd30b8 misc fixes: remove extra debug logging 
add --framed option to 'live-rewrite-server' cli app
 2015-03-31 23:08:56 -07:00
Ilya Kreymer
199f552f73 rewrite: if no charset specified, attempt to read first 1024 bytes and set charset in header, 
to avoid charset warning if head insert exceeds 1024 bytes ()
also encode head insert with detected charset, if possible
chunkeddatareader: add read() function to ensure read will read upto specified
length across chunks
 2015-03-31 22:38:20 -07:00
Ilya Kreymer
30ab27bb1c indexing: support indexing (and even replay of) records where target-uri is a 'urn:' identifier () 
for canonicalzation, treat urns as is, already canonical
for wburl, don't add http:// prefix if urn: prefix is present
add example-wpull warc for testing
 2015-03-30 17:23:50 -07:00
Ilya Kreymer
002fe6a338 certauth: change 'get_cert_for_host' -> 'cert_for_host' 2015-03-30 15:47:53 -07:00
Ilya Kreymer
dd30e3f2a7 refactor: fixes for compat with latest certauth>=1.1.0 2015-03-30 09:38:42 -07:00
Ilya Kreymer
cda7705075 split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth cli as 
the 'certauth' tool superceeds this functionality. ().
To use https proxy mode, 'pip install certauth' is required. (update travis config)
 2015-03-29 17:38:57 -07:00
Ilya Kreymer
273176bce5 cdx: when reading cdxj, and run into non-ascii chars in url, utf-8 encode and %-encode 2015-03-29 09:21:50 -07:00
Ilya Kreymer
fc9d659b5d loaders: switch BlockLoader to use requests instead of urliib2 2015-03-28 16:41:52 -07:00
Ilya Kreymer
f3a066f58b cdx-server query & zipnum: fixes for showNumPages query: 
- if query contained in <1 secondary index block, must read first line of cdx to determine if any matches
- if no matches, don't throw 404 exception but always return json info with 0 pages
 2015-03-28 16:15:24 -07:00
Ilya Kreymer
313a2efeac bump version to 0.9.3-dev 2015-03-28 16:12:28 -07:00
Ilya Kreymer
c3a108b169 minor readme tweaks 2015-03-27 09:31:17 -07:00
Ilya Kreymer
d2be90d4a1 test case tweak 2015-03-27 08:56:43 -07:00
Ilya Kreymer
41487dd9d4 update changelist for 0.9.2 
cdx: include match type in cdx query error
 2015-03-27 07:58:51 -07:00