Ilya Kreymer
1d49a9fd3b
tests: improved tests for loaders module
2015-04-17 11:02:57 -07:00
Ilya Kreymer
52a7dd87c6
loaders: s3: import boto just once, store s3_avail flag
2015-04-17 11:02:57 -07:00
Ilya Kreymer
c35fd86c76
add boto to .travis.yml for testing s3
2015-04-17 11:02:57 -07:00
Ilya Kreymer
c8a9a3ddd4
loaders: add support for loading from s3:// using boto
...
if auth connection fails, attempt anon connection, #97
2015-04-17 11:02:57 -07:00
Ilya Kreymer
4db661a09e
update README and CHANGES for 0.9.5!
2015-04-17 11:01:54 -07:00
Ilya Kreymer
74c6b60d5e
wombat customization: pass custom options from config.yaml 'rewrite_opts.client' as a json obj
...
to wombat.js #96
currently supporting no_rewrite_prefixes, and skipping dom, setAttribute and postmessage overrides
(used by via.hypothes.is) -- other options to be added later
2015-04-16 12:24:01 -07:00
Ilya Kreymer
2e4427100c
wombat init refactor: init wbinfo first, then pass wbinfo to wombat
...
to allow for additional options for wombat, in prep for #96
2015-04-16 11:51:25 -07:00
Ilya Kreymer
fbb9de264a
rewrite: cookie: add 'removeall' cookie rewriter which removes all the cookies!
2015-04-14 18:04:51 -07:00
Ilya Kreymer
dcc2139fc8
fuzzy: add fuzzy match for vine
2015-04-13 13:02:55 -07:00
Ilya Kreymer
307809bbe9
live-rewrite-server: switch to 'inverse' frame mode by default,
...
switch from /rewrite/ to /live/ path, update tests
2015-04-13 13:00:06 -07:00
Ilya Kreymer
756b80cac4
rewrite: remove urlrewriter optimization which uses wburl.original_url, as its
...
not realiable (%-encoding, no trailing slash issues) and not really needed.. rename original_url -> _original_url, should not be used
directly.
add test for rewriting when base url has no trailing slash
2015-04-13 12:51:38 -07:00
Ilya Kreymer
9f1ae42d1c
rewrite: security: by default, disable file:// and local access unless explicitly allowed!
...
enable for tests with remote_only=False param
2015-04-13 09:51:12 -07:00
Ilya Kreymer
0aae4264f7
pywb_init: use custom route class for cdx api as well as main handler
2015-04-13 09:48:47 -07:00
Ilya Kreymer
5aee4a193b
cdx fuzzy match: fix when fuzzy replace string is >1 char, keep full replace string (to be examined further)
2015-04-13 09:45:02 -07:00
Ilya Kreymer
b21e288063
wburl to_uri: when parsing scheme, also for first '?' in addition to '/' to catch any irregular urls (to be ignored)
2015-04-11 10:21:52 -07:00
Ilya Kreymer
d39f26872d
error handling: pass in wbrequest to not_found error handler
2015-04-11 00:25:59 -07:00
Ilya Kreymer
4469754a5a
routing: improved support for root collection via empty route. If '' route present,
...
add it last to avoid conflicting w/ other routes (#94 )
templates: pass in custom params to jinja2 template via env 'pywb.template_params' dictionary.
If present, dictionary contents will be added directly to jinja context for the request (#95 )
2015-04-10 14:47:02 -07:00
Ilya Kreymer
4bf1dd4cad
urlrewriter: simply startswith check by using tuple
2015-04-10 14:46:46 -07:00
Ilya Kreymer
8513575dea
cdx: small refactor to CDXFile and RedisCDXSource to facilitate better extensions,
...
move generic methods to statics, add overridable params
2015-04-10 14:36:19 -07:00
Ilya Kreymer
ddfb48ba53
rangecache: if digest is not set, use the urlkey for cache key
2015-04-06 20:29:59 -07:00
Ilya Kreymer
088a316b44
wb.js fix: check for possible null __orig_parent
2015-04-06 20:29:15 -07:00
Ilya Kreymer
f12af3410c
framed replay fix for chrome: set -webkit-transform: none in framed mode on all elements, as otherwise it messes up iframe scrolling #93
2015-04-06 20:14:45 -07:00
Ilya Kreymer
ac06a49d4a
fixes: add correct path for frame_mod and replay_mod
...
wombat: copy Date prototype to appeace 'instanceof Date' check
2015-04-04 17:31:51 -07:00
Ilya Kreymer
1286e9da4c
wburl test: better test for bad domain
2015-04-04 17:31:38 -07:00
Ilya Kreymer
97b4081d89
cdx redis: for empty, use iter instead of list for consistency
2015-04-04 12:56:15 -07:00
Ilya Kreymer
c089ba35bc
proxy init: instead of using first route, find first valid route (eg. not static)
...
move static paths to be checked first
2015-04-04 12:54:32 -07:00
Ilya Kreymer
10d9a6ac9a
jinja templates: add 'templates' as default lookup dir, allow specifying
...
custom dir via config. when specifiying custom paths, need not use full dir
as per usual paradigm
2015-04-04 12:53:07 -07:00
Ilya Kreymer
1844597889
wburl to_uri: catch error on idna encode with very invalid urls
2015-04-04 12:51:49 -07:00
Ilya Kreymer
f9bd2ba55a
jinja template: use shared template in J2Template, init on first use
2015-04-03 10:43:39 -07:00
Ilya Kreymer
4a85869427
cli refactor: use classes in cli to allow custom options
...
get rid of custom init for live_rewrite_handler, just use create_wb_router()
with custom config for consistent init
2015-04-03 10:43:39 -07:00
Ilya Kreymer
6ba5163e72
jinja template: refactor jinja template setup, use a shared jinja env instead of a new env for every template
...
can pass in an existing env via the config
2015-04-03 10:03:12 -07:00
Ilya Kreymer
fcb6e94736
framework refactor: move rel_request_uri() call down to the routers, for easier reuse
...
each router now calls ensure_rel_uri_set() to ensure that REL_REQUEST_URI field is set before
use. allows router to be called directly without setup.
add optional fallback_app to allow acting as middleware
2015-04-03 08:45:18 -07:00
Ilya Kreymer
a34607764e
manager: validate name on collection init: must start with wordchar and can contain wordchar or -
2015-04-03 01:18:35 -07:00
Ilya Kreymer
134b90eca5
bump version to 0.9.4-dev
2015-04-03 00:45:12 -07:00
Ilya Kreymer
b379d15066
changelist fixes
2015-04-01 17:07:27 -07:00
Ilya Kreymer
6064b45bd4
set version to 0.9.3
2015-04-01 17:05:29 -07:00
Ilya Kreymer
b0773ca8b8
pywb_init: ability to override DirectoryCollsLoader with custom class
2015-04-01 17:03:36 -07:00
Ilya Kreymer
7a8a0e5244
update .gitignore
2015-04-01 16:32:49 -07:00
Ilya Kreymer
4d74d65b61
update CHANGES for 0.9.3
2015-04-01 16:32:13 -07:00
Ilya Kreymer
8bd6787595
'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay
...
add additional test for inverse framed replay, #92
fix framed replay url replace slash
2015-04-01 16:21:44 -07:00
Ilya Kreymer
bd21480db9
framed replay: add supporting for 'inverting' frame and replay modifiers,
...
setting default mod to be top-frame and inner frame to be 'mp_' #92
can enable this mode by setting framed_replay: inverse instead of true
modifiers passed to client side script via wbinfo as well
2015-04-01 10:13:56 -07:00
Ilya Kreymer
546cd8ac3a
frame redirect: only attempt redirect if in 'framed' mode (add flag to wbinfo)
...
ensure both uris are decoded before comparing for top-frame redirect
2015-04-01 09:13:55 -07:00
Ilya Kreymer
c378cb5188
rewrite: check for closed before any use of readline() (2.6 may throw if closed),
...
only use readline() if line alignment needed (non-html), related to #86 work
2015-04-01 07:54:17 -07:00
Ilya Kreymer
e806a33289
add unclosed script sample
2015-04-01 07:13:51 -07:00
Ilya Kreymer
8e60a6464c
chunkeddatareader: read(): catch ValueError when attempting to read again in case stream is already closed
2015-03-31 23:31:49 -07:00
Ilya Kreymer
990af5ee79
rewrite: add extra test for rewriting html with <script> tag that's never closed
2015-03-31 23:30:56 -07:00
Ilya Kreymer
c137dd30b8
misc fixes: remove extra debug logging
...
add --framed option to 'live-rewrite-server' cli app
2015-03-31 23:08:56 -07:00
Ilya Kreymer
199f552f73
rewrite: if no charset specified, attempt to read first 1024 bytes and set charset in header,
...
to avoid charset warning if head insert exceeds 1024 bytes (#86 )
also encode head insert with detected charset, if possible
chunkeddatareader: add read() function to ensure read will read upto specified
length across chunks
2015-03-31 22:38:20 -07:00
Ilya Kreymer
30ab27bb1c
indexing: support indexing (and even replay of) records where target-uri is a 'urn:' identifier ( #91 )
...
for canonicalzation, treat urns as is, already canonical
for wburl, don't add http:// prefix if urn: prefix is present
add example-wpull warc for testing
2015-03-30 17:23:50 -07:00
Ilya Kreymer
002fe6a338
certauth: change 'get_cert_for_host' -> 'cert_for_host'
2015-03-30 15:47:53 -07:00