Ilya Kreymer
bfaad224fa
proxy tests: add test for https proxy, using requests and wsgiref server!
...
testing one https replay and static https handling
fix bug in certauth params
2014-08-18 19:58:53 -07:00
Ilya Kreymer
01e8718640
add new test proxy config
2014-08-18 18:03:02 -07:00
Ilya Kreymer
1d1465f320
proxy tests: work on cookie selector tests
2014-08-18 13:34:47 -07:00
Ilya Kreymer
c3c7935546
Merge branch '0.5.4-work' into develop
2014-08-06 13:22:08 -07:00
Ilya Kreymer
501c942a6f
tests: add test for rel self-redirect
2014-08-06 13:19:52 -07:00
Ilya Kreymer
1cd82c1bc4
proxy: move test to seperate file
...
cert: create seperate get_wildcard_cert for clarity
2014-08-06 12:39:06 -07:00
Ilya Kreymer
6e6688beb3
rewrite/testing: add additional test for live rewrite post, invalid post
...
htmlrewrite: annotate untestable sections (unimplemented, 2.6 only exceptions)
2014-08-04 22:51:43 -07:00
Ilya Kreymer
ef8d910d01
banner: remove client side 'capture_str' formatting, just output wbinfo.timestamp,
...
allow js to format as needed, also helps with #41
update tests to only look at timestamp
2014-08-04 22:51:42 -07:00
Ilya Kreymer
8d54153326
refactoring for better extensibility:
...
remove BaseContentView, move top-frame functionality to SearchPageWbUrlHandler
remove RewriteLiveView, fold functionality into the handler
move default mod setting into RewriteContent
2014-08-04 22:51:42 -07:00
Ilya Kreymer
160182ec48
rewrite: add 'bn_' banner only rewrite
...
cleanup rewrite_content/fetch_request api to take a full wb_url
add content-length to responses whenever possible (WbResponse) and static files
bump version to 0.5.2
2014-08-04 22:51:42 -07:00
Ilya Kreymer
a2d86fa495
Merge branch 'develop' into https-proxy
2014-08-04 22:01:16 -07:00
Ilya Kreymer
e1e8f679b2
rewrite/testing: add additional test for live rewrite post, invalid post
...
htmlrewrite: annotate untestable sections (unimplemented, 2.6 only exceptions)
2014-08-04 21:59:46 -07:00
Ilya Kreymer
924f71a4cc
Merge branch 'develop' into https-proxy
2014-08-04 18:44:01 -07:00
Ilya Kreymer
86bc2f17ba
banner: remove client side 'capture_str' formatting, just output wbinfo.timestamp,
...
allow js to format as needed, also helps with #41
update tests to only look at timestamp
2014-08-04 18:19:28 -07:00
Ilya Kreymer
492aaa4a01
Merge branch 'develop' into https-proxy
2014-08-04 13:00:25 -07:00
Ilya Kreymer
95028ab692
refactoring for better extensibility:
...
remove BaseContentView, move top-frame functionality to SearchPageWbUrlHandler
remove RewriteLiveView, fold functionality into the handler
move default mod setting into RewriteContent
2014-08-04 01:18:46 -07:00
Ilya Kreymer
2ca4757599
fix integration test for proxy_pac
2014-07-31 18:03:18 -07:00
Ilya Kreymer
b92eda77f6
rewrite: add 'bn_' banner only rewrite
...
cleanup rewrite_content/fetch_request api to take a full wb_url
add content-length to responses whenever possible (WbResponse) and static files
bump version to 0.5.2
2014-07-29 12:20:22 -07:00
Ilya Kreymer
0b8a8f0ae2
live rewrite: catch errors from live rewrite and raise a new LiveResourceError with a 400 error code,
...
indicating bad request for live resource. Add test for invalid live rewrite requests
2014-07-21 22:43:34 -07:00
Ilya Kreymer
7c57345363
proxy: add 'unaltered_replay' option to proxy_options to replay
...
all content unaltered (no rewriting html, no banner, no wombat)
use 'proxy_options' instead of 'routing_options', add additional
tests for proxy mode
2014-07-21 16:42:14 -07:00
Ilya Kreymer
e4297ddabe
tests: add integration tests for $liveweb rewrite handler and replay
...
with fallback
2014-07-20 18:25:47 -07:00
Ilya Kreymer
b785cd6f08
memento: use mp_ modifier to support memento with frame or non-frame replay
...
change memento test to use frame replay
2014-07-20 15:43:39 -07:00
Ilya Kreymer
96fcaab521
live-rewrite-server: add ability to specify http/https proxy for live fetching
...
(for example, for use with a recording proxy)
2014-07-19 14:43:28 -07:00
Ilya Kreymer
1317b2b10f
route selection via proxy auth!
...
refactor poute request parsing to happen in the actual router class instead of in the route
in proxy mode, add support for picking a route via proxy-auth
improve test for 'top' rewriting
2014-07-10 21:54:23 -07:00
Ilya Kreymer
70b7e29b36
pass raw bytes to htmlparser, assuming ascii-compatibility
...
(todo: add tests for non-ascii compatible encodings)
improved rendering of certain pages, needs more testing
lxml: remove lxml and complexity associated with having the parser,
as its too unpredictable for older html, does its own decoding.
2014-06-27 19:03:06 -07:00
Ilya Kreymer
fb07775d38
tests: add 'bad.cdx' for testing cdx lines with missing original for revisit,
...
missing/non-existant warc
2014-06-25 12:32:57 -07:00
Ilya Kreymer
913a1e9f31
warc: simplify recordloader a bit more, only response and request records
...
get parsed as http (excluding dns: and whois: uris)
All others have an '-' status and no headers parsing
tests: add test for zero-length revisits
2014-06-25 12:11:26 -07:00
Ilya Kreymer
073f1e142e
test_config: test lxml parser still
2014-06-14 21:33:08 -07:00
Ilya Kreymer
80e80e97d3
replay: support 'framed_replay' option in config for both replay and live rewrite
...
split replay view into BaseContentView and ReplayView
refactor RewriteLiveHandler into RewriteLiveView
add additional tests for framed and non-framed mode
default to framed replay!
2014-06-14 18:26:19 -07:00
Ilya Kreymer
0d3f663ef1
rewrite: disable refer-redirect in case of POST, handle request w/o redirect
...
(can't use 307 because of FF)
2014-06-13 16:23:11 -07:00
Ilya Kreymer
41e1809039
update wombat.js (support for write override, fill in WB_wombat_location on new iframe)
...
disable 307 redirects as FF always displays modal confirmation for these, even for same host
2014-06-11 20:12:05 -07:00
Ilya Kreymer
0c9d88f032
POST replay: treat POST form data same as get query, no '&&&' marker
...
additional testing POST
2014-06-11 11:17:06 -07:00
Ilya Kreymer
e2349a74e2
replay: better POST support via post query append!
...
record_loader can optionally parse 'request' records
archiveindexer has -a flag to write all records ('request' included),
-p flag to append post query
post-test.warc.gz and cdx
POST redirects using 307
2014-06-10 19:21:46 -07:00
Ilya Kreymer
f9710d033c
fix integration test for 307
...
update head_insert for new wombat
remove redundant host jinja func, use 'urlsplit' instead
2014-05-30 11:17:12 -07:00
Ilya Kreymer
923421d637
rewrite_content: add a few tests for cs_, js_, remove redundant except
2014-05-16 22:43:53 -07:00
Ilya Kreymer
2600d870d7
improved test: dsrules remove redundant check
...
static: check invalid static paths and file_wrapper
memento: check non-memento paths
test debug handlers and custom '-cdx' suffix
2014-05-16 22:17:51 -07:00
Ilya Kreymer
ca33287051
test: move non-surt-cdx sample to non-surt-cdx/ dir for clarity / avoid confusion
...
when bulk loading cdx/ dir (surt and non-surt cdx should NOT be mixed)
2014-05-16 21:21:14 -07:00
Ilya Kreymer
7d236af7d7
cdx: fix creation and add test for non-surt cdx (pywb-nonsurt/ test)
...
archiveindexer: -u option to generate non-surt cdx
tests: full test coverage for cdxdomainspecific (fuzzy and custom canon)
2014-05-16 21:16:50 -07:00
Ilya Kreymer
85593696fa
remove rfc3987 validation, was rejecting valid urls
...
add extract_referer_wburl_str() to extract WbUrl str, if any,
from the referrer. Use that for live_rewrite_handler to override
default referrer
2014-04-15 16:38:53 -07:00
Ilya Kreymer
bfc2e63793
live rewriter: integrate handler with rewrite_live.py module,
...
clean up css, add unit and integration tests
clean up cli server now known as 'live-rewrite-server', which performs live rewrite using
iframe paradigm
2014-04-09 15:49:55 -07:00
Ilya Kreymer
b0b0adb043
refactor: rename pywb.core -> pywb.webapp
...
move perms/test/test_perms_policy -> tests/perms_fixture
for rules file, use single DEFAULT_RULES_FILE import
2014-04-04 10:09:26 -07:00
Ilya Kreymer
80f2da9548
refactor: move configs/config.yaml to root again
...
remove cdx-server specific config, instead make cdx server api-only
path configurable from regular config
2014-04-02 21:26:53 -07:00
Ilya Kreymer
91184426b7
test coverage pass:
...
refactor and cleanup to improve coverage for corner cases
2014-04-02 13:16:54 -07:00
Ilya Kreymer
41d51a6427
ensure 'cdx_' modifier is working
2014-03-27 14:46:59 -07:00
Ilya Kreymer
c6c9fe680a
memento: add original link to timemap #10
2014-03-24 14:57:41 -07:00
Ilya Kreymer
2a605652c6
add memento timemap support (for archival mode only)
...
add timemap Link headers to timegate and memento responses
timemap accessible via /timemap/*/ path
2014-03-24 14:00:06 -07:00
Ilya Kreymer
79da12348f
limit stream by warc/arc record length instead of
...
http content length.
track length of StatusAndHeaders also.
add tests to verify content length correct for identity
arc and arcgz replays as well
2014-03-22 11:30:51 -07:00
Ilya Kreymer
6461af030b
refactoring: clean up handlers and replay_views for pep8
...
use BlockLoader().load for StaticHandler static file resolving
update static paths to point to pywb/static instead of static
2014-03-14 18:17:22 -07:00
Ilya Kreymer
bdcda1df6f
add test config for memento #10
2014-03-14 11:01:47 -07:00
Ilya Kreymer
a1ab54c340
first pass at memento support #10 !
...
memento support enabled by default, togglable via 'enable_memento' config property
supporting timegate and memento apis, no timemap yet
supporting pattern 2.3 for archival and pattern 1.3 for proxy modes
also:
simplify exception hierarchy a bit more, move down to utils
make WbRequest and WbResponse extensible with mixins (eg for memento)
2014-03-14 10:46:20 -07:00