1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

2307 Commits

Author SHA1 Message Date
Ilya Kreymer
b7bf693885
request-uri handling: use REQUEST_URI if available to maintain %-encoding when constructing WbUrl (#315)
geventserver: use custom handler to set raw 'REQUEST_URI' when running default gevent wsgi server. (uwsgi already sets REQUEST_URI)
testing: add REQUEST_URI check to proxy tests as real server is being used (webtest tests decodes %-encoding)
bump version to 2.0.4
2018-04-10 17:17:38 -07:00
Ilya Kreymer
33cca0bc02
Update CHANGES for 2.0.3 2018-04-03 19:10:08 -07:00
Ilya Kreymer
3101e567f3 config: add support for forcing a scheme for url rewriting, eg: 'force_scheme: https', fixes #314 2018-04-03 19:05:01 -07:00
Ilya Kreymer
4f58111875 update changelist for 2.0.3 2018-04-02 18:04:44 -07:00
Ilya Kreymer
c71611e6b7 cookie rewriter: don't rewrite cookies if not rewriting urls, eg. banner only or proxy mode
tests: update content rewriter tests to test for cookie rewriting
2018-04-02 17:58:23 -07:00
Ilya Kreymer
d732cdd01f aggregator timeout fixes (#310):
- fix memento aggregation if timeout is 0.0
- use default timeout (5.0), instead of default to 0.0 and failing
- add 'timeout' property to warcserver aggregation tests
- docs: mention property in warcserver docs also
2018-04-02 17:52:13 -07:00
Ilya Kreymer
8f981743ae docs: add sample nginx config to deployment section, mention how https is handled, fixes #314 2018-04-02 17:23:04 -07:00
Ilya Kreymer
f32eb608f1 tests: make redis test fix py27 compatible also 2018-03-29 22:12:07 -07:00
Ilya Kreymer
8d9951bc7b misc test fixes: make record_replay tests for consistent, use different url to ensure consistent ordering
fakeredistests: fix for fakenewredis, clear fakeredis databases and pubsub list
2018-03-29 21:43:37 -07:00
Ilya Kreymer
9da5bd1083
Decoding and Recorder Fixes (#313)
* redisindex: use decode_resposes=True for redisindex
* recorder: close_file(): return true if closed, close_key() return filename if closed
* logging: if debug=True, log warc load failures
* appveyor build fix: add pypiwin32 as dependency for windows build
2018-03-29 13:42:00 -07:00
humberthardy
a9cbdc1bd6 rewrite_amf.py: Fix bug introduced by recent refactoring (#308) 2018-03-05 13:20:37 -08:00
Ilya Kreymer
6d879cb8b8 docs: fix typos in memento docs (#307)
- URI-M instead of URL-M
- remove mention of vary: accept-datetime for URI-M
2018-03-05 13:12:12 -08:00
Ilya Kreymer
e812ed2d45 head request replay fix: treat head requests as traditionally GET requests w/o payload, instead of HEAD request replay, see #309, mentioned in #307 2018-03-05 13:10:53 -08:00
Ilya Kreymer
98cdf36626 bump version to 2.0.3 2018-03-05 13:10:53 -08:00
Ilya Kreymer
427dc3e00c update CHANGES for 2.0.2 2018-02-27 18:26:05 -08:00
Ilya Kreymer
e928f8a7e6 replay top-frame redirect: add fast redirect check to top-frame, instead of waiting for check in wombat.js, closes #305
tests: ensure redirect check only added in framed mode, ensure added for banner only mode, but not for proxy mode
2018-02-27 18:13:07 -08:00
Ilya Kreymer
84723c9d7d tests: fix video tests not running, related to #270, fix typo importorskip('youtube-dl') -> importorskip('youtube_dl') 2018-02-27 17:49:36 -08:00
Ilya Kreymer
61bf5e09ca
proxy-mode tweaks: (fixes #302): (#304)
- don't include wombat.js in banner only mode, including in proxy mode
  (instead, do set devicePixelRatio to fix certain fidelity issues)
- default_banner: set title to document.title on load when frameless, including in proxy mode
- improve docs for configuring proxy mode cert
- tests: update tests to ensure no wombat.js injected in proxy or banner-only mode
2018-02-27 15:52:19 -08:00
Ilya Kreymer
e2cbdbc27c cli: add -b/--bind cli option (defaulting to 0.0.0.0) to avoid default geventserver behavior, which doesn't bind to 0.0.0.0 and silently fails if other services are running on same port. related to issue mentioned in #298 2018-02-26 17:07:01 -08:00
Ilya Kreymer
0767bf80d5 client-side override improvements:
- override window.EventTarget.prototype.addEventListener instead of window.add/removeEventListener to work correctly with angular
- add 'document.title' override to detect title change event and propagate to top frame (history title often unused)
- add equivalent wrappers from addEventListener to window.onmessage and window.onstorage properties
2018-02-13 18:30:42 -08:00
Ilya Kreymer
7234bc51f0 client-side top-frame notification fix:
- send_top_message() to wrap all postMessage calls
- url change, hash change, history change notifications only sent if window is top replay frame, cookie notification sent for all windows
- don't send url change notification if 'about:blank' or 'javascript:' url
bump version to 2.0.2
2018-02-13 13:49:10 -08:00
Ilya Kreymer
fc48e23dae docs/README: fix typos, add changes for 2.0.1 2018-02-10 11:48:50 -08:00
Ilya Kreymer
448fb2cf1e client-side rewrite fixes:
- override Function.apply() to de-proxy thisArg and all params before calling native functions (may make per-function overrides unnecessary)
- ensure init_top_frame_notify() is called on $wbwindow object not window
2018-02-05 09:38:48 -08:00
Ilya Kreymer
728d9b3ca1 query ui: display capturse with second precision included 2018-02-05 09:36:42 -08:00
Ilya Kreymer
e2fa14bc2d tests: add 'importorskip' for tests that require 'extra' dependencies, (youtube-dl, socks), addresses #270
setup: remove 2.6 classifier, update repo path
bump to 2.0.1
2018-01-30 18:26:53 -08:00
Ilya Kreymer
610b6414a3 setup: fix for pypi build: remove obsolete 'provides=', package init from tests_disabled 2018-01-30 08:37:51 -08:00
Ilya Kreymer
8304762cf5 setup.py: exclude tests_disabled from install 2018-01-30 08:17:02 -08:00
Ilya Kreymer
008504d055
Text tweaks/Dockerfile update (#288)
README: update features list, contributing section, fix typos
docs: update features list, fix wording, add more links to other sections, fix typos
renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement
Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image
2018-01-30 07:49:54 -08:00
Ilya Kreymer
34902df80c docs work and misc:
- set depth in main toc to 3
- add info on cli apps in apps.rst
- fix typos, update links
setup: add 'pywb' cli script to be same as 'wayback'
appveyor: remove coveralls
2018-01-29 18:36:14 -08:00
Ilya Kreymer
3b72c39da4 README: update links to master 2018-01-29 18:30:54 -08:00
Ilya Kreymer
a954a5470f HEAD requests: fix pywb recording & replay of HEAD requests (force payload of 0 instead of content-length if HEAD request from live web)
tests: fix socks-proxy test to fast-fail to a random unused port to detect proxy hook is enabled
2018-01-29 16:34:25 -08:00
Ilya Kreymer
273b3eec30
warcserver/cdx query: filter improvements (#285)
- pywb.utils.format: add query_to_dict() to convert query string with support for list for certain params
- support multiple values for 'filter' cdx server param (fixes #284)
- pywb.utils.format: add to_bool() to convert string/int to bool (eg. for query args)
- fuzzymatch: add 'allowFuzzy' (default to true) to allow disabling fuzzy matcher
- tests: fuzzymather: test disabling fuzzy matcher with allowFuzzy=0
- tests: cdx-server api: add multiple filter tests, with and without fuzzy matching
2018-01-29 15:08:50 -08:00
Ilya Kreymer
cd304cc2d7 redisindexsource: clear member_key_type if invalid (not hash or set) 2018-01-29 08:10:21 -08:00
Ilya Kreymer
52ca95eba5 redis: redisindexsource and pathresolver:
- for wildcard/multi-key lookup, support redis hashmap as well as redis set to be used as member lookup key
- if using hashmap, the propery names are used for lookup
- track type of redis key in RedisIndexSource
tests: add tests with set and hashmap member keys
2018-01-28 18:17:51 -08:00
Ilya Kreymer
131c5ff5da
SOCKS proxy (#281)
warcserver: SOCKS proxy:
- add support for running warcserver through a socks proxy specified via SOCKS_HOST and SOCKS_PORT
- move socks patch setup, http max_header adjustment to http module
- logging: print stack trace only if debugging
- add pysocks to extra_requirements, enable in ci
- add simple test (not actual proxy) to check that connection through proxy is attempted
- docs: add SOCKS proxy section to docs
2018-01-17 10:51:49 -08:00
Ilya Kreymer
4f340933f3
Add CodeCov (#282)
coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor
coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__
README: badge update, add appveyor, codecov badges
2018-01-17 09:59:51 -08:00
raffaele messuti
c9cea9fc4f wb-manager cli, fix for no arguments (#280) 2018-01-16 08:08:25 -08:00
raffaele messuti
a7d9494363 make manager commands working with python3 (#279) 2018-01-15 23:09:14 -08:00
Ilya Kreymer
85f093e356
Fix Query UI (#278)
* query fix:
setup: ensure all static files included in package_data recursively to add new query assets
test: add test for nested static asset
query: correctly display 0 captures, 'capture' and 'captures' text moved to Text block
2018-01-15 19:54:15 -08:00
Ilya Kreymer
0c24f8a1c1
Docs and README Update for 2.0.0 (#277)
* docs and version update:
- add docs for compatibility features
- add docs for memento
- updat rewriter docs
- bump version to 2.0.0, update README, and changelist
2018-01-11 21:34:04 -08:00
Ilya Kreymer
36b9bdfa2c replay banner tweaks:
- two line display, include title (if available) or url in the banner
- switch to dark theme consistent with query ui
2018-01-09 13:11:44 -08:00
Ilya Kreymer
a65bfcf567 query ui: improvements to new query ui from @Fernando-Melo
- move scripts to query.js, fix formatting
- init ui from cdx list, refactor into single script
- use cdx api to retrieve query via ajax
- tests: update query tests to use cdx lookup instead
- remove server-side cdx lookup for /*/ endpoint
2018-01-09 13:10:42 -08:00
Fernando Melo
831587152d new query.html page
fix bug write 1 version when single version
2018-01-09 13:10:42 -08:00
John Berlin
9c5673968c wombat: improved the fetch override to ensure that a live leak does not occur when input is an instance of WombatLocation or URL, will also handle any object that has href (#276) 2018-01-08 16:08:21 -08:00
John Berlin
3c05f27829 html_rewriter: added the nullification of meta tag delivered CSP policies to HTMLRewriterMixin, treat it like the integrity attribute (#274)
rewrite test: updated the html_rewriter test to cover the changes made for meta CSP rewriting
fixes #273
2018-01-08 13:57:09 -08:00
Rebecca Lynn Cremona
d3b379e788 Improved rewriting of srcset image urls; handle urls with commas (#269)
* rewrite improvement: better srcset parsing for comma-separated urls

* extensive server-side tests for srcset rewriting (with and without spaces and extra srcset modifiers)

* compile regex once for improved performance

* same regex for server and client side rewriting

Work by @rebeccacremona
2018-01-05 12:24:52 -08:00
Anastasia Aizman
777f55f201 add - pass in colls_dir instead of hardcoding (#268) 2018-01-04 16:34:44 -08:00
Ilya Kreymer
8b6eb6d5ca warcserver: routing: use werkzeug default rule instead of 'path:' (currently used for single path segments anyway), fixes issues with
werkzeug 0.14, fixes #271
2018-01-04 15:56:07 -08:00
Ilya Kreymer
df14c67a56 docs: docs update, start rewriter section 2017-12-09 22:51:19 -08:00
Ilya Kreymer
2ddff987be range requests: rewriting disabled only if range response (206) is returned
tests: add test to ensure range request redirect response is correctly rewriting, add 302 replay test
2017-12-07 17:46:50 -08:00