1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

1551 Commits

Author SHA1 Message Date
Ilya Kreymer
ba19ff1cd5 proxy: add custom rewrite prefix in proxy mode with 'pywb-proxy-rewrite: prefix' header 2015-12-23 23:14:47 -08:00
Ilya Kreymer
0cf6b40af9 wombat: add option to def_prop() to make overriden property enumerable, make WombatLocation and other loc overrides enumerable, fixes #163 2015-12-18 21:46:50 -08:00
Ilya Kreymer
79d5ec2b2d statusheaders: when not verifying protocol line, avoid indexerror when no space in first line, add tests 2015-12-18 21:46:00 -08:00
Ilya Kreymer
f6a4f80e47 bump to 0.11.1 2015-12-18 21:27:03 -08:00
Ilya Kreymer
6e58fe7a40 CHANGES.rst clarification 0.11.0 2015-12-13 00:40:01 -08:00
Ilya Kreymer
7d4c76f308 CHANGES.rst fixes 2015-12-13 00:37:01 -08:00
Ilya Kreymer
da01f107c7 Update CHANGES for 0.11.0 2015-12-13 00:31:38 -08:00
Ilya Kreymer
79b35a64b3 bump wombat version to 2.10 2015-12-13 00:31:01 -08:00
Ilya Kreymer
381f350917 proxy: switching not available for ip resolver either
tests: update tests for auth and ip resolver to check that proxy magic is not set
2015-12-12 22:59:32 -08:00
Ilya Kreymer
e9b11fcbf2 proxy: default to cookie resolver, which allows switching collections and datetime, instead of auth resolver
auth resolver can be used by setting 'cookie_resolver: false' explicitly. when using auth resolver,
don't set proxy magic path as switching collections or datetime is not possible with auth resolver
closes #160
2015-12-12 21:58:12 -08:00
Ilya Kreymer
64fe49d703 wombat: add CSSStyleSheet.prototype href override
karma tests: change name to just 'pywb'
2015-12-11 17:33:22 -08:00
Ilya Kreymer
467ea0f68b Merge branch 'client-side-tests' into develop 2015-12-10 12:50:40 -08:00
Ilya Kreymer
9b8b4d8388 tests/typo fix: add tests for truncated record detection (see: ikreymer/webarchiveplayer#14) fix typo, closes #161 2015-12-10 12:31:58 -08:00
Ilya Kreymer
3e3d7e4d87 Merge pull request #159 from robertknight/anchor-override-tostring
wombat: Override HTMLAnchorElement.prototype.toString()
2015-12-10 11:34:56 -08:00
Ilya Kreymer
b12a24e990 Merge pull request #162 from robertknight/resolve_relative_rel_canon
rewrite: Resolve relative canonical paths if rewriting is disabled
2015-12-10 11:33:15 -08:00
Robert Knight
83a33e0541 Resolve relative canonical paths if rewriting is disabled
For Via, we want rel=canonical links to resolve to the same
absolute URL as it did on the original page.

For absolute URLs, no rewriting is necessary. If the original
rel=canonical URL was relative however, it needs to be resolved
relative to the original URL.

See https://github.com/hypothesis/via/issues/65 for context.
2015-12-10 08:31:50 +00:00
Robert Knight
34721a6742 Override HTMLAnchorElement.prototype.toString()
The values returned by getting the 'href' and 'toString'
properties of an anchor element should be the same.

This inconsistency broke the URL polyfill in
https://github.com/inexorabletash/polyfill under Microsoft Edge
2015-12-02 20:09:09 +00:00
Ilya Kreymer
7a0680fb35 memento: for not found timemap query, return empty timemap, instead of html query error page, closes #158 2015-11-30 09:40:07 -08:00
Ilya Kreymer
04104f04d3 travis: move to travis env settings instead 2015-11-26 11:12:39 -08:00
Ilya Kreymer
fa8528b885 travis: testing with sauce connect enabled! 2015-11-26 11:00:01 -08:00
Ilya Kreymer
b3c9a47ec7 Merge pull request #157 from robertknight/wombat-ms-edge-compat
Microsoft Edge compatibility fixes, including Safari fix from #156 and Node.prototype fix from #154
2015-11-26 10:11:21 -08:00
Robert Knight
977647cea9 Override the baseURI property on the correct object
baseURI is a property of Node.prototype
in current browsers and should be overridden there,
except for Safari where it cannot be overridden at all.
2015-11-26 17:09:26 +00:00
Robert Knight
1997c4a180 Run Karma tests against Microsoft Edge
* Increase the default timeouts to account for the relative
   slowness of setting up connections to remote browsers.

 * Change the URL into which Wombat JS is loaded for
   tests to be a valid URL. Under Microsoft Edge, the JS
   code in the page is not run if the URL fetch returns a 404.

 * Change assert.equal() implementation to avoid confusion due
   to Karma's reformatting of URLs in exception error messages.
2015-11-26 17:09:20 +00:00
Robert Knight
e4e3de85e2 Increase the browser inactivity timeout
Increase the inactivity timeout when running
CI tests in case as recommended at
https://docs.travis-ci.com/user/gui-and-headless-browsers/#Karma-and-Firefox-inactivity-timeouts

That guidance applies to Travis but also to Sauce Labs.
2015-11-26 13:57:35 +00:00
Robert Knight
1484b06da6 Avoid changing the writability of the 'href' attr of <base>
Wombat overrides document.baseURI and <base>.href in order
to return the original URL rather than the proxied URL.

The <base>.href override however ended up making a writable
attribute read-only, which could trigger script errors
in strict-mode JS.

Fix this by avoiding replacing the setter for a DOM property
if no replacement setter is provided.

Fixes an error loading Hypothesis under Microsoft Edge.
2015-11-26 13:54:47 +00:00
Robert Knight
70a098cbd4 Enable Sauce Connect in Travis
Rather than specifying the username and encrypted
key in the Travis config, we just enable Sauce Connect
and require SAUCE_USERNAME and SAUCE_ACCESS_KEY env vars
to be set.

This is so that the Karma tests have the same env vars
available to them which they can use to check whether
to run against Sauce Labs or not.
2015-11-26 11:10:38 +00:00
Robert Knight
a153ada08e Configure Travis to make Firefox available for Karma tests
For some Karma tests, we can run them faster by running
against a local browser in the Travis CI instance rather
than Sauce Labs.

This is also useful for:

 - Contributors wishing to run
   the Travis tests against their own forks and have not
   set up Sauce Labs credentials.

 - Running Karma tests against Pull Request builds where
   Sauce Connect is not available.
2015-11-26 11:09:41 +00:00
Robert Knight
4f86a895c5 Enable testing against Safari on Sauce Labs
* Provide a fallback mode in the Karma tests which tests
   against a local browser (defaults to Firefox) if Sauce Labs
   credentials are not set.

   This is useful for local testing for contributors who
   might not have a Sauce Labs account.

 * Add Safari under OS X to the set of Sauce Labs browsers
   that the Karma tests are run against, following the merge
   of the WombatJS fixes for Safari and Edge.

   Although Edge now works under manual testing, automated
   testing against Sauce Labs is not yet working for reasons
   yet to be determined.
2015-11-26 10:16:23 +00:00
Robert Knight
9c88b1fd20 Enable running Karma tests against Safari
Detect in the test whether overriding of DOM properties
is supported in the current environment and skip testing
for the baseURI override in that case.
2015-11-26 10:15:35 +00:00
Robert Knight
f2fdbcc511 Lookup baseURI getter on Node.prototype, not Node
Fix regression in baseURI override, spotted by the Karma
tests.

The getter should be looked up on Node.prototype, not Node.
2015-11-26 10:00:09 +00:00
Ilya Kreymer
cd2390333a Merge pull request #153 from robertknight/safari-dom-fixes
Fix Wombat in Safari 9
2015-11-26 01:15:30 -08:00
Ilya Kreymer
bb8f384244 Merge branch 'develop' into client-side-tests 2015-11-26 01:01:03 -08:00
Ilya Kreymer
9f77ac50e7 setup.py: better fix for test suite with latest setuptool 2015-11-26 00:59:57 -08:00
Ilya Kreymer
4303ce4ecb warc indexing: better handling of records with content-length to small, read first line to get to warc end (fixes indexing of warc in ikreymer/webarchiveplayer#14) 2015-11-26 00:47:15 -08:00
Ilya Kreymer
2922801b7c test: rewrite_live: pass Accepnt-Encoding: identity to disable gzip, simplified version of fix in #151 2015-11-26 00:34:45 -08:00
Ilya Kreymer
26684b0973 Merge pull request #152 from robertknight/wombatjs-tests
Wombatjs tests
2015-11-26 00:25:49 -08:00
Robert Knight
5208117299 Fix Wombat in Safari 9
In Safari 9, Object.getOwnPropertyDescriptor(domObject.prototype, prop)
returns descriptors which have the correct structure but undefined
getters and are marked as unconfigurable.

See https://bugs.webkit.org/show_bug.cgi?id=49739#c19
for details. The getters cannot be retrieved via obj.__lookupGetter__()
either.

 * Resolve the issue by skipping overrides for DOM properties
   where the property is not configurable, or the original getter for
   a property could not be retrieved.

 * Lookup the 'baseURI' property on the correct prototype (Node,
   not document)

This fix also resolves the problem where accesses to document.baseURI
on Edge would fail.
2015-11-25 16:52:11 +00:00
Robert Knight
c38ca5cdd4 Add Python and node_module caching
Speed up Travis CI rebuilds by caching pip
and npm dependencies.
2015-11-25 14:57:51 +00:00
Robert Knight
011487604e Add editorconfig file that documents JS conventions in this repo
Existing JS files in this repo use Python-esque spacing
and naming conventions (4 spaces, snake_cased) rather than
JS conventions (2 spaces, camelCased).
2015-11-25 14:57:51 +00:00
Robert Knight
a4d05d469f Add configuration for running client tests using Sauce Labs
This gives us the capability to test the client-side JS
rewriting against a suite of different browsers on different
platforms.

Note that getting Karma running on Travis CI requires some
additional configuration documented at
github.com/karma-runner/karma-sauce-launcher/issues/73
which is not made clear in the original documentation.

In order to run the Karma tests, the Sauce Labs username
and access keys need to be passed in via the SAUCE_USERNAME
and SAUCE_ACCESS_KEYS env vars.
2015-11-25 14:57:20 +00:00
Robert Knight
f4a43a14f5 Add tests for the client-side part of pywb using Karma
* Add a Karma configuration for unit/integration tests
   for the client-side pywb code.

 * Add an integration test suite which creates an <iframe> loads
   the client-side rewriting code (wombat.js) in it and
   then executes a test script.

   Since wombat.js monkey-patches the DOM and the exact behavior
   of DOM objects varies between browsers, which we want to test,
   the suite does not mock the DOM but instead runs
   a set of tests in an isolated environment against
   the DOM.

 * Add Travis config to run the Karma tests
2015-11-25 11:53:40 +00:00
Ilya Kreymer
6c76675fe7 setup: fix setup.py again 2015-11-04 16:33:19 -08:00
Ilya Kreymer
c107dd3af1 setup: fix 'setup.py test' with new setuptools changes, fix surt==0.2 2015-11-04 16:16:10 -08:00
Ilya Kreymer
d98c1f6cf7 memento/api: add a new /collinfo.json end-point, enabled with 'enable_coll_info' config setting, which returns
the value fo collinfo.json template. Default template returns an entry for each handler route,
including the route path (id), title (name) and memento timegate and timemap paths, to be used with
an aggregator. Using a custom 'info_json' template can specify a different collinfo template, alternative to #69 (local aggregation)
Closes #146
2015-11-04 15:36:44 -08:00
Ilya Kreymer
dc74b14af0 iframe scrolling: use webkit scrollbars for better scrolling iframes due to osx chrome iframe bug
scrollbar now fully functional, though some artifacts still appear #93
2015-10-31 12:06:01 -07:00
Ilya Kreymer
bd2b5181a0 tests: add new tests for redis-based cache, #145 2015-10-30 13:18:58 -07:00
Ilya Kreymer
3132bfa7f4 cache: add a simple RedisCache implementation (alongside local and uwsgi)
proxy_ip_resolver: add option to use RedisCache if redis_cache_key set in config
proxy_ip_resolver: add 'delete' option to delete ip from cache, closes #145
2015-10-30 13:15:07 -07:00
Ilya Kreymer
16cf997a07 proxy: stick with http1.0 as not really supporting 1.1, #143 2015-10-26 15:25:00 -07:00
Ilya Kreymer
eeb35ea3b4 proxy: add ProxyRouter wrapper to check for content-length and, if missing, perform full buffering (http1.0) or chunked encoding (http1.1) (separate from replay view buffering)
add tests for buffering and chunked encoding, fixes #143, also tests no banner url-rewrite only proxy related to #142
2015-10-25 18:02:51 -07:00
Ilya Kreymer
0c96591c49 proxy: change HttpsUrlRewriter to SchemeOnlyUrlRewriter, which fixes http->https or https->http to match
the scheme of the current page.
url-rewrite-only mode: add uo_ mod and use that to rewrite only urls (no banner, no client side rewrite)
addresses #142
2015-10-24 15:10:30 -07:00