- fixed edge case in jsonP rewriting where no callback name is supplied only ? but body has normal jsonP callback (url = https://geolocation.onetrust.com/cookieconsentpub/v1/geo/countries/EU?callback=?)
- made the `!self.__WB_pmw` server side inject match the client side one done via wombat
- added regex's for eval override to JSWombatProxyRules
wombat:
- added eval override in order to ensure AD network JS does not lockup the browser (cnn.com)
- added returning a proxied value for window.parent from the window proxy object in order to ensure AD network JS does not lock up browser (cnn.com, boston.com, et. al.)
- ensured that the read only storageArea property of 'StorageEvents' fired by the custom storage class is present (spec)
- ensured that userland cannot create new instances of Storage and that localStorage instanceof Storage works with our override (spec)
- ensured that assignments to SomeElement.[innerHTML|outerHTML], iframe.srcdoc, or style.textContent are more correctly handled in particular doing script.[innerHTML|outerHTML] = <JS>
- ensured that the test used in the isArgumentsObj function does not throw errors IFF the pages JS has made it so when toString is called (linkedin.com)
- ensured that Web Worker construction when using the optional options object, the options get supplied to the create worker (spec)
- ensured that the expected TypeError exception thrown from DomConstructors of overridden interfaces are thrown when their pre-conditions are not met (spec)
- ensured that wombat worker rewriting skipes data urls which should not be rewritten
- ensured the bound function returned by the window function is the same on subsequent retrievals fixes#474
tests:
- added direct testing of wombat's parts
* removed the definition of `__WB_pmw` from `ensureServerSideInjectsExistOnWindow` in order to allow more proper handling of that definition t occur from `initNewWindowWombat` or `wombatInit`.
`initNewWindowWombat` now initializes wombat for (i)frame's with src values prefixed with about: as about:srcdoc is commonly used
tweaked postMessage and event listener overrides to be more like the previous wombat revision
* rebased on develop and rebuilt bundle
* improved pywb's closing of open file handles and http connects by adding to pywb.util.io no_except_close
replaced close calls with no_except_close
reformatted and optimizes import of files that were modified
additional ci build fixes:
- pin gevent to 1.4.0 in order to ensure build of pywb on ubuntu use gevent's wheel distribution
- youtube-dl fix: use youtube-dl in quiet mode to avoid errors with youtube-dl logging in pytest
wombat:
- I: function overrides applied by wombat now better appear to be the original new function name same as originals when possible
- I: WombatLocation now looks and behaves more like the original Location interface
- I: The custom storage class now looks and behaves more like the original Storage
- I: SVG image rewriting has been improved: both the href and xlink:href deprecated since SVG2 now rewritten always
- I: document.open now handles the case of creation of a new window
- I: Request object rewriting of the readonly href property is now correctly handled
- I: EventTarget.addEventListener, removeEventListener overrides now preserve the original this argument of the wrapped listener
- A: document.close override to ensure wombat is initialized after write or writeln usage
- A: reconstruction of <doctype...> in rewriteHTMLComplete IFF it was included in the original string of HTML
- A: document.body setter override to ensure rewriting of the new body or frameset
- A: Attr.[value, nodeValue, textContent] added setter override to perform URL rewrites
- A: SVGElements rewriting of the filter, style, xlink:href, href, and src attributes
- A: HTMLTrackElement rewriting of the src attribute of the
- A: HTMLQuoteElement and HTMLModElement rewriting of the cite attribute
- A: Worklet.addModule: Loads JS module specified by a URL.
- A: HTMLHyperlinkElementUtils overrides to the areaelement
- A: ShadowRootoverrides to: innerHTML even though inherites from DocumentFragement and Node it still has innerHTML getter setter.
- A: ShadowRoot, Element, DocumentFragment append, prepend: adds strings of HTML or a new Node inherited from ParentNode
- A: StylePropertyMap override: New way to access and set CSS properties.
- A: Response.redirecthttps rewriting of the URL argument.
- A: UIEvent, MouseEvent, TouchEvent, KeyboardEvent, WheelEvent, InputEvent, and CompositionEven constructor and init{even-name} overrides in order to ensure that wombats JS Proxy usage does not affect their defined behaviors
- A: XSLTProcessor override to ensure its usage is not affected by wombats JS Proxy usage.
- A: navigator.unregisterProtocolHandler: Same override as existing navigator.registerProtocolHandler but from the inverse operation
- A: PresentationRequest: Constructor takes a URL or an array of URLs.
- A: EventSource and WebSocket override in order to ensure that they do not cause live leaks
- A: overrides for the child node interface
- Fix: autofetch worker creatation of the backing worker when it is operating within an execution context with a null origin
tests:
- A: 559 tests specific to wombat and client side rewritting
pywb:
- Fix: a few broken tests due to iana.org requiring a user agent in its requests
rewrite:
- introduced a new JSWorkerRewriter class in order to support rewriting via wombat workers in the context of all supported worker variants via
- ensured rewriter app correctly sets the static prefix
ci:
- Modified travis.yml to specifically enumerate jobs
documentation:
- Documented new wombat, wombat proxy moded, wombat workers
auto-fetch:
- switched to mutation observer when in proxy mode so that the behaviors can operate in tandem with the autofetcher
* proxy: add option to set default timestamp for proxy mode, fixes#452
- set via flag --proxy-default-timestamp or config 'proxy_options.default_timestamp'
- can be iso date or all-digit timestamp
- overridable via accept-datetime header
* docs: update docs for proxy timestamp
- add docs on memento support in proxy mode
* update-version: script can update version only, commit with 'update-version.sh commit'
* indexer post append: remove 'WB_wombat_' from POST query, could have been added in previous versions of pywb!
* Misc improvements, including fixes from @funkyfuture:
- Dockerfile: Reduces number of created layers and source contents
- Support for automatic collection creation if INIT_COLLECTION is defined
- Add entry point script docker-entrypoint.sh
- update to latest python (3.7.2 currently)
- additions to .dockerignore
- setup.py and requirements cleanup (just use plain 'gevent' requirement)
* docker-entrypoint.sh improvements:
- before running cmd, match uid/gid to that of volume dir (specified via $VOLUME_DIR, defaulting to /webarchive)
- if volume is owned by root (default if none mounted), just run as root
- if volume is owned by different user, create/update user 'archivist' to match the uid/gid of $VOLUME_DIR, then run cmd as 'su archivist'
* py3.7 fixes:
- add __repr__ to WBException for consistent output in py3.7
- don't raise StopIteration in generator, just return
* ci: add py3.7 builds to travis and appveyor, (don't include in integration test suite for now)
* versioning: new MAJ.MIN.DATE versioning
move version to version.py for easier updates
add update-version.sh for autoupdating version in version.py, pushing new tag with current version
* brotli: if the brotli module can not be loaded, print warning
and also remove `br` from any Accept-Encoding header to avoid recording with brotli, addresses #434
* Reworked query.js to know the difference between date search and advanced searching.
Exposed cdx api's through the query html page
- from, to
- matchType
- filter
Added more appealing styling to the error, index, not-found, query, and search templates
Updated the included jquery and boostrap static files to jQuery v3.3.1, Bootstrap v4.1.3
Implemented optionally using a web worker for making the cdx api request and processing the results
Documented the code
* ensure the display count str function uses the correct "first" value
* added view all captures for an result displayed in the advanced results view
query worker now sends over the recordCount as an integer and as a formatted string
moved the search button to the right after advanced options
* tests: fixed test_intergration.py:test_static_nested_dir failing due to updates
* recoder fix: ensure Transfer-Encoding header is not passed through by RecorderApp,
as may result in duplicate Transfer-Encoding in py2.7, fixes#432
* html rewriter fixes:
- html detection: allow for UTF-8 BOM when detecting if text is html
- html decl parsing: modify base parser regex to allow IE conditional declaration to also
end with -->, eg. support '<![endif]-->' in addition to '<![endif]>', fixes#425
* travis: add allow failure for integration tests (for now)
update pip and setuptools when running install.sh found in .travis
use xenial
removed trailing dash
only run webrecorder-tests using chrome and firefox
only run webrecorder-tests using pywbtest and chrometest marker expression
Added methods to AutoFetchWorker in proxy mode that allow external JS to initiate checks
Updated the actual proxy mode worker implementation to match the functionality added
Changes:
Reworked ContentFrame and the default banner to be ES5 classes.
Introduced an optional relationship between ContentFrame and banners.
If a banner is exposed then ContentFrame controls the initialization of the banner and routes any messages received from the replay iframe to the banner.
When the replay iframe is navigated to a page and the replay iframe loads, the ContentFrame waits 2 seconds before checking to see if the banner still indicates it a loading state and if so updates the displayed information using the URL and timestamp the replay iframe was navigated to.
* html rewrite: when encountering 'text/html' content-type, add html-detection check before assuming content is html (similar to text/plain)
supersedes #426, fixes#424 -- binary files served under mp_/ as text/html should now be served as binary
- when guessing if html, add additional regex to check if text does not start with < -- perhaps html but starting with plain text. only check for text/html content-type and not js_/cs_ mod
- added to the auto-fetch worker of both wombat and wombatProxymode
- added utility function isImageSrcset to wombat for determining if the srcset values being rewritten are from either a image tag or a source tag within a picture tag
- added utility function isImageDataSrcset to wombat to check for img/source data-srcset attributes
- reworked the backing auto-fetch worker to now queue all URLs and perform fetch batching with maximum batch size of 60. A delay of 2 seconds is applied after each batch.
Ensured that the srcset values sent to the auto-fetch worker can be resolved in non-proxy mode fixes#413
Renamed the auto-fetch class named used in proxy mode from AutoFetchWorker to AutoFetchWorkerProxyMode
Added checking of script tage types application/json and text/template to rewrite_script
* client-side rewrite: fix extract_orig() to unrewrite relative urls using current page scheme, don't default to http
* wombat tests: fix karma tests by adding 'wombat_scheme' to test setup
* server-side rewrite: tweak 'location' rewrite to ensure $location is not rewritten!
tests: add additional rewrite tests for 'location', 'this.$location' and 'this.location'
- attempt to encode headers as utf-8 first for live web, then latin-1 (similar to warcio http header parsing)
- only encode headers for py3 (in py2, headers are already bytestrings)
- tests: add tests for utf-8 in header
bump version to 2.1.1
- FullHTMLRegex: performs a case insensitive check for <html, <body, <head and <!doctype html>
updated rewrite_elem to:
- rewrite meta tags that deliever csp policies
- check for additional attributes that could contain un-rewritten URLs (form.style, iframe.style)
Made check for full html into regex
- ensured that autoFetchWorker uses full srcset URLs
- resolves the URL against the img.src or document.baseURI if not rewritten
- otherwise ensures the rewritten URL is not relative or schemeless
wombat.js:
- AutoFetchWorker updated extractFromLocalDoc to send URL resolution information to the worker
- defer extractFromLocalDoc and preserveSrcset postMessages to ensure page viewer can see the images first
* add sample Apache configuration
This configuration can be used when launching `wayback` in the default
configuration, which is useful to add stuff like access control,
authentication, or encryption without going through the trouble of
setting up a UWSGI proxy.
* enable support for X-Forwarded-Proto headers from #395
