Add WACZ support for `wb-manager add` by unpacking WACZ files with --uncompress-wacz.
A future commit will add pywb support for WACZ files without requiring them to be unpacked.
* Add locale-dependent handling of first day of week
The Intl.Locale is a proposed standard not yet supported by Firefox so
in Firefox the first day of week will default to Monday (as specified
in ISO-8601).
* Set top frame document title when Vue updates
* Update template guide for 2.7
* Drop Python 3.6 and add 3.10 in test CI
* Allow either JS mimetype in test_add_static
* Add convenience build script for Vue UI
* Add build flag to docker compose example
* Fix Vue app issue with redirect_to_exact: false
Fixes#779
Undated URLs were resulting in a broken calendar and timeline in the
Vue app when redirect_to_exact was set to false. This was due to
TopFrameView using the current datetime if no timestamp was included,
which caused a failed snapshot lookup in the Vue app.
This commit changes the default timestamp in TopFrameView to None and
adds additional logic in the Vue app to use the last snapshot's
timestamp as the default if one is not present to match the snapshot
that pywb loads by default under the same conditions.
* Add filter instead of submitting form when pressing enter in the filtering expression field
* Make filter expressions translatable
* Add missing tooltip strings to vue_loc
* Add changelog
* Bump version to 2.7.1
* Use empty string as default template timestamp
* Bump wombat to 3.3.13
Co-authored-by: Jonas Linde <jonasjlinde@gmail.com>
templates: add placeholder templates (footer.html, head.html)
templates: allow 'base_html', 'footer_html', 'head_html', 'header_html' to be added via wb-manager 'add-template' cmd
ui: fix logo path, support linking
ui: make url on banner an input field
docs: clarify docs around templates, paths
- add ui-overview as UI toc page
- refactor ui-customization to top-level customizations page
- add template-guide for in-depth template reference
- add new-vue-ui page for docs on new ui, with images
- fix adding logo to old ui, add to docs
The top two access types end in a ".". The final two do not. This simply adds periods to the third and fourth list items to make punctuation consistent among the bullets.
* add support for custom data being added via 'PUT /<coll>/record' when in recording mode and 'enable_put_custom_record: true' set in 'recorder' config
- url specified via 'url' query arg and content type via request Content-Type
- update docs for put custom record options
* bump version to 2.6.0b4
- replace erroneous/outdated `/coll-cdx` API endpoint
by default API endpoint `/<coll>/cdx`
- if clear from preceding context: reduce examples
to params only `?url=...¶m1=...`
* localization / doc fixes:
- add missing header.html
- docs: support 'i18n' extra, mention in docs
- use 'default_locale' for html lang tag
- access control docs: fix documentation for adding user with acl command
* localization: add compile_catalog after extract as well to simplify updates for identity (en) locale
* ui:
- include locale in home page collection listing
- keep locale on error page home link
* autoescape:
- ensure jinja2 templates are autoescaped to prevent xss issues (thanks @sebastian-nagel for suggested fix)
- ensure banner inserts are not double-escaped
- update tests for template autoescaping
* update CHANGES.rst
* bump version to 2.6.0b1
* add localization utilities:
- add locmanager to support extract, update, remove, list using pybabel
- add po2csv/csv2po conversion with translate-utils
- docs: add localization.rst to manual!
* add language switch header (via header.html) to all pages if more than one locale is present.
* localization: wrap more text strings in templates in existing templates
* docs:
- document `wb-manager i18n` commands
- mention `<html lang>` setting
- include csv example
- add info about adding localizable text in templates
* add localization to CHANGES
* embargo: add support for per-collection date range embargo with embargo options of 'before', 'after', 'newer' and 'older'
'before' and 'after' accept a timestamp
'newer' and 'older' options configured with a dictionary consisting of any combo of 'years', 'months', 'days'
add basic test for each embargo option
* acl/embargo work:
- support acl access value 'allow_ignore_embargo' for overriding embargo
- support 'user' in acl setting, matched with value of 'X-Pywb-ACL-User' header
- support passing through 'X-Pywb-ACL-User' setting to warcserver
- aclmanager: support -u/--user param for adding, removing and matching rules
- tests: add test for 'allow_ignore_embargo', user-specific acl rule matching
* docs: add docs for new embargo system!
* docs: add info on how to configure ACL header with short examples to usage page.
sample-deploy: add examples of configuring X-pywb-ACL-user header based on IP for nginx and apache sample deployments
* docs: fix access control page header, text tweaks
* bump version to 2.6.0b0
* dedup improvements on top of #597, work towards patching support (#601)
- single key 'dedup_policy' of 'skip', 'revisit', 'keep'
- optional 'dedup_index_url', defaults to redis urls
- support for 'cache: always' to further add cacheing on all requests that have a referrer
- updated docs to mention latest config, explain 'instant replay' that is possible when dedup_policy is set
- add check to ensure only redis:// URLs can be set for dedup_index_url for now
- config: convert shorthand 'recorder: <source_coll>' setting string to dict, don't override custom config
* Add support for verifying ssl certificates
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
* Add documentation for new certificate configuration options
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
* Add test to check the verification of ssl certificates
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
* docs work on OpenWayback -> pywb transition, part 1
* docs: add config change examples, exclusions and deploy recommendations
* update with path index example
* update terms with collection info
* docs update:
- add zipnum examples to owb-to-pywb config transition
- add working docker compose examples for nginx subdirectory, apache subdirectory and outback cdx deployment in ./sample-deploy
- update usage and owb-to-pywb deployment docs with updated subdiretory deployment info + sample-deploy links
* tweak exclusion info, deploy title
* add missing filee uwsgi_subdir.ini
* Docs: fix typos and clarifications from review (thanks @ldko!)
Co-authored-by: Lauren Ko <lauren.ko@unt.edu>
* docs: explain that existing cdx can be added to outbackcdx, explain reindexing is optional
* docs: elaborate on docker-compose examples
* minor tweaks
* update to latest wombat 3.0.2
* update CHANGES.rst
* bump version to 2.5.0 for release
Co-authored-by: Lauren Ko <lauren.ko@unt.edu>
* ensure that the RemoteCDXIndexSource also adds a 'matchType=' param, fix for ukwa-pywb/ukwa#57
* 2.4.2 fixes:
- cdxindexer: don't treat first param as output, require '-o <output>' instead, update tests
- cleanup: move url-polyfill.min.js to correct static dir, addresses #571
- update to latest wombat
- move logo to ./pywb/static, fix README path
- tests: update indexing tests for cdx-indexer fix
- bump version to 2.4.2
- Fix link in access-control docs to use RST instead of MD syntax (#568) (by @machawk1)
Per the code, the key should use an underscore, not a hyphen. It also seems like the value is parsed as a number instead of a string, which then fails with a type error later, so quote it to force it to be a string.
```
$ pywb
2020-03-10 21:06:33,084: [INFO]: Proxy enabled for collection "web"
Traceback (most recent call last):
File "/tmp/pywb_venv/bin/pywb", line 8, in <module>
sys.exit(wayback())
File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 20, in wayback
desc='pywb Wayback Machine Server').run()
File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 89, in __init__
self.application = self.load()
File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 181, in load
return FrontEndApp(custom_config=self.extra_config)
File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/frontendapp.py", line 79, in __init__
self.init_proxy(config)
File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/frontendapp.py", line 569, in init_proxy
if not self.ALL_DIGITS.match(self.proxy_default_timestamp):
TypeError: expected string or buffer
```
* metadata/coll_config: don't confuse user metadata with collection config, don't display collection config settings as metadata (ukwa/ukwa-pywb#47)
- for collection template, add separate 'coll_config' dict, keep user metadata only in 'metadata' dict (default to empty)
- for static collections, assume metadata is in the 'metadata' dict of collection config
- for dynamic collections, load metadata.yaml into 'metadata' dict
- ensure 'metadata' key is passed to frame_insert
- ensure 'metadata' added consistently in framed and non-framed mode
- tests: update tests to ensure metadata is added consistently
- fuzzymatch: don't match 204 OPTIONS responses, update fuzzymatcher test
* documentation
- add documentation for metadata in ui-customization, rebuild docs,
- add link to ui customization from configuring
- work on access control docs
* fixed small typo's in ui-customization.rst
* frontendapp: fix doc string
- misc: remove warning on urllib3 Retry init
- set version to pywb 2.4.0rc0
Co-Authored-By: John Berlin <n0tan3rd@gmail.com>
- split out the ui customization documentation into its own file ui-customization.rst
- added initial documentation covering the new template setup to the ui-customization.rst
* proxy: add option to set default timestamp for proxy mode, fixes#452
- set via flag --proxy-default-timestamp or config 'proxy_options.default_timestamp'
- can be iso date or all-digit timestamp
- overridable via accept-datetime header
* docs: update docs for proxy timestamp
- add docs on memento support in proxy mode
* update-version: script can update version only, commit with 'update-version.sh commit'
* indexer post append: remove 'WB_wombat_' from POST query, could have been added in previous versions of pywb!
* add sample Apache configuration
This configuration can be used when launching `wayback` in the default
configuration, which is useful to add stuff like access control,
authentication, or encryption without going through the trouble of
setting up a UWSGI proxy.
* enable support for X-Forwarded-Proto headers from #395
Rename:
- rename auto-fetch config to 'enable_auto_fetch' and '--enable-auto-fetch' cli param
- rename 'use_head_insert' -> 'enable_content_rewrite'
- rename 'use_banner' -> 'enable_banner'
- rename 'use_wombat' -> 'enable_wombat'
Misc Cleanup:
- enable_auto_fetch applies to both proxy and non-proxy mode
- remove setting 'wbinfo.use_wombat', implied if wombatProxyMode.js is included
- docs: add docs for auto-fetch system, improved docs for proxy rewrite options
- tests: test with enable_auto_fetch, update tests for renames
- bump version to 2.1.0 due to breaking changes
- changelist: updates to changelist
- requirements: use bounded version for gevent
* Add source_filter option to recorder.
* Add test and docs for source_filter option.
* Update test_record_replay.py - Split source_filter test into skip existing and new recording
* proxy mode options: #316
- add 'use_banner' option, if false, will disable standard banner.html from being added
- add 'use_head_insert' option, if false, will disable injecting head_insert.html in proxy mode
both options default to true
* docs: add docs for new proxy options
* also add 'override_route' option and docs for extending proxy routing
