1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 00:03:28 +01:00

67 Commits

Author SHA1 Message Date
Tessa Walsh
e89924bd39
Rename --uncompress-wacz to --unpack-wacz and add docs (#901)
Also adds help text for wb-manager add --unpack-wacz option in CLI
2024-04-24 05:02:26 -04:00
Tessa Walsh
5712945991
Update usage docs section on creating web archives (#899)
Co-authored-by: Henry Wilkinson <henry@wilkinson.graphics>
2024-04-15 10:22:39 -04:00
Tessa Walsh
86ee3bd752
Allow ACLJs to use *, SURT wildcard to match all URLs (#882)
Also adds tests and documentation
2024-04-03 17:11:58 -04:00
Tessa Walsh
83b2113be2
Add config.yaml UI option to disable printing from replay banner (#815)
* Add UI option to disable printing

* Initialize VueUI.main with config dict
2023-03-27 10:23:37 -04:00
aponb
81b6a57dfb
Update usage.rst Docker examples (#816) 2023-02-20 10:10:08 -05:00
kuechensofa
454486bf75
[#799] wb-manager: Add wacz archives to collection with --uncompress-wacz (#800)
Add WACZ support for `wb-manager add` by unpacking WACZ files with --uncompress-wacz.

A future commit will add pywb support for WACZ files without requiring them to be unpacked.
2023-02-15 17:00:38 -05:00
Tessa Walsh
3d0673e32a
Add ui.logo_home_url as config.yaml option (#790)
* Add ui.logo_home_url as config option
* Add ui.logo_home_url to docs
2023-01-05 17:33:00 -05:00
Tessa Walsh
2d19b6b18d
Merge 2.7.1 development branch (#785)
* Add locale-dependent handling of first day of week

The Intl.Locale is a proposed standard not yet supported by Firefox so
in Firefox the first day of week will default to Monday (as specified
in ISO-8601).

* Set top frame document title when Vue updates

* Update template guide for 2.7

* Drop Python 3.6 and add 3.10 in test CI

* Allow either JS mimetype in test_add_static

* Add convenience build script for Vue UI

* Add build flag to docker compose example

* Fix Vue app issue with redirect_to_exact: false

Fixes #779

Undated URLs were resulting in a broken calendar and timeline in the
Vue app when redirect_to_exact was set to false. This was due to
TopFrameView using the current datetime if no timestamp was included,
which caused a failed snapshot lookup in the Vue app.

This commit changes the default timestamp in TopFrameView to None and
adds additional logic in the Vue app to use the last snapshot's
timestamp as the default if one is not present to match the snapshot
that pywb loads by default under the same conditions.

* Add filter instead of submitting form when pressing enter in the filtering expression field

* Make filter expressions translatable

* Add missing tooltip strings to vue_loc

* Add changelog

* Bump version to 2.7.1

* Use empty string as default template timestamp

* Bump wombat to 3.3.13

Co-authored-by: Jonas Linde <jonasjlinde@gmail.com>
2022-12-07 18:16:18 -08:00
Tessa Walsh
2ad7eaee4b Update docs
- Add thorough Vue theming instructions
- Update frameless replay banner info
2022-11-21 12:46:55 -05:00
Ilya Kreymer
6260b226ce ui/templates: fixes based on feedback from @ldko!
templates: add placeholder templates (footer.html, head.html)
templates: allow 'base_html', 'footer_html', 'head_html', 'header_html' to be added via wb-manager 'add-template' cmd
ui: fix logo path, support linking
ui: make url on banner an input field
docs: clarify docs around templates, paths
2022-11-21 12:46:09 -05:00
Lauren Ko
29860bcb24 New UI corrections (#687)
* Fix typos.

* Make ui variable available to collection page
2022-11-21 12:46:09 -05:00
Ilya Kreymer
790487ca15 docs: add new UI docs:
- add ui-overview as UI toc page
- refactor ui-customization to top-level customizations page
- add template-guide for in-depth template reference
- add new-vue-ui page for docs on new ui, with images
- fix adding logo to old ui, add to docs
2022-11-21 12:46:09 -05:00
Tessa Walsh
f96707d039
Add uwsgi virtualenv information (#770) 2022-11-17 10:47:05 -08:00
Victor "Vito" Gama
7432299079
Add missing org/image to docker run commands (#733) 2022-08-09 13:53:02 -07:00
Mat Kelly
ddcbde573c
Documentation: Add periods to end of list of access types in docs to make consistent (#670)
The top two access types end in a ".". The final two do not. This simply adds periods to the third and fourth list items to make punctuation consistent among the bullets.
2022-01-25 18:19:51 -08:00
Lauren Ko
b2a460c33c
docs: fix broken links (#666) 2021-08-10 08:17:40 -07:00
Ilya Kreymer
98c6fba44d
Support for custom data being added via 'PUT /<coll>/record' when… (#661)
* add support for custom data being added via 'PUT /<coll>/record' when in recording mode and 'enable_put_custom_record: true' set in 'recorder' config
- url specified via 'url' query arg and content type via request Content-Type
- update docs for put custom record options

* bump version to 2.6.0b4
2021-07-18 17:04:34 -07:00
Simon Chan
096850b41d
fix errors in docs/manual/rewriter.rst (#655)
* fix format error in docs/manual/rewriter.rst

* fix incorrect names in docs/manual/rewriter.rst
2021-07-06 21:01:38 -07:00
Sebastian Nagel
f9f5d2dc33
Improve docs about CDXJ Server API endpoint (#651)
- replace erroneous/outdated `/coll-cdx` API endpoint
  by default API endpoint `/<coll>/cdx`
- if clear from preceding context: reduce examples
  to params only `?url=...&param1=...`
2021-06-15 18:12:48 -07:00
Ilya Kreymer
f7bd84cdac
Localization / doc fixes (#650)
* localization / doc fixes:
- add missing header.html
- docs: support 'i18n' extra, mention in docs
- use 'default_locale' for html lang tag
- access control docs: fix documentation for adding user with acl command

* localization: add compile_catalog after extract as well to simplify updates for identity (en) locale

* ui: 
- include locale in home page collection listing
- keep locale on error page home link

* autoescape:
- ensure jinja2 templates are autoescaped to prevent xss issues (thanks @sebastian-nagel for suggested fix)
- ensure banner inserts are not double-escaped
- update tests for template autoescaping

* update CHANGES.rst

* bump version to 2.6.0b1
2021-06-14 17:09:00 -07:00
Lauren Ko
9587954856
Fix typos in localization and access-control docs (#649)
* Fix typos in localization doc

* Fix typos in access-control doc
2021-06-11 22:50:35 -07:00
Ilya Kreymer
12fcc87962
Localization Support (#647)
* add localization utilities:
- add locmanager to support extract, update, remove, list using pybabel
- add po2csv/csv2po conversion with translate-utils
- docs: add localization.rst to manual!

* add language switch header (via header.html) to all pages if more than one locale is present.

* localization: wrap more text strings in templates in existing templates

* docs:
- document `wb-manager i18n` commands
- mention `<html lang>` setting
- include csv example
- add info about adding localizable text in templates

* add localization to CHANGES
2021-06-09 13:12:53 -07:00
Ilya Kreymer
f07d35709a
Access Control Improvements: Embargo + ACL User Support (#642)
* embargo: add support for per-collection date range embargo with embargo options of 'before', 'after', 'newer' and 'older'
'before' and 'after' accept a timestamp
'newer' and 'older' options configured with a dictionary consisting of any combo of 'years', 'months', 'days'
add basic test for each embargo option

* acl/embargo work:
- support acl access value 'allow_ignore_embargo' for overriding embargo
- support 'user' in acl setting, matched with value of 'X-Pywb-ACL-User' header
- support passing through 'X-Pywb-ACL-User' setting to warcserver
- aclmanager: support -u/--user param for adding, removing and matching rules
- tests: add test for 'allow_ignore_embargo', user-specific acl rule matching

* docs: add docs for new embargo system!

* docs: add info on how to configure ACL header with short examples to usage page.
sample-deploy: add examples of configuring X-pywb-ACL-user header based on IP for nginx and apache sample deployments

* docs: fix access control page header, text tweaks

* bump version to 2.6.0b0
2021-05-18 20:09:18 -07:00
Sebastian Nagel
662fc747bf
Fix ACL loading for auto collections (#620)
* Pass collection name to ACL checker to load ACL lists
for automatic collections

* Typo: file suffix must be `.aclj`
2021-04-26 19:58:56 -07:00
Ilya Kreymer
e1cad621b9
Dedup Improvments (#611)
* dedup improvements on top of #597, work towards patching support (#601)
- single key 'dedup_policy' of 'skip', 'revisit', 'keep'
- optional 'dedup_index_url', defaults to redis urls
- support for 'cache: always' to further add cacheing on all requests that have a referrer
- updated docs to mention latest config, explain 'instant replay' that is possible when dedup_policy is set
- add check to ensure only redis:// URLs can be set for dedup_index_url for now
- config: convert shorthand 'recorder: <source_coll>' setting string to dict, don't override custom config
2021-01-26 18:53:54 -08:00
Lukey3332
ddf3207e40
Add configuration options for dedup (#597)
* Add configuration options for dedup

Signed-off-by: Lukas Straub <lukasstraub2@web.de>

* Add documentation for new dedup_index configuration options

Signed-off-by: Lukas Straub <lukasstraub2@web.de>
2021-01-26 17:06:18 -08:00
Lukey3332
f628b40e02
Add support for verifying ssl certificates (#596)
* Add support for verifying ssl certificates

Signed-off-by: Lukas Straub <lukasstraub2@web.de>

* Add documentation for new certificate configuration options

Signed-off-by: Lukas Straub <lukasstraub2@web.de>

* Add test to check the verification of ssl certificates

Signed-off-by: Lukas Straub <lukasstraub2@web.de>
2021-01-26 12:41:26 -08:00
Ilya Kreymer
9e09bcd2a7
Docs Update: OpenWayback -> pywb Transition Guide (#588)
* docs work on OpenWayback -> pywb transition, part 1

* docs: add config change examples, exclusions and deploy recommendations

* update with path index example

* update terms with collection info

* docs update:
- add zipnum examples to owb-to-pywb config transition
- add working docker compose examples for nginx subdirectory, apache subdirectory and outback cdx deployment in ./sample-deploy
- update usage and owb-to-pywb deployment docs with updated subdiretory deployment info + sample-deploy links

* tweak exclusion info, deploy title

* add missing filee uwsgi_subdir.ini

* Docs: fix typos and clarifications from review (thanks @ldko!)

Co-authored-by: Lauren Ko <lauren.ko@unt.edu>

* docs: explain that existing cdx can be added to outbackcdx, explain reindexing is optional

* docs: elaborate on docker-compose examples

* minor tweaks

* update to latest wombat 3.0.2
* update CHANGES.rst

* bump version to 2.5.0 for release

Co-authored-by: Lauren Ko <lauren.ko@unt.edu>
2020-12-04 18:40:58 -08:00
Ilya Kreymer
7b51101b04 license: add NOTICE, update license statement for docs (gplv3) 2020-10-27 16:19:19 -07:00
Max Maass
3f3f8caef1
docs: Fix incorrect example (#574)
minor fix to docs example http://localhost:8080/my-web-archive/record/<url> -> http://localhost:8080/my-web-archive/record/http://example.com/
2020-07-10 20:40:24 -07:00
Ilya Kreymer
9b8c187b3a
2.4.2 Develop->Master (#572)
* ensure that the RemoteCDXIndexSource also adds a 'matchType=' param, fix for ukwa-pywb/ukwa#57

* 2.4.2 fixes:
- cdxindexer: don't treat first param as output, require '-o <output>' instead, update tests
- cleanup: move url-polyfill.min.js to correct static dir, addresses #571
- update to latest wombat
- move logo to ./pywb/static, fix README path
- tests: update indexing tests for cdx-indexer fix
- bump version to 2.4.2
- Fix link in access-control docs to use RST instead of MD syntax (#568) (by @machawk1)
2020-07-10 20:22:58 -07:00
thomas536
8f0ce45b27
docs: fix proxy default timestamp yaml example (#544)
Per the code, the key should use an underscore, not a hyphen. It also seems like the value is parsed as a number instead of a string, which then fails with a type error later, so quote it to force it to be a string.

```
$ pywb
2020-03-10 21:06:33,084: [INFO]: Proxy enabled for collection "web"
Traceback (most recent call last):
  File "/tmp/pywb_venv/bin/pywb", line 8, in <module>
    sys.exit(wayback())
  File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 20, in wayback
    desc='pywb Wayback Machine Server').run()
  File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 89, in __init__
    self.application = self.load()
  File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/cli.py", line 181, in load
    return FrontEndApp(custom_config=self.extra_config)
  File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/frontendapp.py", line 79, in __init__
    self.init_proxy(config)
  File "/tmp/pywb_venv/local/lib/python2.7/site-packages/pywb/apps/frontendapp.py", line 569, in init_proxy
    if not self.ALL_DIGITS.match(self.proxy_default_timestamp):
TypeError: expected string or buffer
```
2020-04-30 16:18:44 -07:00
Ivo Branco
8d8cf7eb58
Fix documentation: replace fl to fields on doc webrecorder/pywb#542 (#543) 2020-04-30 16:16:07 -07:00
Yvan
8baa8cbdb7 docs: fix doc typo in BaseWarcServer example (#507) 2019-10-31 17:09:25 -07:00
Ilya Kreymer
fed3263ac6
Docs: Fix access controls and ui customizations docs links (#513)
* docs: ensure docs added to access controls, fix typos

* begin changelist for 2.4.0
2019-10-31 16:56:36 -07:00
Ilya Kreymer
6f79840b79
Docs, custom metadata improvements (#509)
* metadata/coll_config: don't confuse user metadata with collection config, don't display collection config settings as metadata (ukwa/ukwa-pywb#47)
- for collection template, add separate 'coll_config' dict, keep user metadata only in 'metadata' dict (default to empty)
- for static collections, assume metadata is in the 'metadata' dict of collection config
- for dynamic collections, load metadata.yaml into 'metadata' dict
- ensure 'metadata' key is passed to frame_insert
- ensure 'metadata' added consistently in framed and non-framed mode
- tests: update tests to ensure metadata is added consistently

- fuzzymatch: don't match 204 OPTIONS responses, update fuzzymatcher test

* documentation
- add documentation for metadata in ui-customization, rebuild docs, 
- add link to ui customization from configuring
- work on access control docs
* fixed small typo's in ui-customization.rst
* frontendapp: fix doc string

- misc: remove warning on urllib3 Retry init

- set version to pywb 2.4.0rc0

Co-Authored-By: John Berlin <n0tan3rd@gmail.com>
2019-10-27 01:39:52 +01:00
John Berlin
379f7de1ba
manual
- split out the ui customization documentation into its own file ui-customization.rst
 - added initial documentation covering the new template setup to the ui-customization.rst
2019-09-05 18:13:12 -04:00
Ilya Kreymer
11610f6e04
2.3 Changelist + Docs Update (#487)
* docs: update changelist and add docs about new wombat

* update to latest wombat

* update wombat, fix pytest cmdline in setup
2019-07-09 17:50:57 -07:00
Eoin Kilfeather
96a7a4bbb0 Update configuring.rst to reflect default config.yaml. (#483)
The Docs specify the default value for the warc files path as 'archives' but the default config.yaml file specifies 'archive'
https://github.com/webrecorder/pywb/blob/master/pywb/default_config.yaml#L4
2019-07-08 14:16:57 -07:00
Ilya Kreymer
9448f4fe45 release: update changelist for 2.2.20190311
docs: fix typos
2019-03-11 16:40:53 -07:00
Ilya Kreymer
455efb17ad
Support for default timestamp/date for proxy mode (#454)
* proxy: add option to set default timestamp for proxy mode, fixes #452
- set via flag --proxy-default-timestamp or config 'proxy_options.default_timestamp'
- can be iso date or all-digit timestamp
- overridable via accept-datetime header

* docs: update docs for proxy timestamp
- add docs on memento support in proxy mode

* update-version: script can update version only, commit with 'update-version.sh commit'

* indexer post append: remove 'WB_wombat_' from POST query, could have been added in previous versions of pywb!
2019-03-11 16:28:09 -07:00
Ilya Kreymer
1fcc239ecf
Add Docker info to Docs (#448)
* docs: add docs on running with Docker, Docker image versions, fixes #299
2019-02-27 14:38:59 -08:00
Ilya Kreymer
0db8e5d718 Merge branch 'master' into develop for PR #395 2018-10-23 09:38:53 -07:00
anarcat
40f904af79 add sample Apache configuration (#374)
* add sample Apache configuration

This configuration can be used when launching `wayback` in the default
configuration, which is useful to add stuff like access control,
authentication, or encryption without going through the trouble of
setting up a UWSGI proxy.

* enable support for X-Forwarded-Proto headers from #395
2018-10-23 09:35:15 -07:00
Ilya Kreymer
3a70769c58
Cleanup CLI Switches and Docs for Auto-Fetch System (#394)
Rename:
- rename auto-fetch config to 'enable_auto_fetch' and '--enable-auto-fetch' cli param
- rename 'use_head_insert' -> 'enable_content_rewrite'
- rename 'use_banner' -> 'enable_banner'
- rename 'use_wombat' -> 'enable_wombat'

Misc Cleanup:
- enable_auto_fetch applies to both proxy and non-proxy mode
- remove setting 'wbinfo.use_wombat', implied if wombatProxyMode.js is included
- docs: add docs for auto-fetch system, improved docs for proxy rewrite options
- tests: test with enable_auto_fetch, update tests for renames
- bump version to 2.1.0 due to breaking changes
- changelist: updates to changelist
- requirements: use bounded version for gevent
2018-10-22 17:12:22 -07:00
eszense
6a2423e754 Add recorder option to filter source collection (#368)
* Add source_filter option to recorder.

* Add test and docs for source_filter option.

* Update test_record_replay.py - Split source_filter test into skip existing and new recording
2018-08-24 17:57:47 -07:00
Frank Sachsenheim
538ce88abc Fixes an enumeration issue in docs/usage.rst (#364)
Thanks! put it on develop so it can be part of next release.
2018-08-17 19:33:42 -07:00
Ilya Kreymer
819e8adf48
text updates: (#352)
- Update CHANGES.rst for 2.0.4
- Docs: Improve new proxy docs for (#316), fix URL-T->URI-T
- Requirements: bump to wsgiprox>=1.5.1
2018-06-27 09:02:01 -07:00
Ilya Kreymer
de3ec0e1bc proxy: use FrontEndApp.proxy_route_request() to determine proxy route
Extensions can override this function to provide custom proxy routing
Update docs
2018-04-20 15:20:56 -07:00
Ilya Kreymer
5349d0518c
Proxy Options (#317)
* proxy mode options: #316
- add 'use_banner' option, if false, will disable standard banner.html from being added
- add 'use_head_insert' option, if false, will disable injecting head_insert.html in proxy mode
both options default to true

* docs: add docs for new proxy options

* also add 'override_route' option and docs for extending proxy routing
2018-04-20 10:04:34 -07:00