backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 00:03:28 +01:00
Code Issues Projects Releases Activity
pywb/tests/test_config.yaml

163 lines
4.2 KiB
YAML
Raw Normal View History

add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# pywb config file
# ========================================
#
# Settings for each collection
collections:
# <name>: <cdx_path>
# collection will be accessed via /<name>
# <cdx_path> is a string or list of:
# - string or list of one or more local .cdx file
# - string or list of one or more local dirs with .cdx files
# - a string value indicating remote http cdx server
pywb: ./sample_archive/cdx/
add optional filters to default Route add examples to config.yaml and test_config.yaml and integration test per route config is inherited globally if only name is set
2014-02-06 17:28:08 -08:00
# ex with filtering: filter CDX lines by filename starting with 'dupe'
replay: support 'framed_replay' option in config for both replay and live rewrite split replay view into BaseContentView and ReplayView refactor RewriteLiveHandler into RewriteLiveView add additional tests for framed and non-framed mode default to framed replay!
2014-06-14 18:26:19 -07:00
pywb-filt:
index_paths: './sample_archive/cdx/'
filters: ['filename:dupe*']
tests rangecache: added integration tests for range support via range cache, using enable_cache option
2014-12-23 11:09:19 -08:00
route selection via proxy auth! refactor poute request parsing to happen in the actual router class instead of in the route in proxy mode, add support for picking a route via proxy-auth improve test for 'top' rewriting
2014-07-10 21:54:23 -07:00
pywb-filt-2:
index_paths: './sample_archive/cdx/'
filters: ['!filename:dupe*']
replay: support 'framed_replay' option in config for both replay and live rewrite split replay view into BaseContentView and ReplayView refactor RewriteLiveHandler into RewriteLiveView add additional tests for framed and non-framed mode default to framed replay!
2014-06-14 18:26:19 -07:00
pywb-nonframe:
index_paths: './sample_archive/cdx/'
framed_replay: false
add optional filters to default Route add examples to config.yaml and test_config.yaml and integration test per route config is inherited globally if only name is set
2014-02-06 17:28:08 -08:00
test: move non-surt-cdx sample to non-surt-cdx/ dir for clarity / avoid confusion when bulk loading cdx/ dir (surt and non-surt cdx should NOT be mixed)
2014-05-16 21:21:14 -07:00
# collection of non-surt CDX
tests rangecache: added integration tests for range support via range cache, using enable_cache option
2014-12-23 11:09:19 -08:00
pywb-nosurt:
replay: support 'framed_replay' option in config for both replay and live rewrite split replay view into BaseContentView and ReplayView refactor RewriteLiveHandler into RewriteLiveView add additional tests for framed and non-framed mode default to framed replay!
2014-06-14 18:26:19 -07:00
index_paths: './sample_archive/non-surt-cdx/'
surt_ordered: false
cdx: fix creation and add test for non-surt cdx (pywb-nonsurt/ test) archiveindexer: -u option to generate non-surt cdx tests: full test coverage for cdxdomainspecific (fuzzy and custom canon)
2014-05-16 21:16:50 -07:00
tests: add integration tests for $liveweb rewrite handler and replay with fallback
2014-07-20 18:25:47 -07:00
# live collection
live: $liveweb
# coll with fallback
pywb-fallback:
index_paths: ./sample_archive/cdx/
fallback: live
replay: make range cache available by default in replay_views since its inited on first use. remove separate subclass. 'enable_ranges' can be set to false to disable range cache altogether improve tests
2014-12-23 14:34:59 -08:00
pywb-norange:
tests rangecache: added integration tests for range support via range cache, using enable_cache option
2014-12-23 11:09:19 -08:00
index_paths: ./sample_archive/cdx/
rewrite: improvements to non-exact replay mode, redir_to_exact option set to false frames: add request_ts to wbinfo and use that as the timestamp in the top-frame. for exact replay, request_ts == timestamp for latest replay / no timestamp / memento timegate, redirect to current time instead of time of last capture, while serving last capture. timeutils: add timestamp_now() function to return timestamp of current datetime Add extra tests for this mode Tracked via #72
2015-02-17 17:47:30 -08:00
enable_ranges: false
pywb-non-exact:
index_paths: ./sample_archive/cdx/
redir_to_exact: false
cdxj: support loading cdxj (#76) cdx obj: allow alt field names to be used (eg. mime, mimetype, m) (status/statuscode/s) in querying and reading cdx cdx minimal: (#75) now implies cdxj to avoid more formats minimal includes digest always and mime when warc/revisit tests for cdxj loading indexing optimization: reuse same entry obj for records of same type
2015-03-19 11:20:40 -07:00
pywb-cdxj:
index_paths: ./sample_archive/cdxj/
rewrite: improvements to non-exact replay mode, redir_to_exact option set to false frames: add request_ts to wbinfo and use that as the timestamp in the top-frame. for exact replay, request_ts == timestamp for latest replay / no timestamp / memento timegate, redirect to current time instead of time of last capture, while serving last capture. timeutils: add timestamp_now() function to return timestamp of current datetime Add extra tests for this mode Tracked via #72
2015-02-17 17:47:30 -08:00
add optional filters to default Route add examples to config.yaml and test_config.yaml and integration test per route config is inherited globally if only name is set
2014-02-06 17:28:08 -08:00
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# indicate if cdx files are sorted by SURT keys -- eg: com,example)/
# SURT keys are recommended for future indices, but non-SURT cdxs
# are also supported
#
# * Set to true if cdxs start with surts: com,example)/
# * Set to false if cdx start with urls: example.com)/
surt_ordered: true
# list of paths prefixes for pywb look to 'resolve' WARC and ARC filenames
# in the cdx to their absolute path
#
# if path is:
# * local dir, use path as prefix
# * local file, lookup prefix in tab-delimited sorted index
# * http:// path, use path as remote prefix
# * redis:// path, use redis to lookup full path for w:<warc> as key
simplify pywb_init config: - add defaults dictionary, chain dictionaries rather than copying - allow custom classes to be loaded explicitly via yaml - for LineReader, assume ungzipped if first decompress fails - properly ignore bad local paths - add optional reporter object
2014-02-11 14:10:40 -08:00
archive_paths: ['./invalid/path/to/ignore/', './sample_archive/warcs/']
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# ==== Optional UI: HTML/Jinja2 Templates ====
# template for <head> insert into replayed html content
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
head_insert_html: templates/head_insert.html
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# template to for 'calendar' query,
# eg, a listing of captures in response to a ../*/<url>
#
# may be a simple listing or a more complex 'calendar' UI
# if omitted, will list raw cdx in plain text
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
query_html: templates/query.html
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# template for search page, which is displayed when no search url is entered
# in a collection
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
search_html: templates/search.html
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# template for home page.
# if no other route is set, this will be rendered at /, /index.htm and /index.html
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
home_html: templates/index.html
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# error page temlpate for may formatting error message and details
# if omitted, a text response is returned
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
error_html: templates/error.html
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
templates: add new 'not_found.html' template, which will be called for any missing replay request instead of default error.html 'not_found_html' settable in the config per collection, as per #65 for not found index query, still use query.html but add condition to check for 0 results add more query and replay not found remove unused conditional (for search_view -- always exists)
2015-01-24 12:32:50 -08:00
# template for 404 not found error, may be customized per collection
rename for 0.9.0: rename default templates package from ui/* templates to templates/* rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
not_found_html: templates/not_found.html
templates: add new 'not_found.html' template, which will be called for any missing replay request instead of default error.html 'not_found_html' settable in the config per collection, as per #65 for not found index query, still use query.html but add condition to check for 0 results add more query and replay not found remove unused conditional (for search_view -- always exists)
2015-01-24 12:32:50 -08:00
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# ==== Other Paths ====
paths cleanup: - don't store explicit static path, but allow it to be set in the insert - store host_prefix, which is either server name or empty - for archival mode, absolute_paths settings controls if using absolute paths, - for proxy always use absolute_paths - default static path is: /static/default/ - allow extension apps to provide custom /static/X/ path Route overriding: - ability to set Route class - custom init method Archival Relative Redirect: - if starting with timestamp, drop timestamp and assume host-relative path Integration Tests: - test proxy mode by using REQUEST_URI - test archival relative redirect!
2014-02-08 20:07:16 -08:00
# Rewrite urls with absolute paths instead of relative
absoulte_paths: true
add support for pywb static content routes (seperate from uwsgi) adding StaticHandler and loading templates and static resources from current package add default template and static data to be included in the pywb package add test for custom static route
2014-02-07 19:32:58 -08:00
# List of route names:
# <route>: <package or file path>
static_routes:
refactoring: clean up handlers and replay_views for pep8 use BlockLoader().load for StaticHandler static file resolving update static paths to point to pywb/static instead of static
2014-03-14 18:17:22 -07:00
static/test/route: pywb/static/
static paths: ensure consistent renaming of static/default -> static/__pywb for bundled static path
2015-03-23 16:15:37 -07:00
static/__pywb: pywb/static/
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
first pass at simple http proxy! #8 * proxy router for handling only proxy * proxy/archival router for handling both archival and proxy mode, togglable with 'enable_http_proxy' setting in config * supports only most recent capture playback -- no support for selecting replay date/calendar view yet * not testable with WebTest -- need better way to unit test proxy mode
2014-02-05 13:08:10 -08:00
# Enable simple http proxy mode
enable_http_proxy: true
proxy: add 'unaltered_replay' option to proxy_options to replay all content unaltered (no rewriting html, no banner, no wombat) use 'proxy_options' instead of 'routing_options', add additional tests for proxy mode
2014-07-21 16:42:14 -07:00
# Additional proxy options (defaults)
proxy_options:
proxy: ensure proxy cert download sets content length proxy options: 'use_default_coll' must specify exact default coll (otherwise a random coll is chosen, as ordering is not defined) travis: add py3.4, py3.5!
2016-02-23 18:09:09 -08:00
use_default_coll: pywb
proxy: add 'unaltered_replay' option to proxy_options to replay all content unaltered (no rewriting html, no banner, no wombat) use 'proxy_options' instead of 'routing_options', add additional tests for proxy mode
2014-07-21 16:42:14 -07:00
update CHANGES, config.yaml docs for proxy mode ensure proxy_options match defaults in config.yaml default cookie_resolver to true
2014-09-06 17:03:04 -07:00
cookie_resolver: false
proxy: update tests for new use_banner, use_client_rewrite options, #107
2015-08-15 19:14:59 -07:00
use_client_rewrite: true
use_wombat: true
proxy: add 'unaltered_replay' option to proxy_options to replay all content unaltered (no rewriting html, no banner, no wombat) use 'proxy_options' instead of 'routing_options', add additional tests for proxy mode
2014-07-21 16:42:14 -07:00
memento/api: add a new /collinfo.json end-point, enabled with 'enable_coll_info' config setting, which returns the value fo collinfo.json template. Default template returns an entry for each handler route, including the route path (id), title (name) and memento timegate and timemap paths, to be used with an aggregator. Using a custom 'info_json' template can specify a different collinfo template, alternative to #69 (local aggregation) Closes #146
2015-11-04 15:36:44 -08:00
#enable coll info JSON
enable_coll_info: true
add new test_config.yaml!
2014-02-05 10:17:06 -08:00
# enable cdx server api for querying cdx directly (experimental)
improved test: dsrules remove redundant check static: check invalid static paths and file_wrapper memento: check non-memento paths test debug handlers and custom '-cdx' suffix
2014-05-16 22:17:51 -07:00
#enable_cdx_api: True
# or specify suffix
enable_cdx_api: -cdx
simplify pywb_init config: - add defaults dictionary, chain dictionaries rather than copying - allow custom classes to be loaded explicitly via yaml - for LineReader, assume ungzipped if first decompress fails - properly ignore bad local paths - add optional reporter object
2014-02-11 14:10:40 -08:00
- cdx handler refactoring: factor out CDXHandler and init to seperate cdx_handler module - Make wsgi app a class, add port as an optional field in wsgi app and router. (not required to be specified)
2014-03-03 10:35:57 -08:00
# test different port
port: 9000
simplify pywb_init config: - add defaults dictionary, chain dictionaries rather than copying - allow custom classes to be loaded explicitly via yaml - for LineReader, assume ungzipped if first decompress fails - properly ignore bad local paths - add optional reporter object
2014-02-11 14:10:40 -08:00
# optional reporter callback func
# if set, called with request and cdx object
address test cases broken by previous commit. move py.test fixture and fixture classes (TestExclusionPerms, PrintReporter) to tests.fixture module. update test_config.yaml accordingly.
2014-02-28 01:39:04 +00:00
reporter: !!python/object/new:tests.fixture.PrintReporter []
cdx: add domain-specific rules at cdx layer for custom canonicalization! and 'fuzzy' matching when not found handled via cdxdomainspecific.py BaseCDXServer contains a canonicalizer object and a fuzzy query canonicalizer abstracted to seperate class (in canonicalizer.py) clean up cdx related exceptions default rules read from cdx/rules.yaml filename configurable via 'domain_specific_rules' setting in config.yaml fix typo in pywb/rewrite
2014-02-18 14:47:48 -08:00
# custom rules for domain specific matching
#domain_specific_rules: rules.yaml
exclusions: add AllAllowPerms and refactor exclusions interface add TestExclusionPerms and a sample exclusion integration test refactor cdx server init params into **kwargs convert all cdx params to use camelCase
2014-02-19 20:20:31 -08:00
replay: support 'framed_replay' option in config for both replay and live rewrite split replay view into BaseContentView and ReplayView refactor RewriteLiveHandler into RewriteLiveView add additional tests for framed and non-framed mode default to framed replay!
2014-06-14 18:26:19 -07:00
# Use lxml parser, if available
pass raw bytes to htmlparser, assuming ascii-compatibility (todo: add tests for non-ascii compatible encodings) improved rendering of certain pages, needs more testing lxml: remove lxml and complexity associated with having the parser, as its too unpredictable for older html, does its own decoding.
2014-06-27 19:03:06 -07:00
# use_lxml_parser: true
replay: support 'framed_replay' option in config for both replay and live rewrite split replay view into BaseContentView and ReplayView refactor RewriteLiveHandler into RewriteLiveView add additional tests for framed and non-framed mode default to framed replay!
2014-06-14 18:26:19 -07:00
# Replay content in an iframe
framed_replay: true
first pass at memento support #10! memento support enabled by default, togglable via 'enable_memento' config property supporting timegate and memento apis, no timemap yet supporting pattern 2.3 for archival and pattern 1.3 for proxy modes also: simplify exception hierarchy a bit more, move down to utils make WbRequest and WbResponse extensible with mixins (eg for memento)
2014-03-14 10:46:20 -07:00
# ==== New / Experimental Settings ====
# Not yet production ready -- used primarily for testing
exclusions: add AllAllowPerms and refactor exclusions interface add TestExclusionPerms and a sample exclusion integration test refactor cdx server init params into **kwargs convert all cdx params to use camelCase
2014-02-19 20:20:31 -08:00
#perms_checker: !!python/object/new:pywb.cdx.perms.AllowAllPerms []
refactor: rename pywb.core -> pywb.webapp move perms/test/test_perms_policy -> tests/perms_fixture for rules file, use single DEFAULT_RULES_FILE import
2014-04-04 10:09:26 -07:00
perms_policy: !!python/name:tests.perms_fixture.perms_policy
first pass at memento support #10! memento support enabled by default, togglable via 'enable_memento' config property supporting timegate and memento apis, no timemap yet supporting pattern 2.3 for archival and pattern 1.3 for proxy modes also: simplify exception hierarchy a bit more, move down to utils make WbRequest and WbResponse extensible with mixins (eg for memento)
2014-03-14 10:46:20 -07:00
# not testing memento here
enable_memento: False
improved test: dsrules remove redundant check static: check invalid static paths and file_wrapper memento: check non-memento paths test debug handlers and custom '-cdx' suffix
2014-05-16 22:17:51 -07:00
# Debug Handlers
debug_echo_env: True
debug_echo_req: True
Reference in New Issue Copy Permalink