backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 00:03:28 +01:00
Code Issues Projects Releases Activity
1,706 Commits 56 Branches 63 Tags
Commit Graph

32 Commits

Author SHA1 Message Date
Ilya Kreymer
97182b71b7 refactor: 
- merge pywb.urlrewrite -> pywb.rewrite, remove obsolete stuff (rewrite_content.py, rewrite_live.py, dsrules.py)
- move wbrequestresponse -> pywb.apps
- move pywb.webapp.handlers -> pywb.apps.static_handler
- remove pywb.webapp, pywb.framework packages
- disable old header_rewriter, content_rewriter tests
- finish renaming from previous warcserver refactor
- all other tests passing!
 2017-05-23 19:08:29 -07:00
Ilya Kreymer
58f39f0558 setup: update to warcio==1.2 
add ensure_http_headers=True when reading WARC records
tests: fix pytest warnings, use webtest.TestApp instead of TestApp
 2017-04-29 13:47:54 -07:00
Ilya Kreymer
082487ab3c support per-collection assets again: 
- wb-manager added metadata now loaded dynamically, cached, for search and index pages (#196)
- metadata updated w/o restart (#87)
- per-collection template overrides and per-template static file support
tests: test_auto_colls.py fully ported to new system
(per-collection config.yaml no longer supported)
 2017-04-26 12:18:36 -07:00
Ilya Kreymer
52dc46fe6a remove obsolete code and tests! 
disable test_auto_colls for now until fully supported in new system
 2017-04-25 19:39:19 -07:00
Ilya Kreymer
8fe2c1b5bd apps & cli: remove old apps, keep: 
- webagg-server
- wayback
- live-rewrite-server
support adding custom settings to AutoApp
support for --live flag that automatically adds live-web source at '/live'
tests: disable cdx_server tests as old cdx_server removed
 2017-03-12 12:21:54 -07:00
Ilya Kreymer
50a3353da3 wsgi server: default to gevent-based wsgi server for all cmd line server apps, add -s command for specifying server #201 
cli: add 'webagg-server' cli command for running new webagg system
tests: fix cli test for gevent server
 2016-12-09 16:46:33 -08:00
Ilya Kreymer
ab77c1b6d9 refactor autoindex: switch to gevent-based simple polling, as watchdog doesn't work with gevent #200 2016-11-11 10:31:48 -08:00
Ilya Kreymer
3f734e1c98 tests: remove 3.2, fix auto_index test assert 2016-03-10 13:07:57 -08:00
Ilya Kreymer
3a584a1ec3 py3: all tests pass, at last! 
but not yet py2... need to resolve encoding in rewriting issues
 2016-02-23 13:26:53 -08:00
Ilya Kreymer
4ba4521b56 tests: use random port instead of 8080 for cli test to avoid conflicts with running services 2015-10-23 11:53:28 -07:00
Ilya Kreymer
e37636de84 cdxindexer: if latest ujson (with forward slash not-escaping) is available, use that when indexing, closes #140 
tests: update indexer CDXJ tests to be order-independent
travis: install ujson for testing
 2015-10-22 17:46:05 -07:00
Ilya Kreymer
e1a9334a54 tests: update test to match cdx-convert 2015-08-25 23:06:00 +03:00
Ilya Kreymer
63c6efc851 autocolls test: patch wsgiref not waitress as it is default 2015-07-31 09:26:48 -07:00
Ilya Kreymer
f9bd2ba55a jinja template: use shared template in J2Template, init on first use 2015-04-03 10:43:39 -07:00
Ilya Kreymer
a34607764e manager: validate name on collection init: must start with wordchar and can contain wordchar or - 2015-04-03 01:18:35 -07:00
Ilya Kreymer
90eee03cdb fixes for windows: 
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
 2015-03-25 10:56:53 -07:00
Ilya Kreymer
da7532a1f8 wb-manager: rename 'migrate' to 'cdx-convert' for clarity 2015-03-23 11:05:02 -07:00
Ilya Kreymer
ae363ad368 autoindex and cli: add autoindex to cli with 'wayback -a' option, #81 2015-03-22 23:03:39 -07:00
Ilya Kreymer
e8db31d066 cli: improve wayback cli to take optional port, threads and working dir arguments 
switch to waitress as default WSGI server instead of wsgiref
 2015-03-22 21:50:56 -07:00
Ilya Kreymer
733642551d manager: support autoindexing! (#91) wb-manager autoindex will use watchdog library to detect creation/updates 
to any warc/arc in specified collection or across all and update autoindex cdx
cdx indexing: add --dir-root option to specify custom relative root dir for filenames used in cdx
 2015-03-22 17:55:38 -07:00
Ilya Kreymer
b43a7f94f3 manager: add cdx -> cdxj migration tool #80, which will convert all cdxs in a directory to cdxj, removing original files 
migration will also recanonicalize the urlkey to surt form
add migration test using non-surt, 9-field cdx (created from samples)
cdxindexer: fix multi warc->multi cdx indexing options
 2015-03-19 20:57:33 -07:00
Ilya Kreymer
c5b5c8ee4b manager: fix index path to index.cdxj 2015-03-19 13:41:48 -07:00
Ilya Kreymer
ea460bb0f0 cdxj: support cdx json output from cdx server with output='json' (not yet default) 
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
 2015-03-19 13:33:49 -07:00
Ilya Kreymer
73f24f5a2b manager: fixes for windows: use shutil.move instead of os.rename to allow move to 
existing file
tests: reset workdir before deleting temp dir
 2015-03-18 13:14:05 -07:00
Ilya Kreymer
bfe590996b auto-config: add support for loading from root ./static/ directory, 
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
 2015-03-17 19:05:39 -07:00
Ilya Kreymer
4b45e789df templates: ensure shared templates are loaded from root templates/ subdir 
manager: add shared templates to templates subdir, not root dir #55 and #74
 2015-03-16 19:57:28 -07:00
Ilya Kreymer
2f6780a576 rename for 0.9.0: 
rename default templates package from ui/* templates to templates/*
rename default subdirs: warcs -> archive, cdx -> indexes
 2015-03-16 18:48:09 -07:00
Ilya Kreymer
19b8650891 manager: templates: add collections manager (#74) commands for adding, removing and listing 
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
 2015-03-16 16:55:06 -07:00
Ilya Kreymer
be5139b635 fix tests for coll listing, #78 
config override: when loading from coll-specific config.yaml, resolve
relative paths to that collection, not to root #55
 2015-03-15 22:23:08 -07:00
Ilya Kreymer
30454abb6b metadata: add support for user-defined per-collection metadata! #78 
metadata stored in wbrequest.user_metadata and available to all templates

collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
 2015-03-15 21:24:15 -07:00
Ilya Kreymer
b417b47835 collections manager: support for merge when adding warc, explicit --index-warcs 
option to index and merge instead of reindexing whole dir, #74
additional testing for recursive indexing, index merge
timeutils: add timestamp20_now() function
 2015-03-14 14:56:15 -07:00
Ilya Kreymer
759d151551 tests: add test for directory auto collection loader, 
collection manager and new 6-field minimal cdx format
 2015-03-13 19:53:50 -07:00