backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-16 08:28:52 +01:00
Code Issues Projects Releases Activity
1,621 Commits 56 Branches 63 Tags
Commit Graph

129 Commits

Author SHA1 Message Date
Ilya Kreymer
2a7da54be9 Merge branch 'master' into new-pywb 2017-02-17 18:49:01 -08:00
Ilya Kreymer
31bf7a47f1 new-wayback cli script, using new FrontEndApp (rewriting) + AutoConfigApp (config-driven aggregator) 
support for dynamic collections: check all .cdxj files in /<coll>/indexes/*.cdxj when accessing /<coll>
support for fixed routes: specified in config.yaml as per https://github.com/ikreymer/pywb/wiki/Distributed-Archive-Config
werkzeug routing in FrontEndApp: default query, replay, search pages working
route listing: /_coll_info.json for listing fixed + dynamic routes
autoindexing enabled, indexing WARCs added to archives directory to .cdxj index
Addresses #196
 2017-02-17 18:04:07 -08:00
Ilya Kreymer
60f3c0a213 setup: further update trove classifiers, add python 2 and 3, switch to production stable, closes #208 2017-02-16 15:12:22 -08:00
Ilya Kreymer
77960c1311 setup: bound webtest for 2.6 support 2017-02-16 14:32:14 -08:00
Ilya Kreymer
7f95396be0 setup: use jinja2<2.9 for now 2017-02-16 11:29:30 -08:00
Ilya Kreymer
58b141bd53 add python 3 classifiers (#208) 2017-02-16 11:02:53 -08:00
Ilya Kreymer
9773eba47d setup reqs: use webassets==0.12.1 (with pyinstaller support), remove dependency on custom branch 2017-01-26 01:37:57 -08:00
Ilya Kreymer
84796ba810 setup req: fix jinja<2.9 for now due to issues in 2.9+ 2017-01-26 01:14:10 -08:00
Ilya Kreymer
2d54bb87be setup.py: ensure gevent monkey-patch is called before running tests with python setup.py test 2017-01-26 00:37:35 -08:00
raffaele messuti
524d9bfd26 portalocker for file locking check instead of fcntl. more portable on windows 2016-12-26 10:27:20 +01:00
Ilya Kreymer
0e414acfda setup: remove pyamf as default dep for now 2016-12-21 17:15:41 -08:00
Ilya Kreymer
3b82416ad3 setup: add specific dependencies for webassets, pyamf 2016-12-21 16:11:48 -08:00
Ilya Kreymer
50a3353da3 wsgi server: default to gevent-based wsgi server for all cmd line server apps, add -s command for specifying server #201 
cli: add 'webagg-server' cli command for running new webagg system
tests: fix cli test for gevent server
 2016-12-09 16:46:33 -08:00
Ilya Kreymer
e37900b9c6 tests: add test dependency, remove 2.6 from travis 2016-11-11 11:03:16 -08:00
Ilya Kreymer
8765de4fe7 refactor: updated dependencies, remove watchdog, add gevent and webassets 
update tests, tests should pass for python 2 and 3!
 2016-11-11 10:32:19 -08:00
Ilya Kreymer
6b4b038471 refactor: fix pywb.webagg package paths 
all webagg tests working!
move testdata cdxj into sample_archive, remove rest (duplicates) #200
 2016-11-08 14:30:09 -08:00
Ilya Kreymer
457a1a564c bufferedreader: support brotli decompression 
rewrite: handle Content-Encoding: br using brotli decompressor
setup: add brotlipy as dependency
 2016-06-15 01:37:29 -04:00
Ilya Kreymer
0f530a3e0e dependencies: remove pyamf, update to latest surt (0.3.0) 2016-06-12 00:44:52 -04:00
Ilya Kreymer
87da25c703 post request mapping improvements: work on #178, including: 
- mapping multipart/form-data same as x-www-form-urlencoded
- parsing application/x-amf with pyamf
- RewriteContentAMF for rewriting AMF response to match request
- default encoding of other POST data as base64 encoded __wb_post_data param
 2016-05-06 10:19:08 -07:00
Ilya Kreymer
e8c77c0538 encoding: encode before quote 
setup: enable zip_safe=True again
 2016-04-30 15:15:35 -07:00
Ilya Kreymer
c1bdeac92b redis: fix redis key lookup, add tests for zrangebylex with new fakeredis 2016-03-09 18:33:04 -08:00
Ilya Kreymer
b7008920de fix setup.py typo 2016-02-16 16:14:10 -08:00
Ilya Kreymer
3c85f7b7ac py3: make pywb.utils work with python 3! 2016-02-16 14:52:20 -08:00
Ilya Kreymer
9f77ac50e7 setup.py: better fix for test suite with latest setuptool 2015-11-26 00:59:57 -08:00
Ilya Kreymer
6c76675fe7 setup: fix setup.py again 2015-11-04 16:33:19 -08:00
Ilya Kreymer
c107dd3af1 setup: fix 'setup.py test' with new setuptools changes, fix surt==0.2 2015-11-04 16:16:10 -08:00
Jack Cushman
633eb31f57 Use webencodings to encode head_insert_str. 2015-10-22 16:40:59 -04:00
Ilya Kreymer
d84185aea6 setup: remove waitress from require pkgs, now optional 2015-07-31 02:49:03 -07:00
Ilya Kreymer
f0359877f0 youtube-dl: remove from dependency, installation is optional. Return 404 if attempting live 
proxy of videos and youtube-dl is not available (the only use case).
HTTPParser wrapping logic no longer needed in latest versions
Modify tests to only run if youtube-dl is installed in cases where it is not available #118
 2015-06-27 16:11:59 -07:00
Ilya Kreymer
cda7705075 split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth cli as 
the 'certauth' tool superceeds this functionality. (#90).
To use https proxy mode, 'pip install certauth' is required. (update travis config)
 2015-03-29 17:38:57 -07:00
Ilya Kreymer
0faa6aac3e setup: set version in pywb __init__.py 2015-03-23 11:04:41 -07:00
Ilya Kreymer
df76bc3500 cli: change cdx-server and live-rewrite-server to go through shared cli 
entry point
 2015-03-23 09:08:09 -07:00
Ilya Kreymer
e8db31d066 cli: improve wayback cli to take optional port, threads and working dir arguments 
switch to waitress as default WSGI server instead of wsgiref
 2015-03-22 21:50:56 -07:00
Ilya Kreymer
6a9a09d602 setup: add 'watchdog' as a dependency 2015-03-22 18:24:56 -07:00
Ilya Kreymer
ea460bb0f0 cdxj: support cdx json output from cdx server with output='json' (not yet default) 
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
 2015-03-19 13:33:49 -07:00
Ilya Kreymer
fe1c32c8f7 cdxj: support loading cdxj (#76) 
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
 2015-03-19 12:36:49 -07:00
Ilya Kreymer
bfe590996b auto-config: add support for loading from root ./static/ directory, 
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
 2015-03-17 19:05:39 -07:00
Ilya Kreymer
138aed3ddd change version to 0.9.0b1 2015-03-16 19:15:00 -07:00
Ilya Kreymer
2f6780a576 rename for 0.9.0: 
rename default templates package from ui/* templates to templates/*
rename default subdirs: warcs -> archive, cdx -> indexes
 2015-03-16 18:48:09 -07:00
Ilya Kreymer
19b8650891 manager: templates: add collections manager (#74) commands for adding, removing and listing 
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
 2015-03-16 16:55:06 -07:00
Ilya Kreymer
30454abb6b metadata: add support for user-defined per-collection metadata! #78 
metadata stored in wbrequest.user_metadata and available to all templates

collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
 2015-03-15 21:24:15 -07:00
Ilya Kreymer
759d151551 tests: add test for directory auto collection loader, 
collection manager and new 6-field minimal cdx format
 2015-03-13 19:53:50 -07:00
Ilya Kreymer
60f33412ff collections manager: add new collections manager, first pass #74 
add cli 'wb-manager' tool
very preliminary, needs testing still
 2015-02-25 13:19:20 -08:00
Ilya Kreymer
7c60bf17f7 bump version to 0.9.0-beta! 2015-02-24 16:54:49 -08:00
Ilya Kreymer
83f8d7d29b bump version to 0.8.2 2015-02-22 22:51:23 -08:00
Ilya Kreymer
adeb8bfb27 bump version to 0.8.1, (fix blank spacing in changelist) 2015-02-20 02:02:34 -08:00
Ilya Kreymer
59630c08f6 bump version to 0.8.0! 2015-01-26 11:08:08 -08:00
Ilya Kreymer
48b7751f80 bump version to 0.7.6 
jinja2: allow adding multiple packages to search path
 2015-01-19 21:54:11 -05:00
Ilya Kreymer
7ece05d022 bump version to 0.7.5 
update CHANGES
fix .gitattributes to use standard flags
 2015-01-12 00:09:02 -08:00
Ilya Kreymer
1eb0f96f92 windows support work: fix loaders to use pathname2url to convert to 
file:/// url, use urlopen to open file paths
fix some tests to use universal line breaks
 2015-01-10 14:06:15 -08:00