Ilya Kreymer
2a7da54be9
Merge branch 'master' into new-pywb
2017-02-17 18:49:01 -08:00
Ilya Kreymer
31bf7a47f1
new-wayback cli script, using new FrontEndApp (rewriting) + AutoConfigApp (config-driven aggregator)
...
support for dynamic collections: check all .cdxj files in /<coll>/indexes/*.cdxj when accessing /<coll>
support for fixed routes: specified in config.yaml as per https://github.com/ikreymer/pywb/wiki/Distributed-Archive-Config
werkzeug routing in FrontEndApp: default query, replay, search pages working
route listing: /_coll_info.json for listing fixed + dynamic routes
autoindexing enabled, indexing WARCs added to archives directory to .cdxj index
Addresses #196
2017-02-17 18:04:07 -08:00
Ilya Kreymer
60f3c0a213
setup: further update trove classifiers, add python 2 and 3, switch to production stable, closes #208
2017-02-16 15:12:22 -08:00
Ilya Kreymer
77960c1311
setup: bound webtest for 2.6 support
2017-02-16 14:32:14 -08:00
Ilya Kreymer
7f95396be0
setup: use jinja2<2.9 for now
2017-02-16 11:29:30 -08:00
Ilya Kreymer
58b141bd53
add python 3 classifiers ( #208 )
2017-02-16 11:02:53 -08:00
Ilya Kreymer
9773eba47d
setup reqs: use webassets==0.12.1 (with pyinstaller support), remove dependency on custom branch
2017-01-26 01:37:57 -08:00
Ilya Kreymer
84796ba810
setup req: fix jinja<2.9 for now due to issues in 2.9+
2017-01-26 01:14:10 -08:00
Ilya Kreymer
2d54bb87be
setup.py: ensure gevent monkey-patch is called before running tests with python setup.py test
2017-01-26 00:37:35 -08:00
raffaele messuti
524d9bfd26
portalocker for file locking check instead of fcntl. more portable on windows
2016-12-26 10:27:20 +01:00
Ilya Kreymer
0e414acfda
setup: remove pyamf as default dep for now
2016-12-21 17:15:41 -08:00
Ilya Kreymer
3b82416ad3
setup: add specific dependencies for webassets, pyamf
2016-12-21 16:11:48 -08:00
Ilya Kreymer
50a3353da3
wsgi server: default to gevent-based wsgi server for all cmd line server apps, add -s command for specifying server #201
...
cli: add 'webagg-server' cli command for running new webagg system
tests: fix cli test for gevent server
2016-12-09 16:46:33 -08:00
Ilya Kreymer
e37900b9c6
tests: add test dependency, remove 2.6 from travis
2016-11-11 11:03:16 -08:00
Ilya Kreymer
8765de4fe7
refactor: updated dependencies, remove watchdog, add gevent and webassets
...
update tests, tests should pass for python 2 and 3!
2016-11-11 10:32:19 -08:00
Ilya Kreymer
6b4b038471
refactor: fix pywb.webagg package paths
...
all webagg tests working!
move testdata cdxj into sample_archive, remove rest (duplicates) #200
2016-11-08 14:30:09 -08:00
Ilya Kreymer
457a1a564c
bufferedreader: support brotli decompression
...
rewrite: handle Content-Encoding: br using brotli decompressor
setup: add brotlipy as dependency
2016-06-15 01:37:29 -04:00
Ilya Kreymer
0f530a3e0e
dependencies: remove pyamf, update to latest surt (0.3.0)
2016-06-12 00:44:52 -04:00
Ilya Kreymer
87da25c703
post request mapping improvements: work on #178 , including:
...
- mapping multipart/form-data same as x-www-form-urlencoded
- parsing application/x-amf with pyamf
- RewriteContentAMF for rewriting AMF response to match request
- default encoding of other POST data as base64 encoded __wb_post_data param
2016-05-06 10:19:08 -07:00
Ilya Kreymer
e8c77c0538
encoding: encode before quote
...
setup: enable zip_safe=True again
2016-04-30 15:15:35 -07:00
Ilya Kreymer
c1bdeac92b
redis: fix redis key lookup, add tests for zrangebylex with new fakeredis
2016-03-09 18:33:04 -08:00
Ilya Kreymer
b7008920de
fix setup.py typo
2016-02-16 16:14:10 -08:00
Ilya Kreymer
3c85f7b7ac
py3: make pywb.utils work with python 3!
2016-02-16 14:52:20 -08:00
Ilya Kreymer
9f77ac50e7
setup.py: better fix for test suite with latest setuptool
2015-11-26 00:59:57 -08:00
Ilya Kreymer
6c76675fe7
setup: fix setup.py again
2015-11-04 16:33:19 -08:00
Ilya Kreymer
c107dd3af1
setup: fix 'setup.py test' with new setuptools changes, fix surt==0.2
2015-11-04 16:16:10 -08:00
Jack Cushman
633eb31f57
Use webencodings to encode head_insert_str.
2015-10-22 16:40:59 -04:00
Ilya Kreymer
d84185aea6
setup: remove waitress from require pkgs, now optional
2015-07-31 02:49:03 -07:00
Ilya Kreymer
f0359877f0
youtube-dl: remove from dependency, installation is optional. Return 404 if attempting live
...
proxy of videos and youtube-dl is not available (the only use case).
HTTPParser wrapping logic no longer needed in latest versions
Modify tests to only run if youtube-dl is installed in cases where it is not available #118
2015-06-27 16:11:59 -07:00
Ilya Kreymer
cda7705075
split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth
cli as
...
the 'certauth' tool superceeds this functionality. (#90 ).
To use https proxy mode, 'pip install certauth' is required. (update travis config)
2015-03-29 17:38:57 -07:00
Ilya Kreymer
0faa6aac3e
setup: set version in pywb __init__.py
2015-03-23 11:04:41 -07:00
Ilya Kreymer
df76bc3500
cli: change cdx-server and live-rewrite-server to go through shared cli
...
entry point
2015-03-23 09:08:09 -07:00
Ilya Kreymer
e8db31d066
cli: improve wayback cli to take optional port, threads and working dir arguments
...
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
6a9a09d602
setup: add 'watchdog' as a dependency
2015-03-22 18:24:56 -07:00
Ilya Kreymer
ea460bb0f0
cdxj: support cdx json output from cdx server with output='json' (not yet default)
...
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
fe1c32c8f7
cdxj: support loading cdxj ( #76 )
...
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75 ) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
2015-03-19 12:36:49 -07:00
Ilya Kreymer
bfe590996b
auto-config: add support for loading from root ./static/ directory,
...
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
2015-03-17 19:05:39 -07:00
Ilya Kreymer
138aed3ddd
change version to 0.9.0b1
2015-03-16 19:15:00 -07:00
Ilya Kreymer
2f6780a576
rename for 0.9.0:
...
rename default templates package from ui/* templates to templates/*
rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
Ilya Kreymer
19b8650891
manager: templates: add collections manager ( #74 ) commands for adding, removing and listing
...
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
2015-03-16 16:55:06 -07:00
Ilya Kreymer
30454abb6b
metadata: add support for user-defined per-collection metadata! #78
...
metadata stored in wbrequest.user_metadata and available to all templates
collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
2015-03-15 21:24:15 -07:00
Ilya Kreymer
759d151551
tests: add test for directory auto collection loader,
...
collection manager and new 6-field minimal cdx format
2015-03-13 19:53:50 -07:00
Ilya Kreymer
60f33412ff
collections manager: add new collections manager, first pass #74
...
add cli 'wb-manager' tool
very preliminary, needs testing still
2015-02-25 13:19:20 -08:00
Ilya Kreymer
7c60bf17f7
bump version to 0.9.0-beta!
2015-02-24 16:54:49 -08:00
Ilya Kreymer
83f8d7d29b
bump version to 0.8.2
2015-02-22 22:51:23 -08:00
Ilya Kreymer
adeb8bfb27
bump version to 0.8.1, (fix blank spacing in changelist)
2015-02-20 02:02:34 -08:00
Ilya Kreymer
59630c08f6
bump version to 0.8.0!
2015-01-26 11:08:08 -08:00
Ilya Kreymer
48b7751f80
bump version to 0.7.6
...
jinja2: allow adding multiple packages to search path
2015-01-19 21:54:11 -05:00
Ilya Kreymer
7ece05d022
bump version to 0.7.5
...
update CHANGES
fix .gitattributes to use standard flags
2015-01-12 00:09:02 -08:00
Ilya Kreymer
1eb0f96f92
windows support work: fix loaders to use pathname2url to convert to
...
file:/// url, use urlopen to open file paths
fix some tests to use universal line breaks
2015-01-10 14:06:15 -08:00