Ilya Kreymer
85f093e356
Fix Query UI ( #278 )
...
* query fix:
setup: ensure all static files included in package_data recursively to add new query assets
test: add test for nested static asset
query: correctly display 0 captures, 'capture' and 'captures' text moved to Text block
2018-01-15 19:54:15 -08:00
Ilya Kreymer
61f825330c
Docs Update ( #256 )
...
* docs work:
- write warcserver and beginnings of recorder docs!
- add cdx api docs!
- add indexing docs
- refactor architecture section, remove readme
- update readme with better new features list, work-in-progress list
- add placeholder docs for apps, indexing
- remove unused readme
- update README with better docs link, features
2017-10-18 10:12:44 -07:00
Ilya Kreymer
903fa6c6a2
renaming pass:
...
- webagg->warcserver
- setup.py: packages and entry points
- templateview param: 'webrec.template_params' -> 'pywb.template_params'
2017-10-01 10:09:17 -07:00
Ilya Kreymer
582966bb2f
rewriterapp: add 'matchType=exact' to avoid edge case issues
...
setup: fix cdx-indexer cli entry point
2017-06-20 20:42:03 -04:00
Ilya Kreymer
58f39f0558
setup: update to warcio==1.2
...
add ensure_http_headers=True when reading WARC records
tests: fix pytest warnings, use webtest.TestApp instead of TestApp
2017-04-29 13:47:54 -07:00
Ilya Kreymer
26662f7df3
setup: generate current git_hash into autogenerated 'pywb.git_hash' file, add to .gitignore
2017-03-28 10:31:43 -07:00
Ilya Kreymer
544df71302
setup: use latest webtest again
...
tests: use geventwebserver for LiveServerTests instead of separate process
2017-03-10 11:19:27 -08:00
Ilya Kreymer
d4321792b7
tests: convert test_inputreq to use werkzeug (same as the app), remove bottle from test dependencies
2017-03-08 23:09:19 -08:00
Ilya Kreymer
e86e3e6d32
build process: simplify build process by moving essential deps to requirements.txt, and extras to extra_requirements.txt
...
setup.py just loads from requirements.txt
Dockerfile pip installs requirements, then extra requirements for improved cacheing
travis runs setup install, then installs extra requirements
2017-03-08 17:05:29 -08:00
Ilya Kreymer
0784e4e5aa
spin-off warcio!
...
update imports to point to warcio
warcio rename fixes:
- ArcWarcRecord.stream -> raw_stream
- ArcWarcRecord.status_headers -> http_headers
- ArchiveLoadFailed single param init
2017-03-07 10:58:00 -08:00
Ilya Kreymer
2a7da54be9
Merge branch 'master' into new-pywb
2017-02-17 18:49:01 -08:00
Ilya Kreymer
31bf7a47f1
new-wayback cli script, using new FrontEndApp (rewriting) + AutoConfigApp (config-driven aggregator)
...
support for dynamic collections: check all .cdxj files in /<coll>/indexes/*.cdxj when accessing /<coll>
support for fixed routes: specified in config.yaml as per https://github.com/ikreymer/pywb/wiki/Distributed-Archive-Config
werkzeug routing in FrontEndApp: default query, replay, search pages working
route listing: /_coll_info.json for listing fixed + dynamic routes
autoindexing enabled, indexing WARCs added to archives directory to .cdxj index
Addresses #196
2017-02-17 18:04:07 -08:00
Ilya Kreymer
60f3c0a213
setup: further update trove classifiers, add python 2 and 3, switch to production stable, closes #208
2017-02-16 15:12:22 -08:00
Ilya Kreymer
77960c1311
setup: bound webtest for 2.6 support
2017-02-16 14:32:14 -08:00
Ilya Kreymer
7f95396be0
setup: use jinja2<2.9 for now
2017-02-16 11:29:30 -08:00
Ilya Kreymer
58b141bd53
add python 3 classifiers ( #208 )
2017-02-16 11:02:53 -08:00
Ilya Kreymer
9773eba47d
setup reqs: use webassets==0.12.1 (with pyinstaller support), remove dependency on custom branch
2017-01-26 01:37:57 -08:00
Ilya Kreymer
84796ba810
setup req: fix jinja<2.9 for now due to issues in 2.9+
2017-01-26 01:14:10 -08:00
Ilya Kreymer
2d54bb87be
setup.py: ensure gevent monkey-patch is called before running tests with python setup.py test
2017-01-26 00:37:35 -08:00
raffaele messuti
524d9bfd26
portalocker for file locking check instead of fcntl. more portable on windows
2016-12-26 10:27:20 +01:00
Ilya Kreymer
0e414acfda
setup: remove pyamf as default dep for now
2016-12-21 17:15:41 -08:00
Ilya Kreymer
3b82416ad3
setup: add specific dependencies for webassets, pyamf
2016-12-21 16:11:48 -08:00
Ilya Kreymer
50a3353da3
wsgi server: default to gevent-based wsgi server for all cmd line server apps, add -s command for specifying server #201
...
cli: add 'webagg-server' cli command for running new webagg system
tests: fix cli test for gevent server
2016-12-09 16:46:33 -08:00
Ilya Kreymer
e37900b9c6
tests: add test dependency, remove 2.6 from travis
2016-11-11 11:03:16 -08:00
Ilya Kreymer
8765de4fe7
refactor: updated dependencies, remove watchdog, add gevent and webassets
...
update tests, tests should pass for python 2 and 3!
2016-11-11 10:32:19 -08:00
Ilya Kreymer
6b4b038471
refactor: fix pywb.webagg package paths
...
all webagg tests working!
move testdata cdxj into sample_archive, remove rest (duplicates) #200
2016-11-08 14:30:09 -08:00
Ilya Kreymer
457a1a564c
bufferedreader: support brotli decompression
...
rewrite: handle Content-Encoding: br using brotli decompressor
setup: add brotlipy as dependency
2016-06-15 01:37:29 -04:00
Ilya Kreymer
0f530a3e0e
dependencies: remove pyamf, update to latest surt (0.3.0)
2016-06-12 00:44:52 -04:00
Ilya Kreymer
87da25c703
post request mapping improvements: work on #178 , including:
...
- mapping multipart/form-data same as x-www-form-urlencoded
- parsing application/x-amf with pyamf
- RewriteContentAMF for rewriting AMF response to match request
- default encoding of other POST data as base64 encoded __wb_post_data param
2016-05-06 10:19:08 -07:00
Ilya Kreymer
e8c77c0538
encoding: encode before quote
...
setup: enable zip_safe=True again
2016-04-30 15:15:35 -07:00
Ilya Kreymer
c1bdeac92b
redis: fix redis key lookup, add tests for zrangebylex with new fakeredis
2016-03-09 18:33:04 -08:00
Ilya Kreymer
b7008920de
fix setup.py typo
2016-02-16 16:14:10 -08:00
Ilya Kreymer
3c85f7b7ac
py3: make pywb.utils work with python 3!
2016-02-16 14:52:20 -08:00
Ilya Kreymer
9f77ac50e7
setup.py: better fix for test suite with latest setuptool
2015-11-26 00:59:57 -08:00
Ilya Kreymer
6c76675fe7
setup: fix setup.py again
2015-11-04 16:33:19 -08:00
Ilya Kreymer
c107dd3af1
setup: fix 'setup.py test' with new setuptools changes, fix surt==0.2
2015-11-04 16:16:10 -08:00
Jack Cushman
633eb31f57
Use webencodings to encode head_insert_str.
2015-10-22 16:40:59 -04:00
Ilya Kreymer
d84185aea6
setup: remove waitress from require pkgs, now optional
2015-07-31 02:49:03 -07:00
Ilya Kreymer
f0359877f0
youtube-dl: remove from dependency, installation is optional. Return 404 if attempting live
...
proxy of videos and youtube-dl is not available (the only use case).
HTTPParser wrapping logic no longer needed in latest versions
Modify tests to only run if youtube-dl is installed in cases where it is not available #118
2015-06-27 16:11:59 -07:00
Ilya Kreymer
cda7705075
split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth cli as
...
the 'certauth' tool superceeds this functionality. (#90 ).
To use https proxy mode, 'pip install certauth' is required. (update travis config)
2015-03-29 17:38:57 -07:00
Ilya Kreymer
0faa6aac3e
setup: set version in pywb __init__.py
2015-03-23 11:04:41 -07:00
Ilya Kreymer
df76bc3500
cli: change cdx-server and live-rewrite-server to go through shared cli
...
entry point
2015-03-23 09:08:09 -07:00
Ilya Kreymer
e8db31d066
cli: improve wayback cli to take optional port, threads and working dir arguments
...
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
6a9a09d602
setup: add 'watchdog' as a dependency
2015-03-22 18:24:56 -07:00
Ilya Kreymer
ea460bb0f0
cdxj: support cdx json output from cdx server with output='json' (not yet default)
...
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
fe1c32c8f7
cdxj: support loading cdxj ( #76 )
...
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75 ) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
2015-03-19 12:36:49 -07:00
Ilya Kreymer
bfe590996b
auto-config: add support for loading from root ./static/ directory,
...
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
2015-03-17 19:05:39 -07:00
Ilya Kreymer
138aed3ddd
change version to 0.9.0b1
2015-03-16 19:15:00 -07:00
Ilya Kreymer
2f6780a576
rename for 0.9.0:
...
rename default templates package from ui/* templates to templates/*
rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
Ilya Kreymer
19b8650891
manager: templates: add collections manager ( #74 ) commands for adding, removing and listing
...
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
2015-03-16 16:55:06 -07:00
