mirror of
https://github.com/webrecorder/pywb.git
synced 2025-03-15 00:03:28 +01:00
ec0df7b9ae
- Split wombat and auto-fetch worker into two files (proxy mode and non-proxy mode) - Renamed preservationWorker to autoFetchWorker in order to better convey what it does - Root config file control over including wombat and auto-fetch worker in proxy or non-proxy mode - Added additional proxy mode + auto-fetch worker only route for fetching the auto-fetch worker code nicely for CORS - templateview: add 'tobool' formatter to more cleanly format python bools to JS 'true'/'false' - proxy options: config and command line: 'use_auto_fetch_worker' and '--proxy-with-auto-fetch' 'use_wombat' and '--proxy-with-wombat' - head_insert.html: only include wombat in proxy mode when use_wombat or use_auto_fetch_worker are set. - wombatProxyMode.js: slimmed down wombat for proxy mode only including auto-fetch support. - more consistent naming: rename 'preserveWorker' and 'autoArchive' to 'auto-fetch' Updated tests: - test_wbrequestresponse.py: added tests covering constructor defaults, _init_derived, options_response, json_response, encode_stream, text_stream - test_auto_colls.py: fixed broken test test_more_custom_templates, reason using ujson now not json so spacing was off - test_proxy.py: updated existing tests to reflect splitting wombat into proxy and non-proxy mode, added tests covering auto-fetch worker specific endpoints in proxy mode removed duplicate addons key in .travis.yml - test_cli.py: updated to properly test the cli with these changes added ultrajon dep to tests_require in setup.py to reflect its usage by wbrequestresponse.py Fully documented: - cli.py - frontendapp.py - templateview.py - wbrequestresponse.py Removed duplicate addons key in .travis.yml Added ultrajson dependency to tests_require in setup.py to reflect its usage by wbrequestresponse.py Fixes #371
Webrecorder pywb 2.0
====================
.. image:: https://travis-ci.org/webrecorder/pywb.svg?branch=master
:target: https://travis-ci.org/webrecorder/pywb
.. image:: https://ci.appveyor.com/api/projects/status/qxnbunw65o929599/branch/master?svg=true
:target: https://ci.appveyor.com/project/webrecorder/pywb/branch/master
.. image:: https://codecov.io/gh/webrecorder/pywb/branch/master/graph/badge.svg
:target: https://codecov.io/gh/webrecorder/pywb
Web Archiving Tools for All
---------------------------
`View the full pywb 2.0 documentation <https://pywb.readthedocs.org>`_
**pywb** is a Python (2 and 3) web archiving toolkit for replaying web archives large and small as accurately as possible.
The toolkit now also includes new features for creating high-fidelity web archives.
This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit
that is used by other web archives, including the traditional "Wayback Machine" functionality.
New Features
^^^^^^^^^^^^
The 2.0 release includes a major overhaul of pywb and introduces many new features, including the following:
* Dynamic multi-collection configuration system with no-restart updates.
* New recording capability to create new web archives from the live web or other archives.
* Componentized architecture with standalone Warcserver, Recorder and Rewriter components.
* Support for Memento API aggregation and fallback chains for querying multiple remote and local archival sources.
* HTTP/S Proxy Mode with customizable certificate authority for proxy mode recording and replay.
* Flexible rewriting system with pluggable rewriters for different content-types.
* Significantly improved client-side rewriting to handle most modern web sites.
* Improved 'calendar' query UI, grouping results by year and month, and updated replay banner.
Please see the `full documentation <https://pywb.readthedocs.org>`_ for more detailed info on all these features.
Installation
------------
To run and install locally you can:
* Install with ``python setup.py install``
* Run tests with ``python setup.py test``
* Run Wayback with ``wayback`` (see docs for info on how to setup collections)
* Build docs locally with: ``cd docs; make html``. (The docs will be built in ``./_build/html/index.html``)
Consult the local or `online docs <https://pywb.readthedocs.org>`_ for latest usage and configuration details.
Contributions & Bug Reports
---------------------------
Users are encouraged to fork and contribute to this project to keep improving web archiving tools.
A few key features are high on list of priorities, but have not yet been implemented, including:
* Url Exclusion System
* UI Improvements
If you are interested in contributing, especially to any of these areas, please let us know!
Otherwise, please take a look at `list of current issues <https://github.com/webrecorder/pywb/issues>`_ and feel free to open new ones about any aspect of pywb, including the new documentation.