pywb/README.rst

Webrecorder pywb 2.4
====================

.. image:: https://travis-ci.org/webrecorder/pywb.svg?branch=master
      :target: https://travis-ci.org/webrecorder/pywb
.. image:: https://ci.appveyor.com/api/projects/status/qxnbunw65o929599/branch/master?svg=true
      :target: https://ci.appveyor.com/project/webrecorder/pywb/branch/master
.. image:: https://codecov.io/gh/webrecorder/pywb/branch/master/graph/badge.svg
      :target: https://codecov.io/gh/webrecorder/pywb

Web Archiving Tools for All
---------------------------

`View the full pywb documentation <https://pywb.readthedocs.org>`_

**pywb** is a Python (2 and 3) web archiving toolkit for replaying web archives large and small as accurately as possible.
The toolkit now also includes new features for creating high-fidelity web archives.

This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit
that is used by other web archives, including the traditional "Wayback Machine" functionality.


New Features
^^^^^^^^^^^^

The 2.x release included a major overhaul of pywb and introduces many new features, including the following:

* Dynamic multi-collection configuration system with no-restart updates.

* New recording capability to create new web archives from the live web or other archives.

* Componentized architecture with standalone Warcserver, Recorder and Rewriter components.

* Support for Memento API aggregation and fallback chains for querying multiple remote and local archival sources.

* HTTP/S Proxy Mode with customizable certificate authority for proxy mode recording and replay.

* Flexible rewriting system with pluggable rewriters for different content-types.

* Standalone, modular `client-side rewriting system (wombat.js) <https://github.com/webrecorder/wombat>`_ to handle most modern web sites.

* Improved 'calendar' query UI with incremental loading, grouping results by year and month, and updated replay banner.

* New in 2.4: Extensible UI customizations system for modifying all aspects of the UI.

* New in 2.4: Robust access control system for blocking or excluding URLs, by prefix or by exact match.


Please see the `full documentation <https://pywb.readthedocs.org>`_ for more detailed info on all these features.


Installation
------------

To run and install locally you can:

* Install with ``python setup.py install``

* Run tests with ``python setup.py test``

* Run Wayback with ``wayback`` (see docs for info on how to setup collections)

* Build docs locally with:  ``cd docs; make html``. (The docs will be built in ``./_build/html/index.html``)


Consult the local or `online docs <https://pywb.readthedocs.org>`_ for latest usage and configuration details.


Contributions & Bug Reports
---------------------------

Users are encouraged to fork and contribute to this project to keep improving web archiving tools.

If you are interested in contributing, especially to any of these areas, please let us know!

Otherwise, please take a look at `list of current issues <https://github.com/webrecorder/pywb/issues>`_ and feel free to open new ones about any aspect of pywb, including the new documentation.
Docs, custom metadata improvements (#509) * metadata/coll_config: don't confuse user metadata with collection config, don't display collection config settings as metadata (ukwa/ukwa-pywb#47) - for collection template, add separate 'coll_config' dict, keep user metadata only in 'metadata' dict (default to empty) - for static collections, assume metadata is in the 'metadata' dict of collection config - for dynamic collections, load metadata.yaml into 'metadata' dict - ensure 'metadata' key is passed to frame_insert - ensure 'metadata' added consistently in framed and non-framed mode - tests: update tests to ensure metadata is added consistently - fuzzymatch: don't match 204 OPTIONS responses, update fuzzymatcher test * documentation - add documentation for metadata in ui-customization, rebuild docs, - add link to ui customization from configuring - work on access control docs * fixed small typo's in ui-customization.rst * frontendapp: fix doc string - misc: remove warning on urllib3 Retry init - set version to pywb 2.4.0rc0 Co-Authored-By: John Berlin <n0tan3rd@gmail.com> 2019-10-27 01:39:52 +01:00			`Webrecorder pywb 2.4`
client-side top-frame notification fix: - send_top_message() to wrap all postMessage calls - url change, hash change, history change notifications only sent if window is top replay frame, cookie notification sent for all windows - don't send url change notification if 'about:blank' or 'javascript:' url bump version to 2.0.2 2018-02-13 13:49:10 -08:00			`====================`
add README.rst 2014-03-10 19:01:20 -07:00
README: update links to master 2018-01-29 18:30:54 -08:00			`.. image:: https://travis-ci.org/webrecorder/pywb.svg?branch=master`
Add CodeCov (#282) coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__ README: badge update, add appveyor, codecov badges 2018-01-17 09:59:51 -08:00			`:target: https://travis-ci.org/webrecorder/pywb`
README: update links to master 2018-01-29 18:30:54 -08:00			`.. image:: https://ci.appveyor.com/api/projects/status/qxnbunw65o929599/branch/master?svg=true`
			`:target: https://ci.appveyor.com/project/webrecorder/pywb/branch/master`
			`.. image:: https://codecov.io/gh/webrecorder/pywb/branch/master/graph/badge.svg`
Add CodeCov (#282) coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__ README: badge update, add appveyor, codecov badges 2018-01-17 09:59:51 -08:00			`:target: https://codecov.io/gh/webrecorder/pywb`
video work: live rewrite pings proxy with full rewrite, proxies direct range request reorg rangecache to support is_range() check, yt-specific logic (experimental) wombat: add date override (experimental) bump tentative version to 0.7.0! yt replays work with native player! (though still issues remain) 2014-11-04 22:11:25 -08:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Web Archiving Tools for All`
			`---------------------------`
add README.rst 2014-03-10 19:01:20 -07:00
README: update 2.0 -> 2.1 (#396) cli: fix typo in enable-auto-fetch, add test 2018-10-23 09:58:10 -07:00			`View the full pywb documentation <https://pywb.readthedocs.org>`_
readme tweaks and typo fixes 2015-03-17 01:06:06 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`pywb is a Python (2 and 3) web archiving toolkit for replaying web archives large and small as accurately as possible.`
			`The toolkit now also includes new features for creating high-fidelity web archives.`

			`This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit`
			`that is used by other web archives, including the traditional "Wayback Machine" functionality.`


			`New Features`
			`^^^^^^^^^^^^`

README: Update for 2.3 2019-07-09 19:37:03 -07:00			`The 2.x release included a major overhaul of pywb and introduces many new features, including the following:`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			`* Dynamic multi-collection configuration system with no-restart updates.`

			`* New recording capability to create new web archives from the live web or other archives.`

			`* Componentized architecture with standalone Warcserver, Recorder and Rewriter components.`

Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`* Support for Memento API aggregation and fallback chains for querying multiple remote and local archival sources.`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`* HTTP/S Proxy Mode with customizable certificate authority for proxy mode recording and replay.`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			`* Flexible rewriting system with pluggable rewriters for different content-types.`

README: Update for 2.3 2019-07-09 19:37:03 -07:00			* Standalone, modular `client-side rewriting system (wombat.js) <https://github.com/webrecorder/wombat>`_ to handle most modern web sites.
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
Docs: Fix access controls and ui customizations docs links (#513) * docs: ensure docs added to access controls, fix typos * begin changelist for 2.4.0 2019-10-31 16:56:36 -07:00			`* Improved 'calendar' query UI with incremental loading, grouping results by year and month, and updated replay banner.`
Docs and README Update for 2.0.0 (#277) * docs and version update: - add docs for compatibility features - add docs for memento - updat rewriter docs - bump version to 2.0.0, update README, and changelist 2018-01-11 21:34:04 -08:00
Docs: Fix access controls and ui customizations docs links (#513) * docs: ensure docs added to access controls, fix typos * begin changelist for 2.4.0 2019-10-31 16:56:36 -07:00			`* New in 2.4: Extensible UI customizations system for modifying all aspects of the UI.`

			`* New in 2.4: Robust access control system for blocking or excluding URLs, by prefix or by exact match.`
Docs, custom metadata improvements (#509) * metadata/coll_config: don't confuse user metadata with collection config, don't display collection config settings as metadata (ukwa/ukwa-pywb#47) - for collection template, add separate 'coll_config' dict, keep user metadata only in 'metadata' dict (default to empty) - for static collections, assume metadata is in the 'metadata' dict of collection config - for dynamic collections, load metadata.yaml into 'metadata' dict - ensure 'metadata' key is passed to frame_insert - ensure 'metadata' added consistently in framed and non-framed mode - tests: update tests to ensure metadata is added consistently - fuzzymatch: don't match 204 OPTIONS responses, update fuzzymatcher test * documentation - add documentation for metadata in ui-customization, rebuild docs, - add link to ui customization from configuring - work on access control docs * fixed small typo's in ui-customization.rst * frontendapp: fix doc string - misc: remove warning on urllib3 Retry init - set version to pywb 2.4.0rc0 Co-Authored-By: John Berlin <n0tan3rd@gmail.com> 2019-10-27 01:39:52 +01:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			Please see the `full documentation <https://pywb.readthedocs.org>`_ for more detailed info on all these features.


			`Installation`
			`------------`

			`To run and install locally you can:`
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Install with ``python setup.py install``
Update README to 0.9.1, add cdx api link, fix typo 2015-03-25 12:06:05 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Run tests with ``python setup.py test``
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Run Wayback with ``wayback`` (see docs for info on how to setup collections)
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
Docs and README Update for 2.0.0 (#277) * docs and version update: - add docs for compatibility features - add docs for memento - updat rewriter docs - bump version to 2.0.0, update README, and changelist 2018-01-11 21:34:04 -08:00			* Build docs locally with: ``cd docs; make html``. (The docs will be built in ``./_build/html/index.html``)
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00

			Consult the local or `online docs <https://pywb.readthedocs.org>`_ for latest usage and configuration details.
versions and readme: update version to 0.30.0, update README with python 2 and 3 support 2016-03-10 12:51:14 -08:00
Update README with News! 2014-10-19 08:32:11 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Contributions & Bug Reports`
			`---------------------------`
Update README with News! 2014-10-19 08:32:11 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Users are encouraged to fork and contribute to this project to keep improving web archiving tools.`
Update README with News! 2014-10-19 08:32:11 -07:00
Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`If you are interested in contributing, especially to any of these areas, please let us know!`

			Otherwise, please take a look at `list of current issues <https://github.com/webrecorder/pywb/issues>`_ and feel free to open new ones about any aspect of pywb, including the new documentation.

Add Gratipay link 2014-10-18 17:00:33 -07:00