pywb/README.rst

Webrecorder pywb 2.1
====================

.. image:: https://travis-ci.org/webrecorder/pywb.svg?branch=master
      :target: https://travis-ci.org/webrecorder/pywb
.. image:: https://ci.appveyor.com/api/projects/status/qxnbunw65o929599/branch/master?svg=true
      :target: https://ci.appveyor.com/project/webrecorder/pywb/branch/master
.. image:: https://codecov.io/gh/webrecorder/pywb/branch/master/graph/badge.svg
      :target: https://codecov.io/gh/webrecorder/pywb

Web Archiving Tools for All
---------------------------

`View the full pywb documentation <https://pywb.readthedocs.org>`_

**pywb** is a Python (2 and 3) web archiving toolkit for replaying web archives large and small as accurately as possible.
The toolkit now also includes new features for creating high-fidelity web archives.

This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit
that is used by other web archives, including the traditional "Wayback Machine" functionality.


New Features
^^^^^^^^^^^^

The 2.0 release included a major overhaul of pywb and introduces many new features, including the following:

* Dynamic multi-collection configuration system with no-restart updates.

* New recording capability to create new web archives from the live web or other archives.

* Componentized architecture with standalone Warcserver, Recorder and Rewriter components.

* Support for Memento API aggregation and fallback chains for querying multiple remote and local archival sources.

* HTTP/S Proxy Mode with customizable certificate authority for proxy mode recording and replay.

* Flexible rewriting system with pluggable rewriters for different content-types.

* Significantly improved client-side rewriting to handle most modern web sites.

* Improved 'calendar' query UI, grouping results by year and month, and updated replay banner.


Please see the `full documentation <https://pywb.readthedocs.org>`_ for more detailed info on all these features.


Installation
------------

To run and install locally you can:

* Install with ``python setup.py install``

* Run tests with ``python setup.py test``

* Run Wayback with ``wayback`` (see docs for info on how to setup collections)

* Build docs locally with:  ``cd docs; make html``. (The docs will be built in ``./_build/html/index.html``)


Consult the local or `online docs <https://pywb.readthedocs.org>`_ for latest usage and configuration details.


Contributions & Bug Reports
---------------------------

Users are encouraged to fork and contribute to this project to keep improving web archiving tools.

A few key features are high on list of priorities, but have not yet been implemented, including:

* Url Exclusion System

* UI Improvements

If you are interested in contributing, especially to any of these areas, please let us know!

Otherwise, please take a look at `list of current issues <https://github.com/webrecorder/pywb/issues>`_ and feel free to open new ones about any aspect of pywb, including the new documentation.
README: update 2.0 -> 2.1 (#396) cli: fix typo in enable-auto-fetch, add test 2018-10-23 09:58:10 -07:00			`Webrecorder pywb 2.1`
client-side top-frame notification fix: - send_top_message() to wrap all postMessage calls - url change, hash change, history change notifications only sent if window is top replay frame, cookie notification sent for all windows - don't send url change notification if 'about:blank' or 'javascript:' url bump version to 2.0.2 2018-02-13 13:49:10 -08:00			`====================`
add README.rst 2014-03-10 19:01:20 -07:00
README: update links to master 2018-01-29 18:30:54 -08:00			`.. image:: https://travis-ci.org/webrecorder/pywb.svg?branch=master`
Add CodeCov (#282) coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__ README: badge update, add appveyor, codecov badges 2018-01-17 09:59:51 -08:00			`:target: https://travis-ci.org/webrecorder/pywb`
README: update links to master 2018-01-29 18:30:54 -08:00			`.. image:: https://ci.appveyor.com/api/projects/status/qxnbunw65o929599/branch/master?svg=true`
			`:target: https://ci.appveyor.com/project/webrecorder/pywb/branch/master`
			`.. image:: https://codecov.io/gh/webrecorder/pywb/branch/master/graph/badge.svg`
Add CodeCov (#282) coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__ README: badge update, add appveyor, codecov badges 2018-01-17 09:59:51 -08:00			`:target: https://codecov.io/gh/webrecorder/pywb`
video work: live rewrite pings proxy with full rewrite, proxies direct range request reorg rangecache to support is_range() check, yt-specific logic (experimental) wombat: add date override (experimental) bump tentative version to 0.7.0! yt replays work with native player! (though still issues remain) 2014-11-04 22:11:25 -08:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Web Archiving Tools for All`
			`---------------------------`
add README.rst 2014-03-10 19:01:20 -07:00
README: update 2.0 -> 2.1 (#396) cli: fix typo in enable-auto-fetch, add test 2018-10-23 09:58:10 -07:00			`View the full pywb documentation <https://pywb.readthedocs.org>`_
readme tweaks and typo fixes 2015-03-17 01:06:06 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`pywb is a Python (2 and 3) web archiving toolkit for replaying web archives large and small as accurately as possible.`
			`The toolkit now also includes new features for creating high-fidelity web archives.`

			`This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit`
			`that is used by other web archives, including the traditional "Wayback Machine" functionality.`


			`New Features`
			`^^^^^^^^^^^^`

README: update 2.0 -> 2.1 (#396) cli: fix typo in enable-auto-fetch, add test 2018-10-23 09:58:10 -07:00			`The 2.0 release included a major overhaul of pywb and introduces many new features, including the following:`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			`* Dynamic multi-collection configuration system with no-restart updates.`

			`* New recording capability to create new web archives from the live web or other archives.`

			`* Componentized architecture with standalone Warcserver, Recorder and Rewriter components.`

Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`* Support for Memento API aggregation and fallback chains for querying multiple remote and local archival sources.`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`* HTTP/S Proxy Mode with customizable certificate authority for proxy mode recording and replay.`
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			`* Flexible rewriting system with pluggable rewriters for different content-types.`

			`* Significantly improved client-side rewriting to handle most modern web sites.`

docs/README: fix typos, add changes for 2.0.1 2018-02-10 11:48:50 -08:00			`* Improved 'calendar' query UI, grouping results by year and month, and updated replay banner.`
Docs and README Update for 2.0.0 (#277) * docs and version update: - add docs for compatibility features - add docs for memento - updat rewriter docs - bump version to 2.0.0, update README, and changelist 2018-01-11 21:34:04 -08:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00
			Please see the `full documentation <https://pywb.readthedocs.org>`_ for more detailed info on all these features.


			`Installation`
			`------------`

			`To run and install locally you can:`
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Install with ``python setup.py install``
Update README to 0.9.1, add cdx api link, fix typo 2015-03-25 12:06:05 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Run tests with ``python setup.py test``
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
config and docs work: (#255) config and docs work: - autoindexing now set in config via 'autoindex: <secs>' option - autoindexing only runs in first uwsgi worker if in uwsgi - recorder config: rename props to 'rollover_' to match docs - docs: write configuring.rst section for recording mode, autoindexing and proxy mode! - update README for new pywb release, point to new docs! 2017-10-15 22:47:23 -07:00			* Run Wayback with ``wayback`` (see docs for info on how to setup collections)
Update README.rst update for 0.4.0 2014-05-30 10:29:22 -07:00
Docs and README Update for 2.0.0 (#277) * docs and version update: - add docs for compatibility features - add docs for memento - updat rewriter docs - bump version to 2.0.0, update README, and changelist 2018-01-11 21:34:04 -08:00			* Build docs locally with: ``cd docs; make html``. (The docs will be built in ``./_build/html/index.html``)
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00

			Consult the local or `online docs <https://pywb.readthedocs.org>`_ for latest usage and configuration details.
versions and readme: update version to 0.30.0, update README with python 2 and 3 support 2016-03-10 12:51:14 -08:00
Update README with News! 2014-10-19 08:32:11 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Contributions & Bug Reports`
			`---------------------------`
Update README with News! 2014-10-19 08:32:11 -07:00
Docs Update (#256) * docs work: - write warcserver and beginnings of recorder docs! - add cdx api docs! - add indexing docs - refactor architecture section, remove readme - update readme with better new features list, work-in-progress list - add placeholder docs for apps, indexing - remove unused readme - update README with better docs link, features 2017-10-18 10:12:44 -07:00			`Users are encouraged to fork and contribute to this project to keep improving web archiving tools.`
Update README with News! 2014-10-19 08:32:11 -07:00
Text tweaks/Dockerfile update (#288) README: update features list, contributing section, fix typos docs: update features list, fix wording, add more links to other sections, fix typos renaming: change 'ikreymer/pywb' -> 'webrecorder/pywb', add Rhizome to copyright statement Dockerfile: remove deprecated MAINTAINER, add 'ARG PYTHON' to support custom base python image 2018-01-30 07:49:54 -08:00			`A few key features are high on list of priorities, but have not yet been implemented, including:`

			`* Url Exclusion System`

			`* UI Improvements`

			`If you are interested in contributing, especially to any of these areas, please let us know!`

			Otherwise, please take a look at `list of current issues <https://github.com/webrecorder/pywb/issues>`_ and feel free to open new ones about any aspect of pywb, including the new documentation.

Add Gratipay link 2014-10-18 17:00:33 -07:00