backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 08:04:49 +01:00
Code Issues Projects Releases Activity
pywb/pywb/warc
History
Ilya Kreymer 0784e4e5aa spin-off warcio! 
update imports to point to warcio
warcio rename fixes:
- ArcWarcRecord.stream -> raw_stream
- ArcWarcRecord.status_headers -> http_headers
- ArchiveLoadFailed single param init
2017-03-07 10:58:00 -08:00
..
test
spin-off warcio!
2017-03-07 10:58:00 -08:00
__init__.py
pywb 0.2!
2014-02-17 10:01:09 -08:00
archiveindexer.py
spin-off warcio!
2017-03-07 10:58:00 -08:00
blockrecordloader.py
spin-off warcio!
2017-03-07 10:58:00 -08:00
cdxindexer.py
warc & recorder refactor: split BaseWARCWriter from MultiWARCWriter, move to warc/warcwriter.py, recorder/multifilewarcwriter.py
2017-03-01 14:18:44 -08:00
pathresolvers.py
encoding: cdx: only quote-encode 'url'
2016-04-30 14:38:48 -07:00
README.md
automatic record (warc/arc) format detection and decompression if needed.
2014-02-19 00:13:15 -08:00
resolvingloader.py
spin-off warcio!
2017-03-07 10:58:00 -08:00

README.md

pywb.warc

This is the WARC/ARC record loading component of pywb wayback tool suite. The package provides the following facilities:

  • Resolve relative WARC/ARC filenames to a full path based on configurable resolvers

  • Resolve 'revisit' records from provided index to find a full record with headers and payload content

  • Load WARC/ARC records either locally or via http using http 1.1 range requests

When loading archived content, the format type (WARC vs ARC) and compressed ARCs/WARCs are decompressed automatically. No assumption is made about format based on filename, content type or other external parameters other than the content itself.

Tests

This package will includes a test suite for loading a variety of WARC and ARC records.

Tests so far:

  • Compressed WARC, ARC Records
  • Uncompressed ARC Records
  • Compressed WARC created by wget 1.14
  • Same Url revisit record resolving

TODO:

  • Different url revisit record resolving