1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

1131 Commits

Author SHA1 Message Date
Ilya Kreymer
be5139b635 fix tests for coll listing, #78
config override: when loading from coll-specific config.yaml, resolve
relative paths to that collection, not to root #55
2015-03-15 22:23:08 -07:00
Ilya Kreymer
30454abb6b metadata: add support for user-defined per-collection metadata! #78
metadata stored in wbrequest.user_metadata and available to all templates

collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
2015-03-15 21:24:15 -07:00
Ilya Kreymer
b417b47835 collections manager: support for merge when adding warc, explicit --index-warcs
option to index and merge instead of reindexing whole dir, #74
additional testing for recursive indexing, index merge
timeutils: add timestamp20_now() function
2015-03-14 14:56:15 -07:00
Ilya Kreymer
759d151551 tests: add test for directory auto collection loader,
collection manager and new 6-field minimal cdx format
2015-03-13 19:53:50 -07:00
Ilya Kreymer
1ba24de357 Merge branch 'develop' into config-work 2015-03-13 11:53:27 -07:00
Ilya Kreymer
b4b92482ad Merge branch 'develop' for 0.8.3 2015-03-13 11:06:52 -07:00
Ilya Kreymer
b2ce3feb80 readme fix 2015-03-13 11:05:32 -07:00
Ilya Kreymer
3e3794d4dc Update CHANGES for 0.8.3 2015-03-13 11:04:37 -07:00
Ilya Kreymer
24021fcd57 html rewrite: add trailing slash for <base> tag rewrite if url is a scheme://host
with no path component #77
cleanup: remove unused code path for tags with no rewriting -- all tags
now checked for dynamic attrs which may need rewriting
update tests, including live rewrite test dependent on live site (FB)
2015-03-13 10:53:57 -07:00
Ilya Kreymer
fe1683da56 indexing: for minimal index, use a single -m flag to create a 6 field index.
minimal index also skips parsing contents of warc/arc records altogether
add cli docs for minimal index, tracked via #75
2015-03-07 11:56:17 -08:00
Ilya Kreymer
499e21233e statusandheaders: make protocol check case-insensitive, eg. accept HTTP/1.0 and http/1.0 for better compatibility 2015-03-07 11:37:06 -08:00
Ilya Kreymer
5aa497dc68 Merge branch 'develop' into config-work 2015-03-06 21:09:21 -08:00
Ilya Kreymer
1fb631870b wb_frame: fix extra slash typo in replaced frame url 2015-03-05 17:04:44 -08:00
Ilya Kreymer
f2d7bd074a bump version to 0.8.3
cookie rewrite: remove 'secure' flag if present
2015-03-05 16:18:56 -08:00
Ilya Kreymer
1eadd35598 Merge branch 'develop' for 0.8.2 2015-02-28 09:05:09 -08:00
Ilya Kreymer
6c8cb806d9 update 0.8.2 changelist, minor fixes 2015-02-28 09:04:15 -08:00
Ilya Kreymer
48eab2662d cdx indexer: refactor indexer into mixins for differnt formats for easier customization 2015-02-25 16:45:47 -08:00
Ilya Kreymer
ee1fabf600 config fix: check for existance of root 'collections dir', #55 2015-02-25 13:51:12 -08:00
Ilya Kreymer
11c8cc92f3 add beta to README 2015-02-25 13:33:42 -08:00
Ilya Kreymer
671f45f69f cdx indexing: wrap record iterator global functions in class DefaultRecordIter to allow for better extensibility
add 'minimal' option to skip digest/mime/status extraction only include minimal data (url+timestamp)
cdx-indexer: add -6 option to create 6-field index
2015-02-25 13:31:37 -08:00
Ilya Kreymer
1d4c54deaa frames ui: update frames to use <!DOCTYPE html>, improved css and html5 compatibility 2015-02-25 13:25:05 -08:00
Ilya Kreymer
60f33412ff collections manager: add new collections manager, first pass #74
add cli 'wb-manager' tool
very preliminary, needs testing still
2015-02-25 13:19:20 -08:00
Ilya Kreymer
69613a0e25 tests: disable 'invalid config' test as its no longer applicable, fix default banner to just 'banner.html' 2015-02-25 13:18:32 -08:00
Ilya Kreymer
5c67782a2c config system: some fixes for auto-init, add trailing '/' for dir paths, #55 2015-02-25 13:15:48 -08:00
Ilya Kreymer
7c60bf17f7 bump version to 0.9.0-beta! 2015-02-24 16:54:49 -08:00
Ilya Kreymer
e39d6e207c config & collections: auto static path and templates working! #55 2015-02-24 14:32:51 -08:00
Ilya Kreymer
a932235f85 Merge branch 'develop' into config-work 2015-02-24 10:40:58 -08:00
Ilya Kreymer
cb857df125 memento: fix MementoTimemapView to have consistent signature with other query views 2015-02-24 10:35:49 -08:00
Ilya Kreymer
39824711f0 memento tweak: ensure rel=memento link for timegate uses exact in Location (cdx original) as opposed to url from request 2015-02-23 23:21:39 -08:00
Ilya Kreymer
435fa390ed config system: initial work on automated directory-convention based config!
config.yaml file now optional, add default_config.yaml which for default settings #55
2015-02-23 21:59:41 -08:00
Ilya Kreymer
5d80d2d891 replay: change strip_scheme() to strip_scheme_www() to also strip away www. prefix for self-redirect checking, #73 2015-02-22 22:51:35 -08:00
Ilya Kreymer
83f8d7d29b bump version to 0.8.2 2015-02-22 22:51:23 -08:00
Ilya Kreymer
de40e2920a update README for 0.8.1 2015-02-21 14:29:40 -08:00
Ilya Kreymer
7989c06ea4 Add webarchiveplayer link to README 2015-02-21 14:28:04 -08:00
Ilya Kreymer
80da0e91da update CHANGELIST for 0.8.1 2015-02-21 14:13:35 -08:00
Ilya Kreymer
9f838241c7 wb.js bug fix: use only window.__orig_parent and not window.parent, as window.parent overriden. window instead of window.self 2015-02-21 12:34:35 -08:00
Ilya Kreymer
c0ff596c68 tests: add tests for recursive cdx indexing, #64
cross-platform: store rel filename path as '/', but convert to os.path.sep
when resolving to full path as prefix
2015-02-20 13:56:35 -08:00
Ilya Kreymer
8d52be4c44 live proxy: enable ssl validation for live proxy, was initially disabled for testing, should be on by default! 2015-02-20 13:22:21 -08:00
Ilya Kreymer
1646c90cd0 cdxindexer: add -r option to support recursive indexing when input is a directory.
filename field in cdx contains relative path including subdir, eg. subdir/file.warc.gz
related to #64
2015-02-20 02:40:32 -08:00
Ilya Kreymer
adeb8bfb27 bump version to 0.8.1, (fix blank spacing in changelist) 2015-02-20 02:02:34 -08:00
Ilya Kreymer
cb6aebf06d Merge CHANGES.rst from 'develop' 0.8.0 2015-02-19 01:29:22 -08:00
Ilya Kreymer
bf203a2dc6 Merge branch 'develop' of https://github.com/ikreymer/pywb into develop 2015-02-19 01:29:03 -08:00
Ilya Kreymer
121e1df3c9 README: update branch config to master 2015-02-19 01:26:55 -08:00
Ilya Kreymer
824587bd90 A few more CHANGES.rst tweaks 2015-02-19 01:24:52 -08:00
Ilya Kreymer
26df8d7784 remove debug logging and spaces 2015-02-19 01:17:31 -08:00
Ilya Kreymer
0ddc490b8d Update CHANGELIST for 0.8.0! 2015-02-19 01:16:25 -08:00
Ilya Kreymer
80dcb6ff27 rewrite: improvements to non-exact replay mode, redir_to_exact option set to false
frames: add request_ts to wbinfo and use that as the timestamp in the top-frame. for exact replay, request_ts == timestamp
for latest replay / no timestamp / memento timegate, redirect to current time instead of time of last capture, while serving
last capture.
timeutils: add timestamp_now() function to return timestamp of current datetime
Add extra tests for this mode
Tracked via #72
2015-02-17 17:51:45 -08:00
Ilya Kreymer
9623f95439 memento: add rel="memento" header to timegate as well, improve memento test, clearly differntiate between
timegate redirect and intermediate resource redirect, related to #70
2015-02-16 09:59:03 -08:00
Ilya Kreymer
c4d5dd4690 rewrite: optimize / sanity, only %-encode urls that are actually idna-encoded,
otherwise return as is, #66
2015-02-15 10:34:56 -08:00
Ilya Kreymer
afe49a91f4 rewrite: more fixes for IDN #66 - add _do_percent_encode field to wburl itself
defaults to true, may be disabled with 'punycode_links'
remove wbrequest and urlrewriter from get_url path, simply call wb_url.get_url() to get properly formatted url
2015-02-14 20:55:36 -08:00