backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 08:04:49 +01:00
Code Issues Projects Releases Activity
1,577 Commits 56 Branches 63 Tags
Commit Graph

1577 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
bdf4f9bc71 static handler: if 'wsgi.file_wrapper' throws exception, default to streaming directly 2016-09-29 15:23:40 -07:00
Ilya Kreymer
e61078ab96 memento: use replace_header() to avoid double adding Link, Memento-Datetime, Vary when using range request cache 2016-09-29 15:22:44 -07:00
Ilya Kreymer
4cdb99f415 rewrite: strip www redir check: use re.MULTILINE to include urls that may have a \r 2016-09-29 15:20:25 -07:00
Ilya Kreymer
98e8a75920 vidrw: more permissive flash video rewriting: consider any <object> with flashvars, attempt any youtube-dl playlist 
bump version to 0.32.2
 2016-09-21 11:37:31 -07:00
Ilya Kreymer
64cd408de4 update CHANGES for 0.32.1 2016-09-20 15:44:50 -07:00
Ilya Kreymer
a6a186891e wbrequestresponse: text response: calculate Content-Length from encoded utf-8 bytes, not the original text 2016-09-20 15:44:50 -07:00
Ilya Kreymer
1bb7aa01ce wburl improved scheme detection: use regex to match acceptable scheme before :/, don't treat something like 'a.com/?x=http://' as having a scheme, update tests to check for this 2016-09-20 15:44:50 -07:00
Ilya Kreymer
9a3017bfcd bump version to 0.32.1 2016-09-20 15:44:49 -07:00
Ilya Kreymer
5c499753f8 webrecore Docker: update Docker file to latest pywb, python, starting to use versioning! 2016-09-16 18:43:26 -07:00
Ilya Kreymer
874bef0ab1 Update CHANGES 0.32.0 2016-09-15 14:18:44 -07:00
Ilya Kreymer
dc05d14934 Merge pull request #194 from nlevitt/cli-desc 
fix/tweak for cli --help
 2016-09-15 14:16:42 -07:00
Ilya Kreymer
c3f98c3d38 Merge branch 'develop' 2016-09-15 14:15:34 -07:00
Ilya Kreymer
d95116885a Update CHANGES for 0.32.0 2016-09-15 14:14:05 -07:00
Ilya Kreymer
86cbb366f3 rules: undo yt rules change (will revisit later) 2016-09-15 10:01:36 -07:00
Ilya Kreymer
0a76a56b91 wombat: edge case: correctly handle <iframe src="javascript:WB_wombat_location=...> assignment created via JS.. custom rewrite_frame_src() added for use with rewrite_elem(), ensures wombat init is inserted first thing after 'javascript:' 2016-09-14 15:44:20 -07:00
Ilya Kreymer
cc65ce914d wombat improvements (2.16): 
- rewrite_elem() also rewrite 'poster'
- extract_orig() don't add http:// if nothing extracted
- new override: navigator.sendBeacon() if available
 2016-09-14 14:13:59 -07:00
Ilya Kreymer
5fede0fea3 wombat: turn off debugging (accidentally committed) 2016-09-14 13:39:10 -07:00
Ilya Kreymer
1fb6e9b5fa rewrite: url rewriter: don't rewrite relative urls, only those that start with scheme, / or contain ../ #195 
update tests to reflect this new behavior
 2016-09-14 13:04:46 -07:00
Noah Levitt
1620668363 fix/tweak for cli --help 2016-09-14 09:58:44 -07:00
Ilya Kreymer
70fdaae2b3 rules: rewrite location string for periscope js 2016-09-12 20:07:14 -07:00
Ilya Kreymer
1a37d789ed cdx-api: when using cdx server api, return no captures 404 error in json format if output=json, plain text otherwise instead of as html #193 2016-09-08 18:59:52 -07:00
Ilya Kreymer
f47ae0bb7e rewrite: for rewriting on* attr, add 'window.' before WB_wombat_ as window may not be in scope (if no '.' before WB_wombat) 2016-09-08 18:38:35 -07:00
Ilya Kreymer
6452c72b4f bump versions 2016-09-08 10:31:07 -07:00
Ilya Kreymer
1fe201c528 rewrite: html: rewrite svg <image> tag 
client: update textContent after rewrite_style() in rewrite_elem()
 2016-09-08 10:06:47 -07:00
Ilya Kreymer
895a01933c wb: allow multiple readystateevent changes, in case data changes (eg. title is available later) 2016-09-02 12:04:30 -07:00
Ilya Kreymer
70a25b6d0f client rewrite: ensure window.open() windows have wombat inited. if they are set to about:blank, use parser from opener to ensure proper relative url resolving 2016-08-20 13:03:17 -04:00
Ilya Kreymer
099a81b786 wb_frame: add support for optional 'wbinfo.outer_prefix' which if set, is used for making the top frame url (#191) 2016-08-20 00:03:21 -04:00
Ilya Kreymer
892ebacead cross-frame improvements: #191 
- make hashchange functions use postMessage(), support setting top->replay and replay->top
- special postMessage() option for sending message from top frame -> replay frame
- fix history navigation, mimic top frame history same as replay frame as much as possible
- remove iframe_loaded() callback, using postMessage() notifications only
- include document title in 'load' message
 2016-08-19 23:44:15 -04:00
Ilya Kreymer
6af1a7856e top-frame handling: don't access contents of top frame directly to support cross-domain frames 
set __WB_top_Frame in wombat if is_framed property is true, don't check wbinfo (#191)
 2016-08-19 13:59:42 -04:00
Ilya Kreymer
2fb1df34c9 recorder: add upload/streaming support with put_record=stream where the content being uploaded is already in WARC record form 2016-08-12 21:23:25 -04:00
Ilya Kreymer
c8b6a48005 webagg: use prepare_auth() to ensure Authorization header is set for http://user:pass@host urls 2016-08-12 21:22:17 -04:00
Ilya Kreymer
82d3b61523 recorder: catch exception in close_idle_files() if file no longer exists and ensure it's removed 2016-08-12 01:19:30 -04:00
Ilya Kreymer
594aff86d3 webagg: response self-redir: don't check if live, throw correct exception 2016-08-10 00:50:43 -04:00
Ilya Kreymer
92dfcbfcbe rewrite: don't rewrite 'www-authenticate' and 'proxy-authenicate' headers 2016-08-10 00:02:53 -04:00
Ilya Kreymer
cca0c01547 urlrewrite misc fixes: 
- ensure content-length is converted to str
- templateview: support optional extensions
- fix test
 2016-08-09 19:53:22 -04:00
Ilya Kreymer
b22a29df5f vidrw: also check for 'src' param as well as movie 2016-08-08 19:50:16 -04:00
Ilya Kreymer
c93d7ecafc webagg: Fix loading of url-lookup (url agnostic) revisits, ensure all params passed to cdx lookup, add tests for url-agnostic revisit lookup 2016-08-04 16:53:24 -04:00
Ilya Kreymer
e04095ffbb rewrite css: leave spaces in css url, eg url(' http://example.com/ ') rewritten with spaces intact 2016-08-01 10:29:04 -04:00
Ilya Kreymer
d5adc05cbb history rewrite check: don't check empty urls (#188) 2016-08-01 10:27:38 -04:00
Ilya Kreymer
20b161bf90 debug: print stracktrace when debugging 2016-08-01 02:12:15 -04:00
Ilya Kreymer
68b94fe671 record parser: arc-to-warc: support converting arc records to warc 'response' records on-the-fly to simplify 
processing for tools that read WARC records. arc headers are converted to equivalent warc header, WARC-Record-ID
generated on the fly #190
 2016-07-31 22:31:21 -04:00
Ilya Kreymer
66ca8d8b26 http block loader: raise exception for 4xx, 5xx responses 
tests: add tests for limitreader posting, fix charset for frame test
 2016-07-31 12:56:00 -04:00
Ilya Kreymer
db3b92e228 writing: add write_stream_to_file()function to be able to write to a WARC an existing input stream 
refactor _do_write_req_resp to pass callback to actual writing (eg. _write_to_file)
 2016-07-31 00:49:57 -04:00
Ilya Kreymer
1b09015954 recorder: split up _open_file() into get_new_filename() and allow_new_file() to customize skipping recording by returning false 
from allow_new_file()
create_warcinfo_record() - switch to dict args over kwargs, update tests
 2016-07-30 13:11:12 -04:00
Ilya Kreymer
c3389987cd frame timestamp extract: fix timestamp extracting timestamp for non-html resources for use with frame display (#189) 2016-07-28 10:06:10 -04:00
Ilya Kreymer
c8c0cecda3 rewrite improvements: if content-type is text/plain but mod is js_ or cs_, treat as js or css (#31) 
header rewriter: ensure removed content-length and content-encoding are added back if no rewriting performed on response body
 2016-07-27 21:34:58 -04:00
Ilya Kreymer
cd15dbfe48 head_insert: add decodeURI() to prefix to ensure unicode prefix string 2016-07-27 10:34:54 -04:00
Ilya Kreymer
498f87fb54 add Dockerfile to git! 2016-07-26 19:42:59 -04:00
Ilya Kreymer
a5696fc2d4 rewriter: range massage for patch as well as record 2016-07-26 19:42:32 -04:00
Ilya Kreymer
14cf68e4e5 custom record: don't override WARC-Date if provided in request header, 
return chosen WARC-Date in json response
 2016-07-26 19:41:47 -04:00