1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

908 Commits

Author SHA1 Message Date
Ilya Kreymer
f21f4fb1ba Merge branch 'develop' into video 2014-12-01 09:10:08 -08:00
Ilya Kreymer
dffa0bcbad video work: vidrw fixes to support object, fallback to flash
rangecache: create dir on first use only
2014-12-01 09:09:23 -08:00
Ilya Kreymer
c996e70a6e wburl: detect and decode partially encoded schemes in url, such as http%3A//,
https%A2F2F// before handling further
add additional tests for wburl
2014-11-29 11:13:57 -08:00
Ilya Kreymer
2f5be9ed9f video: work on extracting video from <object> tags, fallback to
flowplayer if html5 player fails
2014-11-29 11:04:18 -08:00
Ilya Kreymer
7a14e167e1 replay: vi_ handling: properly replace any protocol with metadata:// for
video info lookup
2014-11-29 11:03:40 -08:00
Ilya Kreymer
87d791eba8 html rewrite: rewrite param value only if start with http 2014-11-29 11:03:09 -08:00
Ilya Kreymer
3e3a74619f various fixes: wombat: add Date.UTC and Date.parse
rewrite: support vi_ https -> metadata
video: fallback to vi_ call on current page
remove debug logging
2014-11-25 00:21:28 -08:00
Ilya Kreymer
4b7186a62e wombat: adjust date for timezone! 2014-11-24 13:46:50 -08:00
Ilya Kreymer
c10df57e07 rules: add support for customizing matchType prefix, adding multiple
filters
2014-11-24 11:10:49 -08:00
Ilya Kreymer
d3ef47342c Merge branch 'develop' into video 2014-11-23 18:58:31 -08:00
Ilya Kreymer
d7eb40af20 rewrite: properly rewrite scheme relative JS-escaped urls:
'\/\/example.com', '\\/\\/example.com/', treat same as '//example.com'
adding http: prefix
2014-11-23 18:56:49 -08:00
Ilya Kreymer
f4431c0872 wombat: add extra check for correct scheme for ajax open 2014-11-23 18:44:11 -08:00
Ilya Kreymer
fcb90fde86 rules: work on yt rules 2014-11-23 18:39:58 -08:00
Ilya Kreymer
36887f565a video: more rewriting work, support: html audio if audio format,
fallback to flash (flowplayer) if html video/audio playback has failed
update setup.py to support nested include structure
2014-11-22 18:39:30 -08:00
Ilya Kreymer
0d191b338f rules: fix rules typo 2014-11-22 18:39:17 -08:00
Ilya Kreymer
0eafd06c42 rangecache: use tmp dir for cache and delete on exit 2014-11-22 18:38:57 -08:00
Ilya Kreymer
550d78a181 add flowplayer 3.2 for flash fallback 2014-11-22 18:37:52 -08:00
Ilya Kreymer
b8b8c30573 cookie_rewriter: add tests for exact cookie rewriter 2014-11-13 09:43:50 -08:00
Ilya Kreymer
20070e95b6 cookie_rewriter: add 'exact' cookie rewriter which never changes the
path/domain
2014-11-13 09:24:34 -08:00
Ilya Kreymer
388f31e08f rewrite: don't rewrite rel=canonical links, need to make rewriting more
configurable (#50)
2014-11-11 15:34:14 -08:00
Ilya Kreymer
9929737a8e rangecache: don't redirect when using range header, don't cache non-200
responses
2014-11-06 22:14:41 -08:00
Ilya Kreymer
1a91f514c5 Merge branch 'develop' into video 2014-11-06 22:13:44 -08:00
Ilya Kreymer
49e98e0cdc archiveiterator/cdxindexer: cleaner load path for compressed and
uncompressed, ability to distinguish between chunked and non-chunked
warcs/arcs
Raise error for non-chunked gzip warcs as they can not be indexed for
replay, addressing #48
add 'bad' non-chunked gzip file for testing, using custom ext
2014-11-06 01:32:42 -08:00
Ilya Kreymer
044792f99f bump version to 0.6.5! 2014-11-06 01:28:56 -08:00
Ilya Kreymer
cc776b65bd update README.rst 0.6.4 2014-11-06 00:35:36 -08:00
Ilya Kreymer
71a8abe9c3 Merge branch 'develop' for 0.6.4 2014-11-06 00:34:32 -08:00
Ilya Kreymer
f6053a977b Update changes for 0.6.4 2014-11-05 21:59:54 -08:00
Ilya Kreymer
00121aa165 statusandheaders parsing: properly skip multiline bad headers (missing
header name and ':'), fixes #49
2014-11-05 20:26:23 -08:00
Ilya Kreymer
c6a2c83b66 rangecache: always bound range, set default bound of 16384
wombat: work on date override, disable for now
head_insert: check for wombat not being inited to avoid undef error
2014-11-05 10:55:46 -08:00
Ilya Kreymer
88f553dce7 video work: live rewrite pings proxy with full rewrite, proxies direct
range request
reorg rangecache to support is_range() check, yt-specific logic
(experimental)
wombat: add date override (experimental)
bump tentative version to 0.7.0!
yt replays work with native player! (though still issues remain)
2014-11-04 22:11:25 -08:00
Ilya Kreymer
fea48fd27a Merge branch 'develop' into video 2014-11-04 12:19:58 -08:00
Ilya Kreymer
e4bcef1c8b rewrite: default HTMLParser entityref and charref are treated as plain
data for HTMLRewriter, since they are never rewritten, and to avoid
semicolon ambiguity, since no way to determine if there is a ; or not
at end. Addresses #43
2014-11-04 12:14:00 -08:00
Ilya Kreymer
5e4b830fa7 cdx: ensure cdx file is closed when iterator is done, since cdx files
are opened per-lookup, related to #45
2014-11-04 09:42:53 -08:00
Ilya Kreymer
72aa921ce5 video: work on domain-specific range cache rewrites 2014-11-04 08:44:45 -08:00
Ilya Kreymer
703ec0eb5e banner insert: extra check for document.body before adding banner 2014-11-02 16:11:03 -08:00
Ilya Kreymer
7aac3aa2dd rewrite: add support for srcset rewriting for img tag 2014-11-02 16:10:38 -08:00
Ilya Kreymer
ced94d289d wombat: add generic setAttribute override for href and src, Image object
override, src prop rewrite
2014-11-02 16:09:56 -08:00
Ilya Kreymer
09e461c58c Merge branch 'video' of https://github.com/ikreymer/pywb into video 2014-11-01 18:24:52 -07:00
Ilya Kreymer
1aac5a9f15 cache: move cache wrappers to seperate cache.py in framework from
proxy_resolvers
range cache: and buffering cache for serving range requests, intended
for videos but not only. full response cached in temp file and range
requests served from cache, still experimental
need to add deletion.
youtube_dl: wrap youtube-dl import due to youtube-dl HTMLParser regex
bug
tests: add test for vi_ handler
2014-11-01 15:41:01 -07:00
Ilya Kreymer
07bcf9fbfe vidrw: work on yt rewrite 2014-11-01 15:41:01 -07:00
Ilya Kreymer
50cae15663 wombat: add wombat class to window 2014-11-01 15:41:00 -07:00
Ilya Kreymer
2396d80dc4 setup: add youtube_dl as dependency
rewrite: add call to _live_rewrite_headers to get correct headers for
proxy on video info
2014-11-01 15:41:00 -07:00
Ilya Kreymer
5b9dcba15f video: add video rewriting use vidrw client side and youtube-dl on the server
add vi_ modifier:
-on record, gets video_info from youtube-dl, sends to proxy,
if any, via PUTMETA to create metadata record
-on playback, fetches special metadata record with video info and
returns to client as json
-vidrw script: fetches video info, if any, and attempts to replace
iframe and embed tags (so far) which are videos
wombat: export extract_url function, fix spaces and use object instance
semantics
2014-11-01 15:41:00 -07:00
Ilya Kreymer
a3b931b45e regex rewrite: fix js regex (dashes), add additional test case 2014-11-01 15:39:51 -07:00
Ilya Kreymer
841fd3f7b4 warc: add ability to set read block size (def 16384) in archiveiterator 2014-11-01 13:29:37 -07:00
Ilya Kreymer
5be65f2945 rules: better rule def, cleanup spacing 2014-10-30 00:10:39 -07:00
Ilya Kreymer
f14f37d5b1 tests: use httpbin for redirect tests 2014-10-29 09:47:32 -07:00
Ilya Kreymer
3d4526f994 wombat: add wombat class to window 2014-10-28 11:46:16 -07:00
Ilya Kreymer
d7b1bc8151 setup: add youtube_dl as dependency
rewrite: add call to _live_rewrite_headers to get correct headers for
proxy on video info
2014-10-28 11:11:15 -07:00
Ilya Kreymer
fb85570974 video: add video rewriting use vidrw client side and youtube-dl on the server
add vi_ modifier:
-on record, gets video_info from youtube-dl, sends to proxy,
if any, via PUTMETA to create metadata record
-on playback, fetches special metadata record with video info and
returns to client as json
-vidrw script: fetches video info, if any, and attempts to replace
iframe and embed tags (so far) which are videos
wombat: export extract_url function, fix spaces and use object instance
semantics
2014-10-28 10:36:48 -07:00