backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 08:04:49 +01:00
Code Issues Projects Releases Activity
1,673 Commits 56 Branches 63 Tags
Commit Graph

1673 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
42a2fa02fe wombat: history check fix: ensure check applies to absolute url #188 2016-07-16 13:32:46 -04:00
Ilya Kreymer
64a49b3e4d wombat: history change improvements (#188): 
- ensure back, go, forward also propagated to top frame
- ensure pushState propagated as pushState and replaceState as replaceState to top frame
- security: prevent pushState or replaceState from changing to different domain
 2016-07-16 13:18:08 -04:00
Ilya Kreymer
605ee22bec html rewrite: rewrite href on any element, not just few designated ones, as client side rewriting does the same. 
avoids edge cases where href used on other tags (eg. a div) that results in incorrect rewriting, #187
 2016-07-16 12:55:24 -04:00
Ilya Kreymer
b46cf8492f bump version to 0.31.5 2016-07-16 12:48:26 -04:00
Ilya Kreymer
ae290587f6 temp cookie store: add add_cookie() function for explicitly adding cookie, make expiry configurable 
related to webrecorder/webrecorder#79
 2016-07-01 10:15:59 -04:00
Ilya Kreymer
0b57f4a352 cookie notification: use postMessage() instead of callback to notify top frame of cookie setting with custom domain, #186 2016-07-01 09:58:25 -04:00
Ilya Kreymer
827ba9b50f cookies: add optional callback when setting cookie with domain (to experiment with server side handling of custom domain) 2016-06-30 12:26:18 -04:00
Ilya Kreymer
f4e5a7df5d Merge branch 'develop' 2016-06-16 00:41:08 -04:00
Ilya Kreymer
2fba97683a CHANGES for 0.31.0 2016-06-16 00:40:53 -04:00
Ilya Kreymer
5024234552 CHANGES for 0.31.0 2016-06-16 00:39:51 -04:00
Ilya Kreymer
d457223555 tests: add brotli compression test #184 2016-06-16 00:00:47 -04:00
Ilya Kreymer
457a1a564c bufferedreader: support brotli decompression 
rewrite: handle Content-Encoding: br using brotli decompressor
setup: add brotlipy as dependency
 2016-06-15 01:37:29 -04:00
Ilya Kreymer
bc36ae1302 rewriter: update for moved RewriterAMF in pywb 2016-06-14 00:14:29 -04:00
Ilya Kreymer
c1d7111841 webagg: store original 'source' value in cdx for properly mapping in WARC file resolver 
error handling: ensure 'last_exc' is a string
 2016-06-14 00:13:01 -04:00
Ilya Kreymer
3b68ef6540 html rewriter: cleanup rewrite_srcset, add more tests for empty rewrite 2016-06-12 01:57:21 -04:00
Ilya Kreymer
6a5842d983 Merge branch 'chdorner-fix-empty-srcset' into empty-attr 2016-06-12 01:53:53 -04:00
Ilya Kreymer
1bfec37970 html rewriter: attr rewrite ops check for empty/blank attr value, return empty string 2016-06-12 01:50:55 -04:00
Ilya Kreymer
d2c37f7d91 html parser: attr_value can now be None -- default to '' for string ops, write attr w/o assignment 2016-06-12 01:38:03 -04:00
Ilya Kreymer
0f530a3e0e dependencies: remove pyamf, update to latest surt (0.3.0) 2016-06-12 00:44:52 -04:00
Ilya Kreymer
9f299eb8e9 amf rewriting: move to separate file, mark as experimental, and don't include as default (for now) 2016-06-12 00:40:35 -04:00
Ilya Kreymer
527a3bc89c bufferedreader: be lenient of partially decompressed data: return what was decompressed, rather than just throw exception 
esp. useful if record was decompressed, but an error in crc check
may add additional options for toggling 'leniency' if needed
 2016-06-12 00:37:14 -04:00
Ilya Kreymer
4c7da0f6ef recorder: support overridings get_params() in subclass 
multiwarcwriter: support multiple warcs in same dir, support random component in path, and a custom
key template for selecting current warc file, not related to current directory
 2016-06-07 12:55:04 -04:00
Ilya Kreymer
3fec766e39 webagg: redis lookup: if url contains wildcard, scan redis keys to check multiple keys until one is found 
webagg tests: fix test to include mime in live cdx
 2016-06-07 12:54:28 -04:00
Ilya Kreymer
197ed5be98 loader: profile urls: ensure the profile prefix is removed from url before passing to loader, #180 2016-06-04 14:09:18 -04:00
chdorner
b54347f8d1 Allow rewriting of empty srcset attributes 
Strictly speaking a `srcset` attribute must consist of one or more
strings
(http://w3c.github.io/html/semantics-embedded-content.html#element-attrdef-img-srcset)
However are websites out there that specify an empty string as the
value.

This commit makes sure that the rewriting does not break and just
returns an empty string.
 2016-06-01 11:31:26 +02:00
Ilya Kreymer
d7c74b68de video loader support: add VideoLoader, which uses youtube-dl to create a metadata record 
of video info. Activated with explicit content_type param 'application/vnd.youtube-dl_formats+json'
 2016-05-28 15:01:33 -07:00
Ilya Kreymer
30f9d0aca7 recorder put custom record: add support for put/post of a custom record. If put_record= param is included, the request body 
is written to the specified record type.
move record creation functions to the warcwriter
add tests for custom record
 2016-05-26 20:49:40 -07:00
Ilya Kreymer
ea3efdf84d responseloader: use PreparedRequest() to ensure url properly formatted 
tests: update tests for latest, live data
 2016-05-24 18:01:44 -07:00
Ilya Kreymer
e28f294302 wombat: ensure window.open() rewrite happens even in if open not in prototype 
rewrite mod: allow empty "" as set mod, check for undefined
 2016-05-24 17:55:17 -07:00
Ilya Kreymer
f858be4d7d Merge branch 'frame-postMessage' into develop 2016-05-24 15:40:51 -07:00
Ilya Kreymer
84c829467b framed replay: use postMessage() instead of custom function to notify of replay frame changing url, include different type of change, eg. load, replaceState, pushState, #181 2016-05-23 12:10:10 -07:00
Ilya Kreymer
8ef6eb97b8 cdx: encoding: use to_native_str() consistently for better py2 compat 2016-05-23 11:47:44 -07:00
Ilya Kreymer
80d9805a58 webagg: tests: flush fakeredis for reentrancy 
utils: add load_config() with option for main and override configs
 2016-05-19 17:01:09 -07:00
Ilya Kreymer
8ad66249c7 blockloader: support for loader profiles, specified via 'profile+scheme://...' urls. Profiles specify additional settings (eg. credentials) that are not included in the url. To enabl 
e custom profiles, BlockLoader.set_profile_loader(callable) to a callable that will return custom config, addresses #180
 2016-05-18 16:34:58 -07:00
Ilya Kreymer
d11bd444ad s3 loader: unurlencode username/password 2016-05-17 19:24:14 -07:00
Ilya Kreymer
119074e0ee s3 loader improvements: support AWS cred in username and password part of url, stream s3 response directly 2016-05-17 18:55:10 -07:00
Ilya Kreymer
94afab0bb2 wombat rewrite: don't add duplicate slash in rel-url resolve 2016-05-17 18:53:00 -07:00
Ilya Kreymer
10d8e4b3be bump version to 0.31.0 2016-05-17 18:38:57 -07:00
Ilya Kreymer
45c8fcddbd recorder: add max_idle_secs / close_idle_files() to close any open files that have not been modified longer than set threshold, in prep for webrecorder/webrecorder#92 
indexer: add 'full_warc_prefix' for setting full path prefix in add_warc_file() (eg. for http load) for webrecorder/webrecorder#95
 2016-05-11 21:40:02 -07:00
Ilya Kreymer
94d6098238 app: separate json_encode() func 
compat: py2 fixes
 2016-05-11 11:38:59 -07:00
Ilya Kreymer
c45f5cb749 webagg: use werkzeug routing instead of wrapping Bottle app 2016-05-10 16:31:44 -07:00
Ilya Kreymer
464eca2fa0 test apps: enable debugging for test apps 
test recorder: write to a temp dir for each run
 2016-05-06 16:33:18 -07:00
Ilya Kreymer
e64ae780c6 urlrewrite: improve POST request support for ikreymer/pywb#178 2016-05-06 16:32:13 -07:00
Ilya Kreymer
87da25c703 post request mapping improvements: work on #178, including: 
- mapping multipart/form-data same as x-www-form-urlencoded
- parsing application/x-amf with pyamf
- RewriteContentAMF for rewriting AMF response to match request
- default encoding of other POST data as base64 encoded __wb_post_data param
 2016-05-06 10:19:08 -07:00
Ilya Kreymer
e5e7c5a7df wombat: ensure Math.random() overrides use the current window 2016-05-06 09:48:38 -07:00
Ilya Kreymer
1e7d4d27e3 bump version to 0.30.2 2016-05-06 09:43:11 -07:00
Ilya Kreymer
ab3af90df2 cookie_tracker: add support for redis-based subdomain cookie tracker, which temp caches cookies with Domain= set in redis and passes them upstream 
when rewriting. addresses webrecorder/webrecorder#79
 2016-05-04 16:39:47 -07:00
Ilya Kreymer
8e473f01fa add changelist for 0.30.1 2016-05-04 11:33:43 -07:00
Ilya Kreymer
2795802c77 recordloader: for request/response/revisit records, only parse urls starting with http:/https: as http 2016-05-04 11:20:38 -07:00
Ilya Kreymer
af920d77a0 rules: add fuzzy rules for TW video 2016-05-03 17:33:13 -07:00