backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
1,635 Commits 56 Branches 63 Tags
Commit Graph

1635 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
4f9b963e13 tests: update test to support uncompressed followed after compressed block 2016-12-08 14:20:46 -08:00
Ilya Kreymer
bc219acb33 rewriterapp: fix async requests call typo! (was erroring with invalid params previously) 2016-12-08 13:44:11 -08:00
Ilya Kreymer
d772b05fd6 warc indexing improvements: 
decompressor: allow plaintext after gzipped record fully finished, as next member
warc loader: ignore blank line records -- if empty statusheaders, set length to 0 and ignore, don't read indenfinitely
 2016-12-08 12:52:51 -08:00
Ilya Kreymer
2fa6aa7a20 Merge branch 'develop' into new-pywb 2016-12-02 20:43:10 -08:00
Ilya Kreymer
5690604556 client-side rewrite: add eval() override, add WB_wombat_ prefixes for location 2016-12-02 12:11:54 -08:00
Ilya Kreymer
936b8dfb86 live web loader: add support for optional forward proxy 2016-11-30 12:46:34 -08:00
Ilya Kreymer
fec907a299 responseloader live loader: increase httplib max headers to avoid 'too many headers' error 2016-11-28 10:36:58 -08:00
Ilya Kreymer
577ced76f0 dockerfile: set fixed requests version to avoid encoding issues in latest requests 2016-11-28 10:35:15 -08:00
Ilya Kreymer
1ef0a54988 recorder improvements: 
- make recorder tempfile used by request/response wrappers overridable, better checks to ensure temp file is closed after recording is done/failed
- ensure ParamsFormatter inited for all requests
- writer: ensure writing from temp buffer done in BUFF_SIZE increments
 2016-11-27 21:12:12 -05:00
Ilya Kreymer
d7d002c076 Merge branch 'develop' into new-pywb (rules and rewrite fixes) 2016-11-23 11:47:21 -08:00
Ilya Kreymer
a8c0ff3c06 client rewrite: fix window.fetch override, create new Request object if url is rewritten 2016-11-23 11:46:01 -08:00
Ilya Kreymer
e5adc5ba69 responseloader: ensure Host header is correct when sending non-live remote request 2016-11-23 11:41:13 -08:00
Ilya Kreymer
685d48d531 webagg: split RedisMultiKeyIndexSource into Base* version to make more extensible, support different agg mixin 
indexsource: update __repr__ funcs to use current classname
 2016-11-22 18:21:01 -08:00
Ilya Kreymer
99e533d31b remoteindex: add limit when doing closest query 
responseloader: ignore scheme from self-redir check
 2016-11-21 21:42:07 -08:00
Ilya Kreymer
c9a0259604 dockerfile: install dependencies first to speed up updates 2016-11-21 21:33:04 -08:00
Ilya Kreymer
74276f58f3 webagg improvements: 
responseloader: direct loader: unrewrite location, content-location headers for non-live responses
autoapp: support custom indexsource list
indexsource: ensure closest query is added for RemoteIndexSource
utils res)template: urlencode '{url}' param if after '?'
 2016-11-21 18:59:22 -08:00
Ilya Kreymer
cbe7508afc webagg: add ZipNumIndexSource, add zip and cdxops test using new webagg IndexSource system 
autoapp: add init_index_agg() for initializing indexes from a config dict
autoapp config: use RedisMultiKeyIndexSource for redis url and ZipNumIndexSource as zipnum+
 2016-11-18 16:40:14 -08:00
Ilya Kreymer
1d8ddb8d20 responseloader: support for gzip compressing warc record with 'compress=gzip' param 
prefix resolver: if prefix contains '*', attempt to resolve with glob, ignore none prefix
 2016-11-17 19:11:54 -08:00
Ilya Kreymer
eac5bdce26 webagg: add AutoConfigApp initing the webagg sytsem from config.yaml 
all index sources can be inited from string or dictionary (loaded from yaml)
support for dynamic directory-based collections based on file system, as well as static routes
specified explicitly
add `-cdx` path for compatibility with existing pywb -cdx interface
tests: add tests for AutoConfigApp yaml loading
add WSGI app shortcut for AutoConfigApp
 2016-11-17 19:06:04 -08:00
Ilya Kreymer
34a03a78f6 app: fix missing import, add simple route path 
test: fix typo
 2016-11-17 19:00:29 -08:00
Ilya Kreymer
d24868db7a tests: add MementoOverrideTests as a reusable class, convert memento_agg tests to use class, 
handlers: add saved link header data for memento tests for handlers
 2016-11-15 14:24:34 -08:00
Ilya Kreymer
c7fa8b711c travis: trying 2.7, 3.5 only for now 2016-11-15 10:22:32 -08:00
Ilya Kreymer
cec0db1bdd rules: instagram rules tweak, ignore query args 2016-11-14 13:19:26 -08:00
Ilya Kreymer
41f6ca9bb6 rules: update rules for medium, instagram 
bump version to 0.33.1
 2016-11-13 22:50:53 -08:00
Ilya Kreymer
008bc47fad tests & travis: change live test to httpbin, remove 3.3 for now 2016-11-13 18:37:47 -08:00
Ilya Kreymer
36862fd9e9 recorder test: fix warc/revisit cdx test (don't assume exact order with 14-digit timestamp) 2016-11-13 11:46:10 -08:00
Ilya Kreymer
169915ccc5 Dockerfile: add entire dir, use .dockerignore 2016-11-11 14:26:24 -08:00
Ilya Kreymer
4a94aefead travis fixes: add dependency, remove unnecessary include 2016-11-11 12:07:51 -08:00
Ilya Kreymer
47a3300809 dockerfile: add new Dockerfile for building from local source 2016-11-11 12:07:37 -08:00
Ilya Kreymer
e37900b9c6 tests: add test dependency, remove 2.6 from travis 2016-11-11 11:03:16 -08:00
Ilya Kreymer
8765de4fe7 refactor: updated dependencies, remove watchdog, add gevent and webassets 
update tests, tests should pass for python 2 and 3!
 2016-11-11 10:32:19 -08:00
Ilya Kreymer
ab77c1b6d9 refactor autoindex: switch to gevent-based simple polling, as watchdog doesn't work with gevent #200 2016-11-11 10:31:48 -08:00
Ilya Kreymer
fa247b8fe5 refactor: fix recorder and urlrewrite packages #200 2016-11-08 15:04:22 -08:00
Ilya Kreymer
6b4b038471 refactor: fix pywb.webagg package paths 
all webagg tests working!
move testdata cdxj into sample_archive, remove rest (duplicates) #200
 2016-11-08 14:30:09 -08:00
Ilya Kreymer
99e5008ac0 refactor: move newly merged packages to be pywb subpackages 2016-11-08 07:01:33 -08:00
Ilya Kreymer
88d6b9e097 Merge remote-tracking branch 'webrec-platform' system into pywb for furthering refactoring! 2016-11-08 06:55:37 -08:00
Ilya Kreymer
de44110391 update to pywb 0.33.0 2016-10-24 19:05:45 +00:00
Ilya Kreymer
526db7a1d7 tweaks to CHANGES.rst 2016-10-24 11:34:34 -07:00
Ilya Kreymer
2980a06d03 Update CHANGES.rst for 0.33.0 2016-10-24 11:30:57 -07:00
Ilya Kreymer
c44e780c12 bump version to 0.33.0 for release 2016-10-24 10:45:30 -07:00
Ilya Kreymer
adce15123a rewriter: mark 'is_ajax' in urlrewriter 2016-10-22 07:19:46 +00:00
Ilya Kreymer
3d507c5d68 urlrewrite: webassets: add webassets support to JinjaEnv, if 'assets_path' is set, the specified webassets yaml file is added to the env 2016-10-22 00:13:41 -07:00
Ilya Kreymer
3f8480c37e typo: fix typo after rename! 2016-10-20 11:47:06 -07:00
Ilya Kreymer
40b0a291a9 rewrite: don't rewrite ajax-requested html content 
js regex: add special regex to rewrite '?location:'
 2016-10-20 11:30:14 -07:00
Ilya Kreymer
52ce45beee tests: additional test for new modifier form 2016-10-19 21:17:40 -07:00
Ilya Kreymer
42a31bbebf wombat improvements: 
- history change check: don't reject urls without a slash, check if new url == origin
- new api: override window.fetch() if it exists
- srcset elem rewriting, <source> element srcset override
- ajax: don't add X-Pywb-Requested-With header if url is a data: url
 2016-10-19 21:11:16 -07:00
Ilya Kreymer
8b77f66a10 wb_frame.js: make more safe, check that frame actually exists before accessing 2016-10-19 20:57:56 -07:00
Ilya Kreymer
003d84c371 responseloader: self-redirect: if no status code (eg. for revisits), always parse and look at the actual status code 2016-10-19 11:03:48 -07:00
Ilya Kreymer
7b45df7338 wburl: support for new modifier form: $mod as well as 'mod_' 2016-10-10 17:00:36 -07:00
Ilya Kreymer
06b9e957e6 vidrw: when in proxy mode, use current protocol for vi_ query 2016-10-03 08:17:13 -07:00