69 Commits

Author	SHA1	Message	Date
Ilya Kreymer	dd9f138bab	disable decoding, by default, of content for html parser	2014-06-27 16:53:33 -07:00
Ilya Kreymer	ceeb25a899	rewrite: fix unit tests, add extra closed check for 2.6 (not sure why its needed now)	2014-06-14 01:02:00 -07:00
Ilya Kreymer	923421d637	rewrite_content: add a few tests for cs_, js_, remove redundant except	2014-05-16 22:43:53 -07:00
Ilya Kreymer	46449ac188	rewrite: pass wburl mod to rewritier, so that css/js rewriting ... rules may override default content-type (in cases where it is incorrect) allows for rule based cusomization (to be added later)	2014-05-05 22:12:45 -07:00
Ilya Kreymer	9cf5327e88	bufferedreader cleanup: ... * BufferedReader defaults to no decompression * DecompressingBufferedReader defaults to gzip decomp * ChunkedDataReader defaults to no gzip decomp, but decomp can be set later via set_decomp(). This allow chunked responses to be de-chunked but not decompressed (eg for non-text responses)	2014-04-28 20:15:31 -07:00
Ilya Kreymer	7636c9d3f7	fix: when reading response, only readline() if previous read() ... was non-empty	2014-04-09 16:44:45 -07:00
Ilya Kreymer	bfc2e63793	live rewriter: integrate handler with rewrite_live.py module, ... clean up css, add unit and integration tests clean up cli server now known as 'live-rewrite-server', which performs live rewrite using iframe paradigm	2014-04-09 15:49:55 -07:00
Ilya Kreymer	19f2df4717	refactor: ... - move is_identity(), is_embed() to wburl from wbrequest - add is_mainpage() predicate - add create_template() to each J2TemplateView to create itself - add HeadInsertView to create a reusable head insert for RewriteContent - add 'mp_' as modifier for frames mode to be used as possible modifier with HTMLRewriter	2014-04-09 15:49:55 -07:00
Ilya Kreymer	2a318527df	lxml: use lxml's parse interface instead of feed interface to allow ... xml to handle decoding unicode data, better address #36	2014-04-07 17:13:43 -07:00
Ilya Kreymer	d6006acdc3	rewrite: when using lxml parser, just pass raw stream to lxml ... without decoding. lxml parser expects to have raw bytes and will determine encoding on its own. then serve back as utf-8 if no encoding specified. should address #36	2014-04-06 09:47:34 -07:00
Ilya Kreymer	5dd586cf07	refactor: simplify rewrite_content and replay_views, remove ... redundant code.. everything goes through rewrite_content(), is sanitized (for transfer encoding) if needed additional testing for decode_buff fix failed_files bug in resolvingloader, add tests	2014-04-03 12:44:00 -07:00
Ilya Kreymer	a6b4ae4c47	chardet optimization: using chardet feed() approach to avoid passing in entire buffer	2014-03-17 20:53:42 -07:00
Ilya Kreymer	f35e82a4d5	ensure final output from close() is encoded! ... add config option to 'use_lxml_parser' if available, if not, will default to regular parser testing on travis with lxml (not adding to dep yet)	2014-03-17 13:19:51 -07:00
Ilya Kreymer	a69d565af5	make pywb.rewrite package pep8-compatible ... move doctests to test subdir	2014-03-14 16:44:23 -07:00
Ilya Kreymer	453ab678ed	refactor domain specific rules: ... - head insert callback passed in with rule, up to template to handle additional inserts based on rule properties - ability to pass in custom rules config to both cdx server and content rewriter - move canonicalize to utils pkg - add wombat, modify wb.js to remove wombat-related settings	2014-02-26 22:04:37 -08:00
Ilya Kreymer	5a41f59f39	new unified config system, via rules.yaml! ... contains configs for cdx canon, fuzzy matching and rewriting! rewriting: ability to add custom regexs per domain also, ability to toggle js rewriting and custom rewriting file (default is wombat.js)	2014-02-26 18:02:01 -08:00
Ilya Kreymer	922917a631	rename BufferedReader -> DecompressingBufferedReader ... remove max_len from DecompressingBufferedReader as it applied to the compressed size, not original size. Add integration test for verifying content length of larger file	2014-02-20 11:53:08 -08:00
Ilya Kreymer	a09dec4b3e	cdx: add domain-specific rules at cdx layer for custom canonicalization! ... and 'fuzzy' matching when not found handled via cdxdomainspecific.py BaseCDXServer contains a canonicalizer object and a fuzzy query canonicalizer abstracted to seperate class (in canonicalizer.py) clean up cdx related exceptions default rules read from cdx/rules.yaml filename configurable via 'domain_specific_rules' setting in config.yaml fix typo in pywb/rewrite	2014-02-18 14:56:13 -08:00
Ilya Kreymer	5345459298	pywb 0.2! ... move to distinct packages: pywb.utils, pywb.cdx, pywb.warc, pywb.util, pywb.rewrite! each package will have its own README and tests shared sample_data and install	2014-02-17 10:01:09 -08:00