1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

903 Commits

Author SHA1 Message Date
Jack Cushman
6581f54fad Robust chunked data exception handling. 2014-01-21 20:00:52 -05:00
Ilya Kreymer
a1cd40fba1 support replay of records that have Transfer-Encoding: chunked, but
were not actually rewritten to the warc as chunked.
Attempt to parse chunk length, and if failed, fallback to treating
record as not chunked
2014-01-20 23:06:45 -08:00
Ilya Kreymer
8fd10673e8 refactor: cleanup the revisit resolving logic in replay
also, update documented logic on wiki at:
https://github.com/ikreymer/pywb/wiki/PyWb-Record-Lookup-and-Revisits
2014-01-20 17:52:14 -08:00
ikreymer
9a28a2ec6e Merge pull request #6 from jcushman/master
Handle ArchivalUrl subclasses.
2014-01-20 13:08:35 -08:00
Jack Cushman
903583c3d7 Handle ArchivalUrl subclasses. 2014-01-20 14:13:16 -05:00
Ilya Kreymer
9ff3fc300b Fix #5, bringing back customParams optional params sent to cdx server
Rename archivalrouter.MatchRegex -> archivalrouter.Route, supporting regex/prefix matching
add redir_to_exact to turn off redirect to exact timestamp in RewritingReplayHandler
update README
2014-01-20 10:50:06 -08:00
Ilya Kreymer
80b2585d22 Should resolve #4 -- supports pywb running as a non-root app
* Instead of relying on REQUEST_URI, pywb constructs a
REL_REQUEST_URI, from PATH_INFO + QUERY_STRING.
SCRIPT_NAME auto-added to prefix
* MatchPrefix is now superceded by MatchRegex, which
can match a plain string -- collId defaults to the full match
* Added optional archivalurl_class to router to allow for customized
ArchivalUrl implementations to be specified
* run.sh can test on a non-root mountpoint, eg. ./run.sh "/approot"
2014-01-19 21:13:48 -08:00
Ilya Kreymer
2e4d78d079 request_uri: only generate REQUEST_URI manually if not provided by wsgi framework
only encode chars that are not allowed in path segment, per
http://tools.ietf.org/html/rfc3986#section-3.3
2014-01-19 16:51:17 -08:00
ikreymer
628c130261 Merge pull request #3 from jcushman/master
wsgiref compatibility fixes
2014-01-19 16:00:13 -08:00
Jack Cushman
d8c47415c0 Merge branch 'master' of https://github.com/jcushman/pywb
Conflicts:
	pywb/utils.py
2014-01-19 16:25:17 -05:00
Jack Cushman
595c9b0c3c wsgiref compatibility fixes.
- Manually set env[‘REQUEST_URI’] (which is nonstandard) the same way
it’s set by uwsgi.
- Include HTTP error code reasons in error response. (wsgiref checks
that error code is at least 4 characters, i.e. includes reason)
2014-01-19 16:22:06 -05:00
Ilya Kreymer
6cb1743163 Merge branch 'master' of github.com:ikreymer/pywb into work 2014-01-19 12:31:53 -08:00
Ilya Kreymer
354040a7e0 support for url-agnostic dedup, eg loading payload from a different url
than the revisit
2014-01-19 12:31:19 -08:00
Jack Cushman
3f04f63a3f wsgiref compatibility fixes.
- Manually set env[‘REQUEST_URI’] (which is nonstandard) the same way
it’s set by uwsgi.
- Include HTTP error code reasons in error response. (wsgiref checks
that error code is at least 4 characters, i.e. includes reason)
2014-01-19 15:08:14 -05:00
ikreymer
ab955c411b Merge pull request #2 from jcushman/master
Handle transfer-encoding:chunked; misc. replay bugs.
2014-01-19 12:05:57 -08:00
Jack Cushman
c9d0b0ba7b Handle transfer-encoding:chunked; misc. replay bugs.
- Add a ChunkedLineReader to deal with replays with the
transfer-encoding: chunked header.
- Catch UnicodeDecodeErrors caused by multibyte characters getting
split during buffering.
- A couple of tiny bugs in replay.py
2014-01-18 21:32:49 -05:00
Ilya Kreymer
7ce6d0d22b first pass on html rendering via jinja, support for query (cdx) rendering 2014-01-17 16:24:36 -08:00
Ilya Kreymer
bcc9588c00 * archivalrouter: to take a list of handlers,
currently MatchPrefix and MatchRegex. handler returns a single response
(no chaining for now)
* rewriting: don't rewrite anchor only urls
* perf: add a very basic profiler in WBHandler for testing
2014-01-16 20:33:51 -08:00
Ilya Kreymer
8ff2f2fc0c update gitignore 2014-01-06 21:57:33 -10:00
Ilya Kreymer
bc104321c4 Update README.md 2014-01-04 06:12:27 +00:00
Ilya Kreymer
c60493bfdc update README.md 2014-01-04 05:55:17 +00:00
Ilya Kreymer
c4457abc4c Update README
Rename FullHandler -> WBHandler
Add additional comments!
2014-01-03 21:44:20 -08:00
Ilya Kreymer
d820a8c06a add some comments, make charset parsing lower() 2014-01-03 17:40:20 -08:00
Ilya Kreymer
c255f4e47f fix typos 2014-01-03 17:04:15 -08:00
Ilya Kreymer
246b3fba43 cleanup, setup runnable testwb, or pluggable 'globalwb' 2014-01-04 00:21:52 +00:00
Ilya Kreymer
c3767cd31b fix css url parsing typo
always default to utf-8 if chardet thinks ascii
tweak banner
2014-01-03 21:38:18 +00:00
Ilya Kreymer
1e03cad25c update setup.py, static files 2014-01-03 13:06:27 -08:00
Ilya Kreymer
2357f108a3 rename rewriters
header_rewriter added!
support for encoding detection
various fixes
xmlrewriter
2014-01-03 13:03:03 -08:00
Ilya Kreymer
edbcaaf108 big update: refactor archiveloader,
StatusAndHeaders obj and StatusAndHeaders parser
remove dependency on hanzo
Add sample example.warc.gz for very basic unit testing
2014-01-02 20:21:18 -08:00
Ilya Kreymer
cca9071c53 minor tweaks, increase num closest searched, upper case url check
css remove fixed pos
2013-12-31 21:01:18 +00:00
Ilya Kreymer
d9930322f1 support utf-8 (so far)
support protocol-agnostic prefix //
failedFile list for warc loading
2013-12-31 00:18:12 +00:00
Ilya Kreymer
b8c4a453c9 wbhtml: add utf-8 tests 2013-12-29 22:42:29 -08:00
Ilya Kreymer
997dc5df0f fixes! Fix typos, in html parsing, fix base, support attrs w/o values 2013-12-30 03:03:33 +00:00
Ilya Kreymer
a84ec2abc7 first iteration of archival mode working w/ banner insertion!! 2013-12-28 17:39:43 -08:00
Ilya Kreymer
16f458d5ec archiveloader: Support for loading warc/arc records using hanzo parser (for record header parsing only)
ReplayHandler: load replay from query response, find best option
basic support for matching url, checking self-redirects!
2013-12-28 05:00:06 -08:00
Ilya Kreymer
787dfc136e wbhtml: add script and style doctests
override close() to handle open <script> and <style> tags by forcing an end tag,
otherwise parser does not process the remainder
2013-12-24 22:51:33 -08:00
Ilya Kreymer
6050ea1ffa standard JS and CSS rewriting working, with generic regex rewriter
which supports extensions!
2013-12-23 23:57:13 -08:00
Ilya Kreymer
3a896f7cd3 move norewrite prefixs down to ArchivalUrlRewriter (was in html parser)
Add new general regex match work, (several attempts, though last one is simplest/best!)
2013-12-23 15:52:33 -08:00
Ilya Kreymer
37e57f7013 html parser fleshed out! 2013-12-22 18:12:05 -08:00
Ilya Kreymer
fbf29e80d6 add html parser!
urlrewriter support for changing modifier
2013-12-20 19:11:52 -08:00
Ilya Kreymer
072befe3c8 archivalrouter: support handler chaining, using call convention and pass prev response 2013-12-20 15:10:12 -08:00
Ilya Kreymer
4cf4bf3bbb add wburlrewriter, ReferRedirect uses the rewriter
more refactoring, ReferRedirect moved into archivalrouter module
wbrequest: parses from uri directly, keeps track of wburl and prefix
2013-12-20 14:54:41 -08:00
Ilya Kreymer
0a2b16407d better exception handling, specific status codes for exceptions,
detect access control and not found exceptions more consistently
2013-12-19 12:06:47 -08:00
Ilya Kreymer
ebc76c0791 update readme 2013-12-18 18:57:55 -08:00
Ilya Kreymer
c8d2271e8a archiveurl: add support for url_query, format modifier for more unit tests
archivalrouter: flesh out router seperately
indexreader: RemoteCDXServer reader
unit tests for req/resp
wbapp -- cdx output for query, urlquery, replay and latest_replay!
2013-12-18 18:52:52 -08:00
Ilya Kreymer
5d42cc0cac rename aurl -> archiveurl, add default scheme, test for empty url 2013-12-13 15:43:07 -08:00
Ilya Kreymer
6b78f59e49 Merge branch 'master' of github.com:ikreymer/pywb 2013-12-13 15:22:06 -08:00
Ilya Kreymer
27b35f31e8 add basic wsgi app for parsing archivalurls, fallback on a referrer based redirect 2013-12-13 15:20:13 -08:00
ikreymer
d546fcc82c Merge pull request #1 from nlevitt/setup.py
setuptools config
2013-12-09 20:26:14 -08:00
Noah Levitt
89481f162e setuptools config 2013-12-09 11:58:50 -08:00