backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
903 Commits 56 Branches 63 Tags
Commit Graph

903 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jack Cushman
6581f54fad Robust chunked data exception handling. 2014-01-21 20:00:52 -05:00
Ilya Kreymer
a1cd40fba1 support replay of records that have Transfer-Encoding: chunked, but 
were not actually rewritten to the warc as chunked.
Attempt to parse chunk length, and if failed, fallback to treating
record as not chunked
 2014-01-20 23:06:45 -08:00
Ilya Kreymer
8fd10673e8 refactor: cleanup the revisit resolving logic in replay 
also, update documented logic on wiki at:
https://github.com/ikreymer/pywb/wiki/PyWb-Record-Lookup-and-Revisits
 2014-01-20 17:52:14 -08:00
ikreymer
9a28a2ec6e Merge pull request #6 from jcushman/master 
Handle ArchivalUrl subclasses.
 2014-01-20 13:08:35 -08:00
Jack Cushman
903583c3d7 Handle ArchivalUrl subclasses. 2014-01-20 14:13:16 -05:00
Ilya Kreymer
9ff3fc300b Fix #5, bringing back customParams optional params sent to cdx server 
Rename archivalrouter.MatchRegex -> archivalrouter.Route, supporting regex/prefix matching
add redir_to_exact to turn off redirect to exact timestamp in RewritingReplayHandler
update README
 2014-01-20 10:50:06 -08:00
Ilya Kreymer
80b2585d22 Should resolve #4 -- supports pywb running as a non-root app 
* Instead of relying on REQUEST_URI, pywb constructs a
REL_REQUEST_URI, from PATH_INFO + QUERY_STRING.
SCRIPT_NAME auto-added to prefix
* MatchPrefix is now superceded by MatchRegex, which
can match a plain string -- collId defaults to the full match
* Added optional archivalurl_class to router to allow for customized
ArchivalUrl implementations to be specified
* run.sh can test on a non-root mountpoint, eg. ./run.sh "/approot"
 2014-01-19 21:13:48 -08:00
Ilya Kreymer
2e4d78d079 request_uri: only generate REQUEST_URI manually if not provided by wsgi framework 
only encode chars that are not allowed in path segment, per
http://tools.ietf.org/html/rfc3986#section-3.3
 2014-01-19 16:51:17 -08:00
ikreymer
628c130261 Merge pull request #3 from jcushman/master 
wsgiref compatibility fixes
 2014-01-19 16:00:13 -08:00
Jack Cushman
d8c47415c0 Merge branch 'master' of https://github.com/jcushman/pywb 
Conflicts:
	pywb/utils.py
 2014-01-19 16:25:17 -05:00
Jack Cushman
595c9b0c3c wsgiref compatibility fixes. 
- Manually set env[‘REQUEST_URI’] (which is nonstandard) the same way
it’s set by uwsgi.
- Include HTTP error code reasons in error response. (wsgiref checks
that error code is at least 4 characters, i.e. includes reason)
 2014-01-19 16:22:06 -05:00
Ilya Kreymer
6cb1743163 Merge branch 'master' of github.com:ikreymer/pywb into work 2014-01-19 12:31:53 -08:00
Ilya Kreymer
354040a7e0 support for url-agnostic dedup, eg loading payload from a different url 
than the revisit
 2014-01-19 12:31:19 -08:00
Jack Cushman
3f04f63a3f wsgiref compatibility fixes. 
- Manually set env[‘REQUEST_URI’] (which is nonstandard) the same way
it’s set by uwsgi.
- Include HTTP error code reasons in error response. (wsgiref checks
that error code is at least 4 characters, i.e. includes reason)
 2014-01-19 15:08:14 -05:00
ikreymer
ab955c411b Merge pull request #2 from jcushman/master 
Handle transfer-encoding:chunked; misc. replay bugs.
 2014-01-19 12:05:57 -08:00
Jack Cushman
c9d0b0ba7b Handle transfer-encoding:chunked; misc. replay bugs. 
- Add a ChunkedLineReader to deal with replays with the
transfer-encoding: chunked header.
- Catch UnicodeDecodeErrors caused by multibyte characters getting
split during buffering.
- A couple of tiny bugs in replay.py
 2014-01-18 21:32:49 -05:00
Ilya Kreymer
7ce6d0d22b first pass on html rendering via jinja, support for query (cdx) rendering 2014-01-17 16:24:36 -08:00
Ilya Kreymer
bcc9588c00 * archivalrouter: to take a list of handlers, 
currently MatchPrefix and MatchRegex. handler returns a single response
(no chaining for now)
* rewriting: don't rewrite anchor only urls
* perf: add a very basic profiler in WBHandler for testing
 2014-01-16 20:33:51 -08:00
Ilya Kreymer
8ff2f2fc0c update gitignore 2014-01-06 21:57:33 -10:00
Ilya Kreymer
bc104321c4 Update README.md 2014-01-04 06:12:27 +00:00
Ilya Kreymer
c60493bfdc update README.md 2014-01-04 05:55:17 +00:00
Ilya Kreymer
c4457abc4c Update README 
Rename FullHandler -> WBHandler
Add additional comments!
 2014-01-03 21:44:20 -08:00
Ilya Kreymer
d820a8c06a add some comments, make charset parsing lower() 2014-01-03 17:40:20 -08:00
Ilya Kreymer
c255f4e47f fix typos 2014-01-03 17:04:15 -08:00
Ilya Kreymer
246b3fba43 cleanup, setup runnable testwb, or pluggable 'globalwb' 2014-01-04 00:21:52 +00:00
Ilya Kreymer
c3767cd31b fix css url parsing typo 
always default to utf-8 if chardet thinks ascii
tweak banner
 2014-01-03 21:38:18 +00:00
Ilya Kreymer
1e03cad25c update setup.py, static files 2014-01-03 13:06:27 -08:00
Ilya Kreymer
2357f108a3 rename rewriters 
header_rewriter added!
support for encoding detection
various fixes
xmlrewriter
 2014-01-03 13:03:03 -08:00
Ilya Kreymer
edbcaaf108 big update: refactor archiveloader, 
StatusAndHeaders obj and StatusAndHeaders parser
remove dependency on hanzo
Add sample example.warc.gz for very basic unit testing
 2014-01-02 20:21:18 -08:00
Ilya Kreymer
cca9071c53 minor tweaks, increase num closest searched, upper case url check 
css remove fixed pos
 2013-12-31 21:01:18 +00:00
Ilya Kreymer
d9930322f1 support utf-8 (so far) 
support protocol-agnostic prefix //
failedFile list for warc loading
 2013-12-31 00:18:12 +00:00
Ilya Kreymer
b8c4a453c9 wbhtml: add utf-8 tests 2013-12-29 22:42:29 -08:00
Ilya Kreymer
997dc5df0f fixes! Fix typos, in html parsing, fix base, support attrs w/o values 2013-12-30 03:03:33 +00:00
Ilya Kreymer
a84ec2abc7 first iteration of archival mode working w/ banner insertion!! 2013-12-28 17:39:43 -08:00
Ilya Kreymer
16f458d5ec archiveloader: Support for loading warc/arc records using hanzo parser (for record header parsing only) 
ReplayHandler: load replay from query response, find best option
basic support for matching url, checking self-redirects!
 2013-12-28 05:00:06 -08:00
Ilya Kreymer
787dfc136e wbhtml: add script and style doctests 
override close() to handle open <script> and <style> tags by forcing an end tag,
otherwise parser does not process the remainder
 2013-12-24 22:51:33 -08:00
Ilya Kreymer
6050ea1ffa standard JS and CSS rewriting working, with generic regex rewriter 
which supports extensions!
 2013-12-23 23:57:13 -08:00
Ilya Kreymer
3a896f7cd3 move norewrite prefixs down to ArchivalUrlRewriter (was in html parser) 
Add new general regex match work, (several attempts, though last one is simplest/best!)
 2013-12-23 15:52:33 -08:00
Ilya Kreymer
37e57f7013 html parser fleshed out! 2013-12-22 18:12:05 -08:00
Ilya Kreymer
fbf29e80d6 add html parser! 
urlrewriter support for changing modifier
 2013-12-20 19:11:52 -08:00
Ilya Kreymer
072befe3c8 archivalrouter: support handler chaining, using call convention and pass prev response 2013-12-20 15:10:12 -08:00
Ilya Kreymer
4cf4bf3bbb add wburlrewriter, ReferRedirect uses the rewriter 
more refactoring, ReferRedirect moved into archivalrouter module
wbrequest: parses from uri directly, keeps track of wburl and prefix
 2013-12-20 14:54:41 -08:00
Ilya Kreymer
0a2b16407d better exception handling, specific status codes for exceptions, 
detect access control and not found exceptions more consistently
 2013-12-19 12:06:47 -08:00
Ilya Kreymer
ebc76c0791 update readme 2013-12-18 18:57:55 -08:00
Ilya Kreymer
c8d2271e8a archiveurl: add support for url_query, format modifier for more unit tests 
archivalrouter: flesh out router seperately
indexreader: RemoteCDXServer reader
unit tests for req/resp
wbapp -- cdx output for query, urlquery, replay and latest_replay!
 2013-12-18 18:52:52 -08:00
Ilya Kreymer
5d42cc0cac rename aurl -> archiveurl, add default scheme, test for empty url 2013-12-13 15:43:07 -08:00
Ilya Kreymer
6b78f59e49 Merge branch 'master' of github.com:ikreymer/pywb 2013-12-13 15:22:06 -08:00
Ilya Kreymer
27b35f31e8 add basic wsgi app for parsing archivalurls, fallback on a referrer based redirect 2013-12-13 15:20:13 -08:00
ikreymer
d546fcc82c Merge pull request #1 from nlevitt/setup.py 
setuptools config
 2013-12-09 20:26:14 -08:00
Noah Levitt
89481f162e setuptools config 2013-12-09 11:58:50 -08:00