backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
2,096 Commits 56 Branches 63 Tags
Commit Graph

2096 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
86a093d164 support cdx server query at (/cdx in default config) 
also enable /echo_env and /echo_req debug handlers
 2014-02-01 00:43:24 -08:00
Ilya Kreymer
f00ac826cf fix typo in setup.py 2014-02-01 00:15:31 -08:00
Ilya Kreymer
2f5ffb3a88 switch test framework to use py.test instead of nose 2014-02-01 00:12:11 -08:00
Ilya Kreymer
6d2c8286ca render_response has option to pass in statuscode 2014-01-31 19:45:01 -08:00
Ilya Kreymer
44ef14b022 add first integration tests with WebTest! 
covers home page, search page, replay, calendar, redirect + replay, cdx
 2014-01-31 19:41:44 -08:00
Ilya Kreymer
bd94e3c656 fix replay_resolvers tests, don't use abs paths! 2014-01-31 10:33:47 -08:00
Ilya Kreymer
304ddbec84 Support for new UI, as per #16 
* Refactor views class to support more Jinja2 views (J2Template)
* Add a home page, collection search page, and error pages, all optional
* all exceptions appear on error page
* wbrequest supports a request with an empty or / wb_url
 2014-01-31 10:04:21 -08:00
Ilya Kreymer
57fe9515db - support for running uwsgi with virtualenv 
- text changes in banner
- some info about testing in README
 2014-01-29 17:23:19 -08:00
Ilya Kreymer
467d880681 update README 2014-01-29 15:15:39 -08:00
Ilya Kreymer
53eb5072ec more README tweaks 2014-01-29 15:12:57 -08:00
Ilya Kreymer
28618c69c6 update query.html, listing unique timestamps 
update README
 2014-01-29 15:07:45 -08:00
Ilya Kreymer
e7b70ae496 fix links in README 2014-01-29 12:08:51 -08:00
Ilya Kreymer
f45234f39b README tweaks 2014-01-29 12:07:33 -08:00
Ilya Kreymer
a6cfe9a87b update README.md 2014-01-29 12:01:03 -08:00
Ilya Kreymer
937fc7229e update README, fix typo 2014-01-29 02:12:58 -08:00
Ilya Kreymer
9cde058ccf check for osx uwsgi path and use that, otherwise run 'uwsgi' 2014-01-29 02:12:54 -08:00
Ilya Kreymer
84ffec9b8d update README.md 2014-01-29 01:52:30 -08:00
Ilya Kreymer
eb9cef9e28 update README.md for beta!!! 2014-01-29 01:36:31 -08:00
Ilya Kreymer
7a20d26d5f support non-surt ordered cdx 
add unsurt() util func and surt_ordered init param to LocalCDXServer
test make_best_resolver()
 2014-01-29 00:58:37 -08:00
Ilya Kreymer
9a3449dfd5 add pyyaml to dependency 2014-01-29 00:04:54 -08:00
Ilya Kreymer
411e7fe8a3 cleanup pywb_init, work on documenting config.yaml! 2014-01-29 00:03:24 -08:00
Ilya Kreymer
43a46b373d move sample/test data to ./sample_archive/warcs and ./sample_archive/cdx 
pywb_init now driven by config.yaml! (#14)

Not yet supporting customized handlers, views, etc...
 2014-01-28 22:03:01 -08:00
Ilya Kreymer
35f7cb0477 new-feature: support jinja2 template generated banner 
template receives cdx and wbrequest
default template inserts capture time into banner
 2014-01-28 20:18:47 -08:00
Ilya Kreymer
6de794a4e1 style fixes: convert camelCase func and var names to 'not_camel_case' 
WbHtml -> HTMLRewriter
ArchivalUrl -> WbUrl
 2014-01-28 19:37:37 -08:00
Ilya Kreymer
c0f8edf517 more refactoring: seperate top-level handlers (WBHandler) from 
views (html, text)
Add CDXHandler for interfacing with cdx server directly, #12
 2014-01-28 17:23:44 -08:00
Ilya Kreymer
1a234f2953 refactor: remove intermediate query object. 
rename query -> views
wbhandler queries index, replayer and renders via view

new feature: 'cdx_' modifier can be used to render cdx from any request
 2014-01-28 16:41:19 -08:00
Ilya Kreymer
a83d527702 add surt to dependency list 2014-01-27 22:07:27 -08:00
Ilya Kreymer
a6458b056f some tweaks on transfer-encoding: always remove and serve unchunked 
(should allow front-end serve can rechunk as needed)
 2014-01-27 22:05:49 -08:00
Ilya Kreymer
8732499dd5 - cdx server bootstrap configured, #12 
- pywb_init module inits from ./test directory

misc:
- router has lookahead for '/'
- dechunk even for transparent/binary
- 'text' query mode displays cdx
 2014-01-27 21:46:38 -08:00
Ilya Kreymer
c55bdf0e1f -binsearch: add tests, support both prefix and exact loading, for #11 
-cdx server first pass for #12: implement cdx parsing and transforming
-operations supported: merge sort, regex filter, resolve revisits, closest sort, reverse sort,
timestamp collapse
timestamp parsing utils
 2014-01-27 17:02:48 -08:00
Ilya Kreymer
e1b669fdea improved customization: can setup pywb_init.pywb_config() config, 
or specify custom init module <initmodule>.py_config() by
setting PYWB_INIT=<initmodule>
fix run.sh to support testing with custom mount point
 2014-01-24 12:25:27 -08:00
Ilya Kreymer
44f68158a9 update README and comments 2014-01-24 01:17:18 -08:00
Ilya Kreymer
1033feb2f8 use sample settings if driver file not found 2014-01-24 00:59:15 -08:00
Ilya Kreymer
391f3bf81d remove pycdx_server pkg for now, move binsearch into pywb package, 
update setup.py
 2014-01-24 00:54:48 -08:00
Ilya Kreymer
03b6938b9c referer fallback: check for non empty SCRIPT_NAME when parsing referrer 2014-01-24 00:53:55 -08:00
Ilya Kreymer
94326dafc1 html_rewriter: default attrs without value to empty str value, instead of no value 2014-01-24 00:52:17 -08:00
Ilya Kreymer
5987a0c047 update README.md! 2014-01-23 16:30:37 -08:00
Ilya Kreymer
cbf0e23ad9 add .travis.yml for Travis CI! 2014-01-23 16:20:51 -08:00
Ilya Kreymer
e95e17b9e6 pycdx_server initial binsearch module, with support exact match iterator! 
fix html_rewriter missing ; on entities
js rewriter: only rewrite full document.domain
PathIndexPrefixResolver using binsearch on path index, for #9
resolvers moved to replay_resolvers.py

improve path-resolver logic: each resolver returns an array of possible
files (could be from primary or secondary storage).
then, iterate over all possible files from all resolvers until
a successful load, or raise exception if all failed
 2014-01-23 01:38:09 -08:00
Ilya Kreymer
b237b144ff further refactor steaming of responses related to #13: always create a generator from 
response stream, and if buffering, read entire generator into temp buffer
remove duplicate reading logic
 2014-01-22 17:55:55 -08:00
Ilya Kreymer
2d0cb5745d enable bulk doctest testing via nosetests --with-doctest 
as well as individual doctests
andd utils.enable_doctests() func which checks if executing
app is nosetests (is there a better a way?)
 2014-01-22 15:28:01 -08:00
Ilya Kreymer
7722014a96 Cleanup rewrite interfaces to address #13 
All rewriters can support either buffered or streaming mode.
In buffered mode, the full text content is written into a buffer
and served with a Content-Length
in streaming mode, text is streamed as it is rewritten and
no Content-Length is written
Default is to stream the response
 2014-01-22 14:03:41 -08:00
ikreymer
33c135b337 Merge pull request #7 from jcushman/master 
Robust chunked data exception handling.
 2014-01-21 19:23:03 -08:00
Jack Cushman
6581f54fad Robust chunked data exception handling. 2014-01-21 20:00:52 -05:00
Ilya Kreymer
a1cd40fba1 support replay of records that have Transfer-Encoding: chunked, but 
were not actually rewritten to the warc as chunked.
Attempt to parse chunk length, and if failed, fallback to treating
record as not chunked
 2014-01-20 23:06:45 -08:00
Ilya Kreymer
8fd10673e8 refactor: cleanup the revisit resolving logic in replay 
also, update documented logic on wiki at:
https://github.com/ikreymer/pywb/wiki/PyWb-Record-Lookup-and-Revisits
 2014-01-20 17:52:14 -08:00
ikreymer
9a28a2ec6e Merge pull request #6 from jcushman/master 
Handle ArchivalUrl subclasses.
 2014-01-20 13:08:35 -08:00
Jack Cushman
903583c3d7 Handle ArchivalUrl subclasses. 2014-01-20 14:13:16 -05:00
Ilya Kreymer
9ff3fc300b Fix #5, bringing back customParams optional params sent to cdx server 
Rename archivalrouter.MatchRegex -> archivalrouter.Route, supporting regex/prefix matching
add redir_to_exact to turn off redirect to exact timestamp in RewritingReplayHandler
update README
 2014-01-20 10:50:06 -08:00
Ilya Kreymer
80b2585d22 Should resolve #4 -- supports pywb running as a non-root app 
* Instead of relying on REQUEST_URI, pywb constructs a
REL_REQUEST_URI, from PATH_INFO + QUERY_STRING.
SCRIPT_NAME auto-added to prefix
* MatchPrefix is now superceded by MatchRegex, which
can match a plain string -- collId defaults to the full match
* Added optional archivalurl_class to router to allow for customized
ArchivalUrl implementations to be specified
* run.sh can test on a non-root mountpoint, eg. ./run.sh "/approot"
 2014-01-19 21:13:48 -08:00