backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
1,177 Commits 56 Branches 63 Tags
Commit Graph

1177 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
f578bdf5de add README.rst 2014-03-10 19:01:20 -07:00
Ilya Kreymer
45972df6c4 minor fixes, copyright update 2014-03-10 18:45:45 -07:00
Ilya Kreymer
3322fb233f fixup wb and wombat.js: 
fix formatting to 4-tab snake_case, remove obsolete code
 2014-03-10 00:55:41 -07:00
Ilya Kreymer
e3d700a50f wombat improvements: override history, ajax and use 
seeded random number gen (with seed from capture timestamp)
 2014-03-10 00:10:20 -07:00
Ilya Kreymer
e346dfb024 remove accidental logging 2014-03-09 23:03:55 -07:00
Ilya Kreymer
e384425d48 proxy cleanup: move HttpsUrlRewriter to url_rewriter module, 
move strip_scheme to replay_views where it is used
regex rewriters: use url rewriter for rewriting http:// in JS,
instead of just prefix, to support custom rewriters (such as
https->http rewriter in proxy mode)
 2014-03-09 14:21:32 -07:00
Ilya Kreymer
68878fa72a update domain-specific rules to make flickr replay work better! 2014-03-08 15:53:52 -08:00
archiveit
4fdcdc98ae replay: ignore 304 captures 2014-03-08 23:46:59 +00:00
Ilya Kreymer
584d826f05 rewrite: fix html rewriting, if forcing end </script>, </style>, 
don't actually output to preserve original
wombat: copy over all Location settings
wburl: convert :/ -> :// if 2nd slash missing, only check for <scheme>:/
and ignore subsequent slashes
 2014-03-08 15:10:35 -08:00
Ilya Kreymer
541c076b77 setup: add cli scripts for wayback, cdx-server 
fix logging of app name, make most logging debug
 2014-03-08 15:09:53 -08:00
Ilya Kreymer
40b7a8e921 move pytest args to pytest.ini 2014-03-08 09:30:56 -08:00
Ilya Kreymer
3b1afc3e3d replace StringIO with BytesIO 2014-03-08 09:30:19 -08:00
ikreymer
1a6f2e2fe1 Merge pull request #32 from kngenie/add-api-docs 
add api doc pages for all modules with sphinx-apidoc
 2014-03-07 16:02:36 -08:00
Kenji Nagahashi
4e7ec4dede Merge remote-tracking branch 'origin/master' into add-api-docs 2014-03-07 20:14:40 +00:00
Kenji Nagahashi
1829de2123 add API doc for all packages with sphinx-apidoc 2014-03-07 20:09:33 +00:00
Ilya Kreymer
e3618871c8 proxy: support setting hostname via env variable 2014-03-07 11:42:09 -08:00
Ilya Kreymer
a60ab1f118 routing/proxy: pass in hostpaths to proxy routing 
add PYWB_HOST_NAME env var to allow overriding default hostname
add request_hostname jinja filter
 2014-03-07 10:29:11 -08:00
Ilya Kreymer
702e5e0143 perms test: moved test perms policy to perms/test/test_perms_policy.py 
all perms related configs exist within perms package
 2014-03-06 18:24:53 -08:00
Ilya Kreymer
681c2fd8d5 perms: refactor perms config to make interface much clearer 
'perms_policy' is a callback which returns a Perms object, which may
filter cdx lines from the response
 2014-03-06 18:06:05 -08:00
Ilya Kreymer
7b5cbaa878 cdx: clean up closest, reverse ops 
closest takes precedence over reverse
'reverse closest' not supported, add test to reflect that
 2014-03-06 16:11:46 -08:00
Ilya Kreymer
c42a96386f cdx: fix the 'yield nothing' case when limit==1 
add additional test case for limit==1 and reverse=True,
as limit is optimized out
 2014-03-06 16:01:49 -08:00
Ilya Kreymer
4e71a0b772 better rules.yaml fix 2014-03-06 02:51:54 -08:00
Ilya Kreymer
3718e1d21b rewrite fixes: html_rewriter do not unescape attrs! 
rules: don't rewrite past end of block or line
 2014-03-06 02:29:52 -08:00
Ilya Kreymer
673ff35d15 minor fixes: wombat add document.WB_wombat_location 
loaders: file 'urls' starting with . and / are always file paths
pep8 fixes for cdx, utils packages
 2014-03-05 17:13:14 -08:00
Kenji Nagahashi
28b49f9aeb add doc directory for Sphinx documentation 2014-03-05 23:03:04 +00:00
ikreymer
03ebca47c0 Merge pull request #29 from kngenie/just-a-cleanup 
clean up docstrings: fix reST formatting issues.
 2014-03-05 14:36:07 -08:00
Kenji Nagahashi
64f4699203 clean up docstrings: fix reST formatting issues. 
cherry-picked f03e0a7092 + some more.
 2014-03-05 22:07:27 +00:00
Ilya Kreymer
daf868fd61 README tweaks 
update setup.py to support setup.py test!
.travis.yml uses python setup.py test
 2014-03-05 11:19:26 -08:00
Ilya Kreymer
25a8514352 Update README (move pywb configuration section to wiki), 
recommend running pywb.apps.wayback
make uWSGI optional (but included in Vagrant)
rename run.sh -> run-uwsgi.sh
 2014-03-05 10:42:08 -08:00
Ilya Kreymer
fe1fa43fef zipnum: remove time-based reloading for now, just look at mtime 
and reload if changed
 2014-03-04 21:29:05 -08:00
Ilya Kreymer
df2f7ba496 warc: add digest filter only if digest is present for url-agnostic load 
ensure cdxobject format set on cdx load callback
limit reader: add length wrappign utility func to limitreader
 2014-03-05 05:12:25 +00:00
Ilya Kreymer
9690d84798 travis-ci: attempt to fix 2.6 build 2014-03-04 19:36:29 -08:00
Ilya Kreymer
f25de8af2a tweak travis pip install config 2014-03-04 19:17:33 -08:00
Ilya Kreymer
cc22448cc5 fixes for 2.6 and pypy 2014-03-04 19:11:17 -08:00
Ilya Kreymer
2d48f2d733 add testing of 2.6 and pypy (attempt) 2014-03-04 18:12:36 -08:00
Ilya Kreymer
202f6101e0 coverage work! add additional test for wsgi_wrappers 
additional test for zipnum bad location
for now, not testing cli interfaces which depend on opt params
 2014-03-04 16:13:49 -08:00
Ilya Kreymer
d702a98bbc url-agnostic revisit testing! 
add sample warc and cdx for url-agnostic revisits
add unit test and integration test
resolvingloader: pass callback instead of full cdx server
for use for loading cdx in case of url-agnostic revisit
 2014-03-04 20:12:09 +00:00
Ilya Kreymer
cf5aaf5de4 add new perms_handler for supporting direct permissions api 
currently just returning ["allow"] or ["block"] for a single url
 2014-03-03 19:37:37 -08:00
Ilya Kreymer
577c74be49 cdx: move perms related handling to pywb.perms package, support 
custom processing ops, of which perms is a specific type
add lazy_ops test to ensure all cdx processing ops are lazy

perms: set up a 'perms policy' factory and perms policy implementation
perms policy setting results in a custom processing op
update tests to work with new config
IndexReader handles both cdx server + perms policy
 2014-03-03 18:27:04 -08:00
Ilya Kreymer
e0d5846484 seperate 'perms_checker' config loading as a seperate param 
simplify IndexReader wrapper init, just init with a cdx server
 2014-03-03 13:40:48 -08:00
Ilya Kreymer
331976748e cdxops: make sure sort reverse and closest are lazy (create generators) 
perms: allow_url_lookup() only takes key param for simplicity
 2014-03-03 12:16:07 -08:00
ikreymer
5a28bc6992 Merge pull request #28 from ikreymer/pkg-reorg 
pywb pkg refactoring: create pywb.framework, pywb.core and pywb.apps
 2014-03-03 12:04:12 -08:00
Ilya Kreymer
2d4ae62fbe - cdx handler refactoring: factor out CDXHandler and init to 
seperate cdx_handler module
- Make wsgi app a class, add port as an optional field in wsgi app
and router. (not required to be specified)
 2014-03-03 10:35:57 -08:00
Ilya Kreymer
0bf651c2e3 add cdx_server app! 
port wsgi cdx server tests to test new app!
move base handlers to basehandlers in framework pkg
(remove werkzeug dependency)
 2014-03-02 23:41:44 -08:00
Ilya Kreymer
f0a0976038 more refactoring! 
create 'framework' subpackage for general purpose components!
contains routing, request/response, exceptions and wsgi wrappers
update framework package for pep8
dsrules: using load_config_yaml() (pushed to utils)
to init default config
 2014-03-02 21:42:05 -08:00
Ilya Kreymer
f1acad53fc wsgi wrapper reorg! 
support pluggable wsgi apps
utils: BlockLoader() supports loading from package
exceptions: base WbException moved to utils
 2014-03-02 19:26:06 -08:00
Ilya Kreymer
47271bbfab remove extra .gz file, change test to use zipnum file instead 2014-03-02 08:55:26 -08:00
Ilya Kreymer
19f86305bf update pkg-reorg with changes from master, including 
CDXQuery configuration
 2014-03-02 00:26:29 -08:00
Ilya Kreymer
06a22c845b ensure cdx loading happens lazily 
add perms test to ensure 'short-circuiting' in case of
permission exception
 2014-03-01 18:40:16 -08:00
Ilya Kreymer
15d2cdd1b3 cdx: cleanup regarding and more consistency for RemoteCDXServer 
RemoteCDXServer delegates filter/processing and simply proxies response from remote
RemoteCDXSource (and default usage with CDXServer) only fetches the unfiltered/unprocessed
stream and performs cdx ops locally
 2014-03-01 16:35:27 -08:00