backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00
Code Issues Projects Releases Activity
1,128 Commits 56 Branches 63 Tags
Commit Graph

1128 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ilya Kreymer
134b90eca5 bump version to 0.9.4-dev 2015-04-03 00:45:12 -07:00
Ilya Kreymer
b379d15066 changelist fixes 2015-04-01 17:07:27 -07:00
Ilya Kreymer
6064b45bd4 set version to 0.9.3 2015-04-01 17:05:29 -07:00
Ilya Kreymer
b0773ca8b8 pywb_init: ability to override DirectoryCollsLoader with custom class 2015-04-01 17:03:36 -07:00
Ilya Kreymer
7a8a0e5244 update .gitignore 2015-04-01 16:32:49 -07:00
Ilya Kreymer
4d74d65b61 update CHANGES for 0.9.3 2015-04-01 16:32:13 -07:00
Ilya Kreymer
8bd6787595 'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay 
add additional test for inverse framed replay, #92
fix framed replay url replace slash
 2015-04-01 16:21:44 -07:00
Ilya Kreymer
bd21480db9 framed replay: add supporting for 'inverting' frame and replay modifiers, 
setting default mod to be top-frame and inner frame to be 'mp_' #92
can enable this mode by setting framed_replay: inverse instead of true
modifiers passed to client side script via wbinfo as well
 2015-04-01 10:13:56 -07:00
Ilya Kreymer
546cd8ac3a frame redirect: only attempt redirect if in 'framed' mode (add flag to wbinfo) 
ensure both uris are decoded before comparing for top-frame redirect
 2015-04-01 09:13:55 -07:00
Ilya Kreymer
c378cb5188 rewrite: check for closed before any use of readline() (2.6 may throw if closed), 
only use readline() if line alignment needed (non-html), related to #86 work
 2015-04-01 07:54:17 -07:00
Ilya Kreymer
e806a33289 add unclosed script sample 2015-04-01 07:13:51 -07:00
Ilya Kreymer
8e60a6464c chunkeddatareader: read(): catch ValueError when attempting to read again in case stream is already closed 2015-03-31 23:31:49 -07:00
Ilya Kreymer
990af5ee79 rewrite: add extra test for rewriting html with <script> tag that's never closed 2015-03-31 23:30:56 -07:00
Ilya Kreymer
c137dd30b8 misc fixes: remove extra debug logging 
add --framed option to 'live-rewrite-server' cli app
 2015-03-31 23:08:56 -07:00
Ilya Kreymer
199f552f73 rewrite: if no charset specified, attempt to read first 1024 bytes and set charset in header, 
to avoid charset warning if head insert exceeds 1024 bytes (#86)
also encode head insert with detected charset, if possible
chunkeddatareader: add read() function to ensure read will read upto specified
length across chunks
 2015-03-31 22:38:20 -07:00
Ilya Kreymer
30ab27bb1c indexing: support indexing (and even replay of) records where target-uri is a 'urn:' identifier (#91) 
for canonicalzation, treat urns as is, already canonical
for wburl, don't add http:// prefix if urn: prefix is present
add example-wpull warc for testing
 2015-03-30 17:23:50 -07:00
Ilya Kreymer
002fe6a338 certauth: change 'get_cert_for_host' -> 'cert_for_host' 2015-03-30 15:47:53 -07:00
Ilya Kreymer
dd30e3f2a7 refactor: fixes for compat with latest certauth>=1.1.0 2015-03-30 09:38:42 -07:00
Ilya Kreymer
cda7705075 split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth cli as 
the 'certauth' tool superceeds this functionality. (#90).
To use https proxy mode, 'pip install certauth' is required. (update travis config)
 2015-03-29 17:38:57 -07:00
Ilya Kreymer
273176bce5 cdx: when reading cdxj, and run into non-ascii chars in url, utf-8 encode and %-encode 2015-03-29 09:21:50 -07:00
Ilya Kreymer
fc9d659b5d loaders: switch BlockLoader to use requests instead of urliib2 2015-03-28 16:41:52 -07:00
Ilya Kreymer
f3a066f58b cdx-server query & zipnum: fixes for showNumPages query: 
- if query contained in <1 secondary index block, must read first line of cdx to determine if any matches
- if no matches, don't throw 404 exception but always return json info with 0 pages
 2015-03-28 16:15:24 -07:00
Ilya Kreymer
313a2efeac bump version to 0.9.3-dev 2015-03-28 16:12:28 -07:00
Ilya Kreymer
c3a108b169 minor readme tweaks 2015-03-27 09:31:17 -07:00
Ilya Kreymer
d2be90d4a1 test case tweak 2015-03-27 08:56:43 -07:00
Ilya Kreymer
41487dd9d4 update changelist for 0.9.2 
cdx: include match type in cdx query error
 2015-03-27 07:58:51 -07:00
Ilya Kreymer
8d686a4a98 README typos fix 2015-03-26 19:56:09 -07:00
Ilya Kreymer
6bbbb51f6e manager: relax template requirements, allow any collection template to also be added to shared dir 2015-03-26 19:40:43 -07:00
Ilya Kreymer
753300d5ed manager: use absolute path when adding warcs, (#84) 2015-03-26 19:18:55 -07:00
Ilya Kreymer
6ce75f80f5 replay: remove restricting to provided http Content-Length (in addition to record content-length) as it may be incorrect for variety of reasons 2015-03-26 17:12:38 -07:00
Ilya Kreymer
0a4e97baa1 revisit resolving: if cdx digest is missing, attempt to resolve revisits based on url + timestamp only, if warc-refers-to-target-uri and warc-refers-to-date are available, even if warc-refers-to-target-uri == target-uri (see #88 for more info) 2015-03-26 14:20:08 -07:00
Ilya Kreymer
85082e46bf cdxj: ensure revisit resolve is skipped if the digest is missing, as may be case in cdxj (#85) 2015-03-26 11:11:10 -07:00
Ilya Kreymer
2dbde35d74 bump to version to 0.9.2 2015-03-26 09:14:27 -07:00
Ilya Kreymer
cf4b5c50dd more README.rst fixes 2015-03-25 22:08:53 -07:00
Ilya Kreymer
e8b6a1af88 README typo fixes 2015-03-25 21:52:38 -07:00
Ilya Kreymer
1cfe73c9db zipnum: fix block count off-by-1 error in showNumPages query 2015-03-25 20:43:59 -07:00
Ilya Kreymer
72ddb54f82 Minor README tweaks 2015-03-25 15:01:12 -07:00
Ilya Kreymer
3efbfaa8c8 pywb_init: simplify DictChain usage, remove unused methods 2015-03-25 13:30:16 -07:00
Ilya Kreymer
f808f34ba7 Update CHANGES for 0.9.1 2015-03-25 12:16:26 -07:00
Ilya Kreymer
0e8b305adc Update README to 0.9.1, add cdx api link, fix typo 2015-03-25 12:06:05 -07:00
Ilya Kreymer
15d1aea5ec Update README, improve existing collection instructions. 2015-03-25 12:02:57 -07:00
Ilya Kreymer
a6c24c2882 autoindex: undo stop/join call for indexing, breaks os x unit test.. (autoindex test may need more improvements on windows) 2015-03-25 11:09:17 -07:00
Ilya Kreymer
90eee03cdb fixes for windows: 
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
 2015-03-25 10:56:53 -07:00
Ilya Kreymer
a7307a6d98 pywb_init: auto-collections init: inherit shared archive_paths, if any are set in main config.yaml 2015-03-25 09:36:00 -07:00
Ilya Kreymer
6a3ca566db zipnum: cleanup shared location resolution, in addition .loc file, 
support a prefix resolver, where can be a regex replacement on the index path
(default is unchanged index path) (#83)
 2015-03-25 09:07:54 -07:00
Ilya Kreymer
1a8211d752 cdx server: add simplified matchType notation, using host* for prefix and *.host for domain matchType 
(#34)
 2015-03-24 19:49:54 -07:00
Ilya Kreymer
2af5a25009 zipnum: support for pagination api! #34 and #83. cdx server now bounded by pageSize (default 10 blocks), 
showNumPages=true returns json indicating num pages, page=N can be set to page number 0-numPages - 1
loaders: add read_last_line() to read last line of a seekable file, used to read last line of index file when
at end
tests: additional test for binsearch boundary conditions
zipnum: secondary index output supports json also
 2015-03-24 18:56:13 -07:00
Ilya Kreymer
872607c07d README: move new features towards the top 2015-03-24 10:56:56 -07:00
Ilya Kreymer
3dd600c530 wombat: improve document.write override to write each elem at a time for body as well as head, #82 2015-03-24 10:46:10 -07:00
Ilya Kreymer
e5f321e32f bump version to 0.9.1 for further dev 2015-03-23 20:21:09 -07:00