1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

1128 Commits

Author SHA1 Message Date
Ilya Kreymer
134b90eca5 bump version to 0.9.4-dev 2015-04-03 00:45:12 -07:00
Ilya Kreymer
b379d15066 changelist fixes 2015-04-01 17:07:27 -07:00
Ilya Kreymer
6064b45bd4 set version to 0.9.3 2015-04-01 17:05:29 -07:00
Ilya Kreymer
b0773ca8b8 pywb_init: ability to override DirectoryCollsLoader with custom class 2015-04-01 17:03:36 -07:00
Ilya Kreymer
7a8a0e5244 update .gitignore 2015-04-01 16:32:49 -07:00
Ilya Kreymer
4d74d65b61 update CHANGES for 0.9.3 2015-04-01 16:32:13 -07:00
Ilya Kreymer
8bd6787595 'inverse' framed replay: ensure memento headers point to actual memento in inverse framed replay
add additional test for inverse framed replay, #92
fix framed replay url replace slash
2015-04-01 16:21:44 -07:00
Ilya Kreymer
bd21480db9 framed replay: add supporting for 'inverting' frame and replay modifiers,
setting default mod to be top-frame and inner frame to be 'mp_' #92
can enable this mode by setting framed_replay: inverse instead of true
modifiers passed to client side script via wbinfo as well
2015-04-01 10:13:56 -07:00
Ilya Kreymer
546cd8ac3a frame redirect: only attempt redirect if in 'framed' mode (add flag to wbinfo)
ensure both uris are decoded before comparing for top-frame redirect
2015-04-01 09:13:55 -07:00
Ilya Kreymer
c378cb5188 rewrite: check for closed before any use of readline() (2.6 may throw if closed),
only use readline() if line alignment needed (non-html), related to #86 work
2015-04-01 07:54:17 -07:00
Ilya Kreymer
e806a33289 add unclosed script sample 2015-04-01 07:13:51 -07:00
Ilya Kreymer
8e60a6464c chunkeddatareader: read(): catch ValueError when attempting to read again in case stream is already closed 2015-03-31 23:31:49 -07:00
Ilya Kreymer
990af5ee79 rewrite: add extra test for rewriting html with <script> tag that's never closed 2015-03-31 23:30:56 -07:00
Ilya Kreymer
c137dd30b8 misc fixes: remove extra debug logging
add --framed option to 'live-rewrite-server' cli app
2015-03-31 23:08:56 -07:00
Ilya Kreymer
199f552f73 rewrite: if no charset specified, attempt to read first 1024 bytes and set charset in header,
to avoid charset warning if head insert exceeds 1024 bytes (#86)
also encode head insert with detected charset, if possible
chunkeddatareader: add read() function to ensure read will read upto specified
length across chunks
2015-03-31 22:38:20 -07:00
Ilya Kreymer
30ab27bb1c indexing: support indexing (and even replay of) records where target-uri is a 'urn:' identifier (#91)
for canonicalzation, treat urns as is, already canonical
for wburl, don't add http:// prefix if urn: prefix is present
add example-wpull warc for testing
2015-03-30 17:23:50 -07:00
Ilya Kreymer
002fe6a338 certauth: change 'get_cert_for_host' -> 'cert_for_host' 2015-03-30 15:47:53 -07:00
Ilya Kreymer
dd30e3f2a7 refactor: fixes for compat with latest certauth>=1.1.0 2015-03-30 09:38:42 -07:00
Ilya Kreymer
cda7705075 split and refactor: remove certauth.py / test_certauth.py and instead use this functionality from 'certauth' package. Also remove proxy-cert-auth cli as
the 'certauth' tool superceeds this functionality. (#90).
To use https proxy mode, 'pip install certauth' is required. (update travis config)
2015-03-29 17:38:57 -07:00
Ilya Kreymer
273176bce5 cdx: when reading cdxj, and run into non-ascii chars in url, utf-8 encode and %-encode 2015-03-29 09:21:50 -07:00
Ilya Kreymer
fc9d659b5d loaders: switch BlockLoader to use requests instead of urliib2 2015-03-28 16:41:52 -07:00
Ilya Kreymer
f3a066f58b cdx-server query & zipnum: fixes for showNumPages query:
- if query contained in <1 secondary index block, must read first line of cdx to determine if any matches
- if no matches, don't throw 404 exception but always return json info with 0 pages
2015-03-28 16:15:24 -07:00
Ilya Kreymer
313a2efeac bump version to 0.9.3-dev 2015-03-28 16:12:28 -07:00
Ilya Kreymer
c3a108b169 minor readme tweaks 2015-03-27 09:31:17 -07:00
Ilya Kreymer
d2be90d4a1 test case tweak 2015-03-27 08:56:43 -07:00
Ilya Kreymer
41487dd9d4 update changelist for 0.9.2
cdx: include match type in cdx query error
2015-03-27 07:58:51 -07:00
Ilya Kreymer
8d686a4a98 README typos fix 2015-03-26 19:56:09 -07:00
Ilya Kreymer
6bbbb51f6e manager: relax template requirements, allow any collection template to also be added to shared dir 2015-03-26 19:40:43 -07:00
Ilya Kreymer
753300d5ed manager: use absolute path when adding warcs, (#84) 2015-03-26 19:18:55 -07:00
Ilya Kreymer
6ce75f80f5 replay: remove restricting to provided http Content-Length (in addition to record content-length) as it may be incorrect for variety of reasons 2015-03-26 17:12:38 -07:00
Ilya Kreymer
0a4e97baa1 revisit resolving: if cdx digest is missing, attempt to resolve revisits based on url + timestamp only, if warc-refers-to-target-uri and warc-refers-to-date are available, even if warc-refers-to-target-uri == target-uri (see #88 for more info) 2015-03-26 14:20:08 -07:00
Ilya Kreymer
85082e46bf cdxj: ensure revisit resolve is skipped if the digest is missing, as may be case in cdxj (#85) 2015-03-26 11:11:10 -07:00
Ilya Kreymer
2dbde35d74 bump to version to 0.9.2 2015-03-26 09:14:27 -07:00
Ilya Kreymer
cf4b5c50dd more README.rst fixes 2015-03-25 22:08:53 -07:00
Ilya Kreymer
e8b6a1af88 README typo fixes 2015-03-25 21:52:38 -07:00
Ilya Kreymer
1cfe73c9db zipnum: fix block count off-by-1 error in showNumPages query 2015-03-25 20:43:59 -07:00
Ilya Kreymer
72ddb54f82 Minor README tweaks 2015-03-25 15:01:12 -07:00
Ilya Kreymer
3efbfaa8c8 pywb_init: simplify DictChain usage, remove unused methods 2015-03-25 13:30:16 -07:00
Ilya Kreymer
f808f34ba7 Update CHANGES for 0.9.1 2015-03-25 12:16:26 -07:00
Ilya Kreymer
0e8b305adc Update README to 0.9.1, add cdx api link, fix typo 2015-03-25 12:06:05 -07:00
Ilya Kreymer
15d1aea5ec Update README, improve existing collection instructions. 2015-03-25 12:02:57 -07:00
Ilya Kreymer
a6c24c2882 autoindex: undo stop/join call for indexing, breaks os x unit test.. (autoindex test may need more improvements on windows) 2015-03-25 11:09:17 -07:00
Ilya Kreymer
90eee03cdb fixes for windows:
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
2015-03-25 10:56:53 -07:00
Ilya Kreymer
a7307a6d98 pywb_init: auto-collections init: inherit shared archive_paths, if any are set in main config.yaml 2015-03-25 09:36:00 -07:00
Ilya Kreymer
6a3ca566db zipnum: cleanup shared location resolution, in addition .loc file,
support a prefix resolver, where can be a regex replacement on the index path
(default is unchanged index path) (#83)
2015-03-25 09:07:54 -07:00
Ilya Kreymer
1a8211d752 cdx server: add simplified matchType notation, using host* for prefix and *.host for domain matchType
(#34)
2015-03-24 19:49:54 -07:00
Ilya Kreymer
2af5a25009 zipnum: support for pagination api! #34 and #83. cdx server now bounded by pageSize (default 10 blocks),
showNumPages=true returns json indicating num pages, page=N can be set to page number 0-numPages - 1
loaders: add read_last_line() to read last line of a seekable file, used to read last line of index file when
at end
tests: additional test for binsearch boundary conditions
zipnum: secondary index output supports json also
2015-03-24 18:56:13 -07:00
Ilya Kreymer
872607c07d README: move new features towards the top 2015-03-24 10:56:56 -07:00
Ilya Kreymer
3dd600c530 wombat: improve document.write override to write each elem at a time for body as well as head, #82 2015-03-24 10:46:10 -07:00
Ilya Kreymer
e5f321e32f bump version to 0.9.1 for further dev 2015-03-23 20:21:09 -07:00