Ilya Kreymer
2dbde35d74
bump to version to 0.9.2
2015-03-26 09:14:27 -07:00
Ilya Kreymer
cf4b5c50dd
more README.rst fixes
2015-03-25 22:08:53 -07:00
Ilya Kreymer
e8b6a1af88
README typo fixes
2015-03-25 21:52:38 -07:00
Ilya Kreymer
1cfe73c9db
zipnum: fix block count off-by-1 error in showNumPages query
2015-03-25 20:43:59 -07:00
Ilya Kreymer
72ddb54f82
Minor README tweaks
2015-03-25 15:01:12 -07:00
Ilya Kreymer
3efbfaa8c8
pywb_init: simplify DictChain usage, remove unused methods
2015-03-25 13:30:16 -07:00
Ilya Kreymer
f808f34ba7
Update CHANGES for 0.9.1
2015-03-25 12:16:26 -07:00
Ilya Kreymer
0e8b305adc
Update README to 0.9.1, add cdx api link, fix typo
2015-03-25 12:06:05 -07:00
Ilya Kreymer
15d1aea5ec
Update README, improve existing collection instructions.
2015-03-25 12:02:57 -07:00
Ilya Kreymer
a6c24c2882
autoindex: undo stop/join call for indexing, breaks os x unit test.. (autoindex test may need more improvements on windows)
2015-03-25 11:09:17 -07:00
Ilya Kreymer
90eee03cdb
fixes for windows:
...
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
2015-03-25 10:56:53 -07:00
Ilya Kreymer
a7307a6d98
pywb_init: auto-collections init: inherit shared archive_paths, if any are set in main config.yaml
2015-03-25 09:36:00 -07:00
Ilya Kreymer
6a3ca566db
zipnum: cleanup shared location resolution, in addition .loc file,
...
support a prefix resolver, where can be a regex replacement on the index path
(default is unchanged index path) (#83 )
2015-03-25 09:07:54 -07:00
Ilya Kreymer
1a8211d752
cdx server: add simplified matchType notation, using host* for prefix and *.host for domain matchType
...
(#34 )
2015-03-24 19:49:54 -07:00
Ilya Kreymer
2af5a25009
zipnum: support for pagination api! #34 and #83 . cdx server now bounded by pageSize (default 10 blocks),
...
showNumPages=true returns json indicating num pages, page=N can be set to page number 0-numPages - 1
loaders: add read_last_line() to read last line of a seekable file, used to read last line of index file when
at end
tests: additional test for binsearch boundary conditions
zipnum: secondary index output supports json also
2015-03-24 18:56:13 -07:00
Ilya Kreymer
872607c07d
README: move new features towards the top
2015-03-24 10:56:56 -07:00
Ilya Kreymer
3dd600c530
wombat: improve document.write override to write each elem at a time for body as well as head, #82
2015-03-24 10:46:10 -07:00
Ilya Kreymer
e5f321e32f
bump version to 0.9.1 for further dev
2015-03-23 20:21:09 -07:00
Ilya Kreymer
57be9ca7bc
tweak CHANGES.rst and INSTALL.rst for release
2015-03-23 17:38:22 -07:00
Ilya Kreymer
cda9f435a3
update README for final 0.9.0 release
2015-03-23 17:34:16 -07:00
Ilya Kreymer
c93501e16d
more changes.rst updates
2015-03-23 16:29:18 -07:00
Ilya Kreymer
500a441ea9
README tweaks and edits from Dragan (@despens)
2015-03-23 16:16:16 -07:00
Ilya Kreymer
ec7a29a3ba
static paths: ensure consistent renaming of static/default -> static/__pywb for bundled static path
2015-03-23 16:15:37 -07:00
Ilya Kreymer
5b4d12eb05
wombat: fix wombat_location.href assign when url is already rewritten, compare against current url not passed in url
...
fixes ikreymer/pywb-webrecorder#9
2015-03-23 16:12:58 -07:00
Ilya Kreymer
5020a09004
more CHANGES.rst updates
2015-03-23 15:43:05 -07:00
Ilya Kreymer
4aa6512b05
rewrite: fix WbUrl parsing for urls that start with a digit, eg. 1234.example.com
...
split latest replay url from timestamped replay regex
add additional rewrite tests
2015-03-23 15:38:10 -07:00
Ilya Kreymer
6acac67d3c
rewrite: fix js rewrite again to ensure '// comments' are not rewritten as scheme-rel urls
...
add tests
2015-03-23 11:49:24 -07:00
Ilya Kreymer
bf0996c27a
uwsgi: run with gevent loop by default, install gevent in run script
2015-03-23 11:05:17 -07:00
Ilya Kreymer
da7532a1f8
wb-manager: rename 'migrate' to 'cdx-convert' for clarity
2015-03-23 11:05:02 -07:00
Ilya Kreymer
0faa6aac3e
setup: set version in pywb __init__.py
2015-03-23 11:04:41 -07:00
Ilya Kreymer
ced0ed208e
Update CHANGELIST for 0.9.0
2015-03-23 10:48:58 -07:00
Ilya Kreymer
7681b4a634
Update INSTALL.rst
2015-03-23 10:36:37 -07:00
Ilya Kreymer
317a6c6e8e
Update INSTALL.rst
2015-03-23 10:31:59 -07:00
Ilya Kreymer
6d879c10bb
README work
2015-03-23 10:18:46 -07:00
Ilya Kreymer
4cfeb6d958
More README tweaks
2015-03-23 10:15:33 -07:00
Ilya Kreymer
e2623ed149
Update README.rst for latest update
2015-03-23 09:52:07 -07:00
Ilya Kreymer
df76bc3500
cli: change cdx-server and live-rewrite-server to go through shared cli
...
entry point
2015-03-23 09:08:09 -07:00
Ilya Kreymer
ae363ad368
autoindex and cli: add autoindex to cli with 'wayback -a' option, #81
2015-03-22 23:03:39 -07:00
Ilya Kreymer
e8db31d066
cli: improve wayback cli to take optional port, threads and working dir arguments
...
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
6a9a09d602
setup: add 'watchdog' as a dependency
2015-03-22 18:24:56 -07:00
Ilya Kreymer
733642551d
manager: support autoindexing! ( #91 ) wb-manager autoindex will use watchdog library to detect creation/updates
...
to any warc/arc in specified collection or across all and update autoindex cdx
cdx indexing: add --dir-root option to specify custom relative root dir for filenames used in cdx
2015-03-22 17:55:38 -07:00
Ilya Kreymer
cc068f8ee8
init/import path: move DEFAULT_CONFIG to __init__ for faster shared import
...
proxy: move certauth/openssl init to only happen in enable_https_proxy is set to
make slow openssl import run only when used
2015-03-22 17:52:07 -07:00
Ilya Kreymer
aa427bd6d0
rewrite: js regex: fix js rewrite regex to only match beginning of url for rewriting, since
...
rewrite just adding prefix for abs urls in js use case. (avoid dealing with any invalid chars that
may occur later in url)
2015-03-21 13:58:36 -07:00
Ilya Kreymer
d31ff68b93
auto_init: resolve rel paths only on init only if not http (though should support other protocols eventually)
2015-03-20 20:14:21 +00:00
Ilya Kreymer
b43a7f94f3
manager: add cdx -> cdxj migration tool #80 , which will convert all cdxs in a directory to cdxj, removing original files
...
migration will also recanonicalize the urlkey to surt form
add migration test using non-surt, 9-field cdx (created from samples)
cdxindexer: fix multi warc->multi cdx indexing options
2015-03-19 20:57:33 -07:00
Ilya Kreymer
c5b5c8ee4b
manager: fix index path to index.cdxj
2015-03-19 13:41:48 -07:00
Ilya Kreymer
ea460bb0f0
cdxj: support cdx json output from cdx server with output='json' (not yet default)
...
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
5221cbc64a
add cdxj sample
2015-03-19 12:49:46 -07:00
Ilya Kreymer
fe1c32c8f7
cdxj: support loading cdxj ( #76 )
...
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75 ) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
2015-03-19 12:36:49 -07:00
Ilya Kreymer
73f24f5a2b
manager: fixes for windows: use shutil.move instead of os.rename to allow move to
...
existing file
tests: reset workdir before deleting temp dir
2015-03-18 13:14:05 -07:00
