Ilya Kreymer
0784e4e5aa
spin-off warcio!
...
update imports to point to warcio
warcio rename fixes:
- ArcWarcRecord.stream -> raw_stream
- ArcWarcRecord.status_headers -> http_headers
- ArchiveLoadFailed single param init
2017-03-07 10:58:00 -08:00
Ilya Kreymer
ab77c1b6d9
refactor autoindex: switch to gevent-based simple polling, as watchdog doesn't work with gevent #200
2016-11-11 10:31:48 -08:00
Ilya Kreymer
3a584a1ec3
py3: all tests pass, at last!
...
but not yet py2... need to resolve encoding in rewriting issues
2016-02-23 13:26:53 -08:00
Ilya Kreymer
bc40352bed
cdx: add support for 10-field cdx format (old OpenWayback format) to ensure it can be converted to cdxj
...
manager: fix convert-cdx -> cdx-convert as explained in the README
2015-08-25 22:54:38 +03:00
Ilya Kreymer
10d9a6ac9a
jinja templates: add 'templates' as default lookup dir, allow specifying
...
custom dir via config. when specifiying custom paths, need not use full dir
as per usual paradigm
2015-04-04 12:53:07 -07:00
Ilya Kreymer
a34607764e
manager: validate name on collection init: must start with wordchar and can contain wordchar or -
2015-04-03 01:18:35 -07:00
Ilya Kreymer
6bbbb51f6e
manager: relax template requirements, allow any collection template to also be added to shared dir
2015-03-26 19:40:43 -07:00
Ilya Kreymer
753300d5ed
manager: use absolute path when adding warcs, ( #84 )
2015-03-26 19:18:55 -07:00
Ilya Kreymer
a6c24c2882
autoindex: undo stop/join call for indexing, breaks os x unit test.. (autoindex test may need more improvements on windows)
2015-03-25 11:09:17 -07:00
Ilya Kreymer
90eee03cdb
fixes for windows:
...
indexing: ensure '/' always written to cdx
autoindex: improved test case, ensure threads exit with join
style: fix long lines
2015-03-25 10:56:53 -07:00
Ilya Kreymer
da7532a1f8
wb-manager: rename 'migrate' to 'cdx-convert' for clarity
2015-03-23 11:05:02 -07:00
Ilya Kreymer
ae363ad368
autoindex and cli: add autoindex to cli with 'wayback -a' option, #81
2015-03-22 23:03:39 -07:00
Ilya Kreymer
e8db31d066
cli: improve wayback cli to take optional port, threads and working dir arguments
...
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
733642551d
manager: support autoindexing! ( #91 ) wb-manager autoindex will use watchdog library to detect creation/updates
...
to any warc/arc in specified collection or across all and update autoindex cdx
cdx indexing: add --dir-root option to specify custom relative root dir for filenames used in cdx
2015-03-22 17:55:38 -07:00
Ilya Kreymer
b43a7f94f3
manager: add cdx -> cdxj migration tool #80 , which will convert all cdxs in a directory to cdxj, removing original files
...
migration will also recanonicalize the urlkey to surt form
add migration test using non-surt, 9-field cdx (created from samples)
cdxindexer: fix multi warc->multi cdx indexing options
2015-03-19 20:57:33 -07:00
Ilya Kreymer
ea460bb0f0
cdxj: support cdx json output from cdx server with output='json' (not yet default)
...
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
73f24f5a2b
manager: fixes for windows: use shutil.move instead of os.rename to allow move to
...
existing file
tests: reset workdir before deleting temp dir
2015-03-18 13:14:05 -07:00
Ilya Kreymer
3f084625b0
indexing: cdx json support ( #76 ): use OrderedDict when indexing json to ensure consistent ordering
...
skip empty or '-' fields
add tests for cdx json
2015-03-17 21:11:35 -07:00
Ilya Kreymer
6f9808f090
indexing: refactor ArchiveIndexEntry to be a dict instead of adding attrib. Allows for better track of indexed properties.
...
Add json-based cdx! (cdxj) output where all fields except url + key are in json dict. Support for both minimal and full json cdx, tracked via #76
2015-03-17 19:11:55 -07:00
Ilya Kreymer
bfe590996b
auto-config: add support for loading from root ./static/ directory,
...
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
2015-03-17 19:05:39 -07:00
Ilya Kreymer
4b45e789df
templates: ensure shared templates are loaded from root templates/ subdir
...
manager: add shared templates to templates subdir, not root dir #55 and #74
2015-03-16 19:57:28 -07:00
Ilya Kreymer
19b8650891
manager: templates: add collections manager ( #74 ) commands for adding, removing and listing
...
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
2015-03-16 16:55:06 -07:00
Ilya Kreymer
30454abb6b
metadata: add support for user-defined per-collection metadata! #78
...
metadata stored in wbrequest.user_metadata and available to all templates
collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
2015-03-15 21:24:15 -07:00
Ilya Kreymer
b417b47835
collections manager: support for merge when adding warc, explicit --index-warcs
...
option to index and merge instead of reindexing whole dir, #74
additional testing for recursive indexing, index merge
timeutils: add timestamp20_now() function
2015-03-14 14:56:15 -07:00
Ilya Kreymer
759d151551
tests: add test for directory auto collection loader,
...
collection manager and new 6-field minimal cdx format
2015-03-13 19:53:50 -07:00
Ilya Kreymer
60f33412ff
collections manager: add new collections manager, first pass #74
...
add cli 'wb-manager' tool
very preliminary, needs testing still
2015-02-25 13:19:20 -08:00
