1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 16:14:48 +01:00

822 Commits

Author SHA1 Message Date
Ilya Kreymer
500a441ea9 README tweaks and edits from Dragan (@despens) 2015-03-23 16:16:16 -07:00
Ilya Kreymer
ec7a29a3ba static paths: ensure consistent renaming of static/default -> static/__pywb for bundled static path 2015-03-23 16:15:37 -07:00
Ilya Kreymer
5b4d12eb05 wombat: fix wombat_location.href assign when url is already rewritten, compare against current url not passed in url
fixes ikreymer/pywb-webrecorder#9
2015-03-23 16:12:58 -07:00
Ilya Kreymer
5020a09004 more CHANGES.rst updates 2015-03-23 15:43:05 -07:00
Ilya Kreymer
4aa6512b05 rewrite: fix WbUrl parsing for urls that start with a digit, eg. 1234.example.com
split latest replay url from timestamped replay regex
add additional rewrite tests
2015-03-23 15:38:10 -07:00
Ilya Kreymer
6acac67d3c rewrite: fix js rewrite again to ensure '// comments' are not rewritten as scheme-rel urls
add tests
2015-03-23 11:49:24 -07:00
Ilya Kreymer
bf0996c27a uwsgi: run with gevent loop by default, install gevent in run script 2015-03-23 11:05:17 -07:00
Ilya Kreymer
da7532a1f8 wb-manager: rename 'migrate' to 'cdx-convert' for clarity 2015-03-23 11:05:02 -07:00
Ilya Kreymer
0faa6aac3e setup: set version in pywb __init__.py 2015-03-23 11:04:41 -07:00
Ilya Kreymer
ced0ed208e Update CHANGELIST for 0.9.0 2015-03-23 10:48:58 -07:00
Ilya Kreymer
7681b4a634 Update INSTALL.rst 2015-03-23 10:36:37 -07:00
Ilya Kreymer
317a6c6e8e Update INSTALL.rst 2015-03-23 10:31:59 -07:00
Ilya Kreymer
6d879c10bb README work 2015-03-23 10:18:46 -07:00
Ilya Kreymer
4cfeb6d958 More README tweaks 2015-03-23 10:15:33 -07:00
Ilya Kreymer
e2623ed149 Update README.rst for latest update 2015-03-23 09:52:07 -07:00
Ilya Kreymer
df76bc3500 cli: change cdx-server and live-rewrite-server to go through shared cli
entry point
2015-03-23 09:08:09 -07:00
Ilya Kreymer
ae363ad368 autoindex and cli: add autoindex to cli with 'wayback -a' option, #81 2015-03-22 23:03:39 -07:00
Ilya Kreymer
e8db31d066 cli: improve wayback cli to take optional port, threads and working dir arguments
switch to waitress as default WSGI server instead of wsgiref
2015-03-22 21:50:56 -07:00
Ilya Kreymer
6a9a09d602 setup: add 'watchdog' as a dependency 2015-03-22 18:24:56 -07:00
Ilya Kreymer
733642551d manager: support autoindexing! (#91) wb-manager autoindex will use watchdog library to detect creation/updates
to any warc/arc in specified collection or across all and update autoindex cdx
cdx indexing: add --dir-root option to specify custom relative root dir for filenames used in cdx
2015-03-22 17:55:38 -07:00
Ilya Kreymer
cc068f8ee8 init/import path: move DEFAULT_CONFIG to __init__ for faster shared import
proxy: move certauth/openssl init to only happen in enable_https_proxy is set to
make slow openssl import run only when used
2015-03-22 17:52:07 -07:00
Ilya Kreymer
aa427bd6d0 rewrite: js regex: fix js rewrite regex to only match beginning of url for rewriting, since
rewrite just adding prefix for abs urls in js use case. (avoid dealing with any invalid chars that
may occur later in url)
2015-03-21 13:58:36 -07:00
Ilya Kreymer
d31ff68b93 auto_init: resolve rel paths only on init only if not http (though should support other protocols eventually) 2015-03-20 20:14:21 +00:00
Ilya Kreymer
b43a7f94f3 manager: add cdx -> cdxj migration tool #80, which will convert all cdxs in a directory to cdxj, removing original files
migration will also recanonicalize the urlkey to surt form
add migration test using non-surt, 9-field cdx (created from samples)
cdxindexer: fix multi warc->multi cdx indexing options
2015-03-19 20:57:33 -07:00
Ilya Kreymer
c5b5c8ee4b manager: fix index path to index.cdxj 2015-03-19 13:41:48 -07:00
Ilya Kreymer
ea460bb0f0 cdxj: support cdx json output from cdx server with output='json' (not yet default)
cdx field renaming: canonical cdx field name changes
statuscode -> status
mimetype -> mime
original -> url
old names still accept for query/filtering, however, cdx json will use new names
ensures consistency between .cdxj field names and names used by cdx server json output
collections manager now creates .cdxj by default
bump version to 0.9.0b2!
2015-03-19 13:33:49 -07:00
Ilya Kreymer
5221cbc64a add cdxj sample 2015-03-19 12:49:46 -07:00
Ilya Kreymer
fe1c32c8f7 cdxj: support loading cdxj (#76)
cdx obj: allow alt field names to be used (eg. mime, mimetype, m)
(status/statuscode/s) in querying and reading cdx
cdx minimal: (#75) now implies cdxj to avoid more formats
minimal includes digest always and mime when warc/revisit
tests for cdxj loading
indexing optimization: reuse same entry obj for records of same type
2015-03-19 12:36:49 -07:00
Ilya Kreymer
73f24f5a2b manager: fixes for windows: use shutil.move instead of os.rename to allow move to
existing file
tests: reset workdir before deleting temp dir
2015-03-18 13:14:05 -07:00
Ilya Kreymer
3f084625b0 indexing: cdx json support (#76): use OrderedDict when indexing json to ensure consistent ordering
skip empty or '-' fields
add tests for cdx json
2015-03-17 21:11:35 -07:00
Ilya Kreymer
6f9808f090 indexing: refactor ArchiveIndexEntry to be a dict instead of adding attrib. Allows for better track of indexed properties.
Add json-based cdx! (cdxj) output where all fields except url + key are in json dict. Support for both minimal and full json cdx, tracked via #76
2015-03-17 19:11:55 -07:00
Ilya Kreymer
bfe590996b auto-config: add support for loading from root ./static/ directory,
available under /static/__shared/ path
default path changed from /static/default -> /static/__pywb/
rename wayback-manager to wb-manager
2015-03-17 19:05:39 -07:00
Ilya Kreymer
0b8fd1e82e fix readme typos 2015-03-17 09:58:18 -07:00
Ilya Kreymer
0345e36daa readme: improve samples section 2015-03-17 01:13:10 -07:00
Ilya Kreymer
5b7215a6b1 readme tweaks and typo fixes 2015-03-17 01:06:06 -07:00
Ilya Kreymer
32ed176988 Update CHANGELIST for 0.9.0b1 2015-03-17 00:39:24 -07:00
Ilya Kreymer
e9e0412e1d More README tweaks 2015-03-17 00:28:14 -07:00
Ilya Kreymer
a60a735bd0 Update INSTALL.rst for 0.9.0 2015-03-17 00:14:10 -07:00
Ilya Kreymer
ab89ecd445 Brand new README for 0.9.0! 2015-03-17 00:01:32 -07:00
Ilya Kreymer
4b45e789df templates: ensure shared templates are loaded from root templates/ subdir
manager: add shared templates to templates subdir, not root dir #55 and #74
2015-03-16 19:57:28 -07:00
Ilya Kreymer
138aed3ddd change version to 0.9.0b1 2015-03-16 19:15:00 -07:00
Ilya Kreymer
2f6780a576 rename for 0.9.0:
rename default templates package from ui/* templates to templates/*
rename default subdirs: warcs -> archive, cdx -> indexes
2015-03-16 18:48:09 -07:00
Ilya Kreymer
19b8650891 manager: templates: add collections manager (#74) commands for adding, removing and listing
available ui templates. Support for both collection and shared templates.
confirmation for overwrite/remove
updated full template list in default_config and added tests
2015-03-16 16:55:06 -07:00
Ilya Kreymer
3d53fdde9e cleanup: remove unused __str__ from Handlers / Route, not as useful anymore 2015-03-15 22:55:23 -07:00
Ilya Kreymer
be5139b635 fix tests for coll listing, #78
config override: when loading from coll-specific config.yaml, resolve
relative paths to that collection, not to root #55
2015-03-15 22:23:08 -07:00
Ilya Kreymer
30454abb6b metadata: add support for user-defined per-collection metadata! #78
metadata stored in wbrequest.user_metadata and available to all templates

collections manager: refactor to use subparsers, add list collections and set metadata commands
update tests for new commands
index template: use user metadata title for collections listing
search template: display all metadata and title, if available
2015-03-15 21:24:15 -07:00
Ilya Kreymer
b417b47835 collections manager: support for merge when adding warc, explicit --index-warcs
option to index and merge instead of reindexing whole dir, #74
additional testing for recursive indexing, index merge
timeutils: add timestamp20_now() function
2015-03-14 14:56:15 -07:00
Ilya Kreymer
759d151551 tests: add test for directory auto collection loader,
collection manager and new 6-field minimal cdx format
2015-03-13 19:53:50 -07:00
Ilya Kreymer
1ba24de357 Merge branch 'develop' into config-work 2015-03-13 11:53:27 -07:00
Ilya Kreymer
b2ce3feb80 readme fix 2015-03-13 11:05:32 -07:00