5 Commits

Author	SHA1	Message	Date
Ilya Kreymer	924b983a8f	dyn collection and all coll improvements: (#69) ... support dynamic collections, all collection with remote archives (eg. s3:// paths) - warcserver: allow custom dynamic collections index and archive path templates via 'dyn_index_path' and 'dyn_archive_path' - pathresolver: allow resolving wildcard path prefixes with collection, to support remote paths and avoid globbing - warcserver: don't add fixed collections dir to source to support resolving wildcard - pathresolver: add wildcard resolving s3 path test - referrer unrewrite: ensure referrer not empty	2017-09-29 04:20:51 +00:00
Ilya Kreymer	02f8fa9ff3	windows: fix file path to/from file:// url conversion, add ... from_file_url() and use to_file_url() more consistently resolvers: make_best_resolver() handles file:// urls, but not PrefixResolver itself	2017-09-28 08:37:04 -07:00
Ilya Kreymer	a870f7e91a	memento timemap and test improvements: ... - windows: fix paths for pathresolver test on windows - timemap: add tests for all collection timemap, add cdxj timemap test - timemap: only add original, timegate links for 'link' timemap	2017-09-28 07:15:58 -07:00
Ilya Kreymer	a32c6f089c	auto-all aggregate collection support: (#69) ... - enabled with 'all_coll' in config or --all-coll cli option, eg. --all-coll all to enable - supported for replay, timemap and cdx endpoints, uses wildcard '*' for coll name with directory aggregator - tests: record/replay tests updated to replay via all collection, check all collection cdxj	2017-09-28 02:08:31 -07:00
Ilya Kreymer	93921aadb7	Recorder App Support (#241) ... recording support: now available for dynamic collections via config - config.yaml 'recorder: live' entry enables /record/ subpath which records to any dynamic collections (can record from any collection, though usually live) - autoindex refactor: simplified, standalone AutoIndexer() -- indexes any changed warc files to autoindex.cdxj - windows autoindex support: also check for changed file size, as last modified time may not be changing - manager: remove autoindex, now part of main cli - tests: updated test_auto_colls with autoindex changes - tests: add record/replay tests for recording and replay	2017-09-21 22:12:57 -07:00