Ilya Kreymer
|
00bdddd1e9
|
recorder: SkipDupePolicy only skips if url is an exact match (not just by urlkey)
|
2016-04-07 10:44:05 -07:00 |
|
Ilya Kreymer
|
01c21d3a43
|
recorder: redis indexer accepts arg list, supports separate redis and key_template args
add length param to add_urls_to_index() in redis indexer, return cdx list
|
2016-04-02 21:36:36 -07:00 |
|
Ilya Kreymer
|
ba66d0bb5e
|
recorder: use res_template() to resolve params, rename indexing method to add_urls_to_index
|
2016-03-23 23:55:21 -04:00 |
|
Ilya Kreymer
|
c96e419341
|
recorder: ensure filename is also tracked by the indexer, add tests
for redis file mapping
|
2016-03-19 10:24:28 -07:00 |
|
Ilya Kreymer
|
b64be0dff1
|
recorder: add tests for single file writer, including file locking
dedup policy: support customizable dedup/skip/write policy plugins and add tests
|
2016-03-18 15:28:24 -07:00 |
|
Ilya Kreymer
|
9adb8da3b7
|
recorder: add support for filtering collections to record by regex (default: .*)
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
|
2016-03-11 11:12:25 -08:00 |
|
Ilya Kreymer
|
31fb2f926f
|
add recorder app, initial pass!
|
2016-03-09 14:33:36 -08:00 |
|