5 Commits

Author	SHA1	Message	Date
Ilya Kreymer	ba66d0bb5e	recorder: use res_template() to resolve params, rename indexing method to add_urls_to_index	2016-03-23 23:55:21 -04:00
Ilya Kreymer	d38bb5a1fd	filters: add extensible 'skip filters', with default filters to accept certain collections, filter out ... recording of range requests. Opportunity to skip recording at request or response time RespWrapper handles reading stream fully on close() (no need for old ReadFullyStream), skips recording if read was interrupted/incomplete writer: avoiding writing duplicate content-length/content-type headers	2016-03-21 11:47:12 -07:00
Ilya Kreymer	c96e419341	recorder: ensure filename is also tracked by the indexer, add tests ... for redis file mapping	2016-03-19 10:24:28 -07:00
Ilya Kreymer	3452cf39e0	recorder: use more general MultiFileWARCWriter, supporting both keeping file open ... and one-warc-per record use cases	2016-03-18 21:40:41 -07:00
Ilya Kreymer	e81457df5f	rename WARCRecorder -> WARCWriter, add optional max_size to single warc recorder ... per-record recorder combines http response/req into single file	2016-03-18 19:49:14 -07:00