backup/pywb
1
0
Fork 0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-19 10:19:37 +01:00
Code Issues Projects Releases Activity
103 Commits 56 Branches 63 Tags
Commit Graph

31 Commits

Author SHA1 Message Date
Ilya Kreymer
db3b92e228 writing: add write_stream_to_file()function to be able to write to a WARC an existing input stream 
refactor _do_write_req_resp to pass callback to actual writing (eg. _write_to_file)
 2016-07-31 00:49:57 -04:00
Ilya Kreymer
1b09015954 recorder: split up _open_file() into get_new_filename() and allow_new_file() to customize skipping recording by returning false 
from allow_new_file()
create_warcinfo_record() - switch to dict args over kwargs, update tests
 2016-07-30 13:11:12 -04:00
Ilya Kreymer
14cf68e4e5 custom record: don't override WARC-Date if provided in request header, 
return chosen WARC-Date in json response
 2016-07-26 19:41:47 -04:00
Ilya Kreymer
4c7da0f6ef recorder: support overridings get_params() in subclass 
multiwarcwriter: support multiple warcs in same dir, support random component in path, and a custom
key template for selecting current warc file, not related to current directory
 2016-06-07 12:55:04 -04:00
Ilya Kreymer
d7c74b68de video loader support: add VideoLoader, which uses youtube-dl to create a metadata record 
of video info. Activated with explicit content_type param 'application/vnd.youtube-dl_formats+json'
 2016-05-28 15:01:33 -07:00
Ilya Kreymer
30f9d0aca7 recorder put custom record: add support for put/post of a custom record. If put_record= param is included, the request body 
is written to the specified record type.
move record creation functions to the warcwriter
add tests for custom record
 2016-05-26 20:49:40 -07:00
Ilya Kreymer
45c8fcddbd recorder: add max_idle_secs / close_idle_files() to close any open files that have not been modified longer than set threshold, in prep for webrecorder/webrecorder#92 
indexer: add 'full_warc_prefix' for setting full path prefix in add_warc_file() (eg. for http load) for webrecorder/webrecorder#95
 2016-05-11 21:40:02 -07:00
Ilya Kreymer
94d6098238 app: separate json_encode() func 
compat: py2 fixes
 2016-05-11 11:38:59 -07:00
Ilya Kreymer
464eca2fa0 test apps: enable debugging for test apps 
test recorder: write to a temp dir for each run
 2016-05-06 16:33:18 -07:00
Ilya Kreymer
228ca58c5b recorer: actually fix content-type on warcinfo, add to test! 2016-04-30 13:07:53 -07:00
Ilya Kreymer
0fbae1c7f8 recorder: ensure warcinfo record has a content-type 2016-04-30 10:19:20 -07:00
Ilya Kreymer
f119d05724 recorder: fix simplerec init 
tests: improve tests for skipping request and response headers
 2016-04-27 09:52:56 -07:00
Ilya Kreymer
0b255819ff recorder warcwriter: allow skipping writing of only request or only response by overriding _is_write_req and _is_write_resp in subclass 
(todo: rethink the interface)
 2016-04-15 02:19:34 +00:00
Ilya Kreymer
00bdddd1e9 recorder: SkipDupePolicy only skips if url is an exact match (not just by urlkey) 2016-04-07 10:44:05 -07:00
Ilya Kreymer
d40edfc22d warcwriter: add create_warcinfo_record() for creating a warcinfo and a SimpleTempWARCWriter for writing records to temp buff/file 2016-04-03 12:19:54 -07:00
Ilya Kreymer
01c21d3a43 recorder: redis indexer accepts arg list, supports separate redis and key_template args 
add length param to add_urls_to_index() in redis indexer, return cdx list
 2016-04-02 21:36:36 -07:00
Ilya Kreymer
7884d4394b recorder: close_file() by params rather than exact path, update tests 2016-03-26 13:07:53 -04:00
Ilya Kreymer
61921d6c4a tests: add FakeRedisTests class mixin for patching in FakeRedis for tests 2016-03-24 10:45:48 -04:00
Ilya Kreymer
ba66d0bb5e recorder: use res_template() to resolve params, rename indexing method to add_urls_to_index 2016-03-23 23:55:21 -04:00
Ilya Kreymer
aa80cd6881 recorder: add simple recorder config indexing to redis 2016-03-21 11:50:01 -07:00
Ilya Kreymer
d38bb5a1fd filters: add extensible 'skip filters', with default filters to accept certain collections, filter out 
recording of range requests. Opportunity to skip recording at request or response time
RespWrapper handles reading stream fully on close() (no need for old ReadFullyStream),
skips recording if read was interrupted/incomplete
writer: avoiding writing duplicate content-length/content-type headers
 2016-03-21 11:47:12 -07:00
Ilya Kreymer
c96e419341 recorder: ensure filename is also tracked by the indexer, add tests 
for redis file mapping
 2016-03-19 10:24:28 -07:00
Ilya Kreymer
3452cf39e0 recorder: use more general MultiFileWARCWriter, supporting both keeping file open 
and one-warc-per record use cases
 2016-03-18 21:40:41 -07:00
Ilya Kreymer
e81457df5f rename WARCRecorder -> WARCWriter, add optional max_size to single warc recorder 
per-record recorder combines http response/req into single file
 2016-03-18 19:49:14 -07:00
Ilya Kreymer
b64be0dff1 recorder: add tests for single file writer, including file locking 
dedup policy: support customizable dedup/skip/write policy plugins and add tests
 2016-03-18 15:28:24 -07:00
Ilya Kreymer
cba8e4ee3a filters: more functional filter impl for header exclusion 2016-03-17 18:22:26 -07:00
Ilya Kreymer
06978bd8d2 recorder: check for empty input stream (support for direct proxy?) 2016-03-13 11:17:52 -07:00
Ilya Kreymer
709d2b1ea2 reorg: move StreamIter to utils 2016-03-12 23:29:23 -08:00
Ilya Kreymer
7a828017d1 recorder: clean up logging, ReadFullyStream moves to utils, get_request_uri to inputreq 2016-03-12 22:18:01 -08:00
Ilya Kreymer
9adb8da3b7 recorder: add support for filtering collections to record by regex (default: .*) 
add support for excluding certain headers when writing WARCs
tests: add first batch of tests for recorder, using live upstream server
 2016-03-11 11:12:25 -08:00
Ilya Kreymer
31fb2f926f add recorder app, initial pass! 2016-03-09 14:33:36 -08:00