Ilya Kreymer
41b3789412
update to warcio>=1.3.4
...
http adapter: use same defaults for live and remote
2017-07-03 09:01:21 -07:00
Ilya Kreymer
f2c2829f49
misc improvements:
...
redis multi-key source: store member listing from hgetall 'scan:<key>' key
add 'recorder-skip' to cdx line also
use latest warcio (1.3.3)
2017-05-31 16:08:20 -07:00
Ilya Kreymer
147c3217dd
update to warcio==1.3
...
recorder: use ArcWarcRecordLoader() for parsing response record
multifilewarcwriter: ensure digest is computed before trying to lookup revisits
2017-05-01 21:50:39 -07:00
Ilya Kreymer
58f39f0558
setup: update to warcio==1.2
...
add ensure_http_headers=True when reading WARC records
tests: fix pytest warnings, use webtest.TestApp instead of TestApp
2017-04-29 13:47:54 -07:00
Ilya Kreymer
ac84dcc2e3
setup: cleanup deps: remove urllib3 (installed by requests), add werkzeug to core deps
2017-03-12 12:21:23 -07:00
Ilya Kreymer
7a8fed2681
update to wario1.1
...
archiveindexer: explicitly consume content for each record
2017-03-10 10:05:39 -08:00
Ilya Kreymer
af7bbfd6e1
build: update gevent, support py3.6
2017-03-09 11:59:54 -08:00
Ilya Kreymer
e86e3e6d32
build process: simplify build process by moving essential deps to requirements.txt, and extras to extra_requirements.txt
...
setup.py just loads from requirements.txt
Dockerfile pip installs requirements, then extra requirements for improved cacheing
travis runs setup install, then installs extra requirements
2017-03-08 17:05:29 -08:00
