8 Commits

Author	SHA1	Message	Date
Noah Levitt	6d673ee35f	tests pass with big rethinkdb captures table	2016-01-26 18:47:08 -08:00
Noah Levitt	c430f81883	some refactoring to prep for big rethinkdb capture table	2016-01-26 18:47:08 -08:00
Noah Levitt	e66dc3a9fb	rethinkdb dedup	2016-01-26 18:46:13 -08:00
Noah Levitt	a876152026	fix exception, make some tweaks	2016-01-26 18:46:13 -08:00
Noah Levitt	274a2f6b1d	refactor warc writing, deduplication for somewhat cleaner separation of concerns	2016-01-26 18:45:36 -08:00
Noah Levitt	5f84b061f3	make it work with python 2.7 again	2015-03-18 16:29:44 -07:00
Noah Levitt	9b8ffbbb51	separate WarcWriter and WarcWriterThread	2014-11-15 04:47:26 -08:00
Noah Levitt	b34edf8fb1	split into multiple files	2014-11-15 03:20:05 -08:00