Vangelis Banos 56e0b17dc9 New option --subdir-prefix ...

Save WARCs in subdirectories equal to the current value of Warcprox-Meta['warc-prefix'].
E.g. if warc-prefix=='spn2' and --dir=/warcs, save them in /warcs/spn2/.

2024-06-03 21:21:19 +00:00

..

__init__.py

add explanatory comment

2022-06-24 11:07:35 -07:00

bigtable.py

update rethinkdb import

2023-09-12 14:39:09 -07:00

controller.py

fix logging config which trough interfered with

2020-01-07 15:19:03 -08:00

crawl_log.py

Adding url canonicalization tests and handling of edge cases to reduce log noise

2022-04-26 23:48:54 +00:00

dedup.py

fix logging buglet iii

2021-12-29 12:06:18 -08:00

main.py

New option --subdir-prefix

2024-06-03 21:21:19 +00:00

mitmproxy.py

Add SOCKS proxy options

2023-10-16 18:33:42 +00:00

playback.py

Add bad_hostnames_ports in PlaybackProxy

2019-05-09 15:57:01 +00:00

stats.py

update rethinkdb import

2023-09-12 14:39:09 -07:00

warc.py

do not write incorrect warc-payload-digest to...

2019-05-02 14:25:29 -07:00

warcproxy.py

zlib decompression

2022-08-04 11:14:33 -07:00

writer.py

New option --subdir-prefix

2024-06-03 21:21:19 +00:00

writerthread.py

Moving more variables from RecordedUrl to RequiredUrl

2020-01-04 01:41:28 +00:00