1068 Commits

Author SHA1 Message Date
Adam Miller
31693c5472 Merge branch 'adds-hop-path-logging' into qa 2022-04-21 18:37:39 +00:00
Adam Miller
d96dd5d842 Adjust rfc3986 package version for deployment across more versions 2022-04-21 18:37:27 +00:00
Adam Miller
0f2c94ab9e Merge branch 'adds-hop-path-logging' into qa 2022-04-20 22:50:08 +00:00
Adam Miller
1e3d22aba4 Better handle non-ascii urls for crawl log hop info 2022-04-20 22:48:28 +00:00
Adam Miller
caf7f3b30f Merge branch 'qa' of github.com:internetarchive/warcprox into qa 2022-03-24 21:41:19 +00:00
Adam Miller
28bec1bb14 Merge branch 'adds-hop-path-logging' into qa 2022-03-24 21:41:11 +00:00
Adam Miller
5ae1291e37 Refactor of hop path referer logic 2022-03-24 21:40:55 +00:00
Barbara Miller
a614df69fd Merge branch 'qa' of github.com:internetarchive/warcprox into qa 2022-03-03 18:49:24 -08:00
Barbara Miller
e48a8dda05 Merge branch 'increase_batch_sec' into qa 2022-03-03 18:47:04 -08:00
Barbara Miller
05daafa19e increase MIN_BATCH_SEC, MAX_BATCH_SEC 2022-03-03 18:46:20 -08:00
Adam Miller
c8563b9407 Merge branch 'adds-hop-path-logging' into qa 2022-03-04 02:02:18 +00:00
Adam Miller
ade2373711 Fixing referer on request with null hop path 2022-03-04 02:01:55 +00:00
Adam Miller
60bd2ea2bd Merge branch 'adds-hop-path-logging' into qa 2022-03-03 00:19:00 +00:00
Adam Miller
3a234d0cec Refactor hop_path metadata 2022-03-03 00:18:16 +00:00
Adam Miller
dea2d1c8fa
Merge pull request #168 from internetarchive/adds-hop-path-logging
Adds hop path logging
2022-02-09 10:55:12 -08:00
Adam Miller
366ed5155f Merge branch 'master' into adds-hop-path-logging 2022-02-09 18:18:32 +00:00
Barbara Miller
c027659001
Merge pull request #167 from galgeek/WT-31
fix logging buglet iii
2021-12-29 12:14:56 -08:00
Barbara Miller
6ccd72b8e3 Merge branch 'WT-31' into qa 2021-12-29 12:06:40 -08:00
Barbara Miller
9e8ea5bb45 fix logging buglet iii 2021-12-29 12:06:18 -08:00
Barbara Miller
5fd22a0809 Merge branch 'WT-31' into qa 2021-12-29 11:57:40 -08:00
Barbara Miller
a66a5157c7 bump qa version too 2021-12-29 11:57:35 -08:00
Barbara Miller
bc3d1e6d00 fix logging buglet ii 2021-12-29 11:55:39 -08:00
Barbara Miller
6b372e2f3f
Merge pull request #166 from galgeek/WT-31
fix logging buglet
2021-12-29 11:04:03 -08:00
Barbara Miller
cff2b19745 Merge branch 'WT-31' into qa 2021-12-29 10:25:30 -08:00
Barbara Miller
5d8fbf7038 fix logging buglet 2021-12-29 10:25:04 -08:00
Barbara Miller
a969430b37
Merge pull request #163 from internetarchive/idna2_10
idna==2.10
2021-12-28 13:50:23 -08:00
Barbara Miller
aeecb6515f
bump version 2021-12-28 11:58:30 -08:00
Adam Miller
e1eddb8fa7
Merge pull request #165 from galgeek/WT-31
in-batch dedup
2021-12-28 11:52:41 -08:00
Barbara Miller
48f48c34cd Merge branch 'WT-31' into qa 2021-12-16 18:45:00 -08:00
Barbara Miller
d7aec77597 faster, likely 2021-12-16 18:36:00 -08:00
Barbara Miller
6e65b5ff55 Merge branch 'WT-31' into qa 2021-12-09 12:20:09 -08:00
Barbara Miller
bcaf293081 better logging 2021-12-09 12:19:45 -08:00
Barbara Miller
1d3e3b3671 Merge branch 'WT-31' into qa 2021-12-08 11:04:27 -08:00
Barbara Miller
7d4c8dcb4e recorded_url.do_not_archive = True 2021-12-08 11:04:09 -08:00
Barbara Miller
69529e5845 Merge branch 'WT-31' into qa 2021-12-06 20:33:37 -08:00
Barbara Miller
da089e0a92 bytes not str 2021-12-06 20:33:16 -08:00
Barbara Miller
2ceb0f69f1 Merge branch 'WT-31' into qa 2021-12-06 19:43:37 -08:00
Barbara Miller
3eeccd0016 more hash_plus_url 2021-12-06 19:43:27 -08:00
Barbara Miller
a8944ddea3 Merge branch 'WT-31' into qa 2021-12-06 19:33:32 -08:00
Barbara Miller
5e5a74f204 str, not object 2021-12-06 19:33:10 -08:00
Barbara Miller
533234162e str, not object 2021-12-06 19:32:35 -08:00
Barbara Miller
85bb6ff437 Merge branch 'WT-31' into qa 2021-12-06 17:30:25 -08:00
Barbara Miller
b67f1ad0f3 add logging 2021-12-06 17:29:27 -08:00
Barbara Miller
e6a1a7dd7e increase trough dedup batch window 2021-12-06 17:29:02 -08:00
Barbara Miller
4e7a4c3eae Merge branch 'WT-31' into qa 2021-12-02 11:46:58 -08:00
Barbara Miller
e744075913 python 3.5 version, mostly 2021-12-02 11:46:39 -08:00
Barbara Miller
16412d64dc Merge branch 'WT-31' into qa 2021-12-02 11:18:44 -08:00
Barbara Miller
1476bfec8c discard batch hash+url match 2021-12-02 11:17:59 -08:00
Adam Miller
b57ec9c589 Check warcprox meta headers for hop information necessary to record a hop path if provided 2021-08-31 17:09:06 +00:00
Barbara Miller
bab938f080 Merge branch 'idna2_10' into qa 2021-04-27 10:28:25 -07:00