Barbara Miller
ccb67e03b4
bump qa version
2023-06-19 14:30:33 -07:00
Barbara Miller
98f7032419
cryptography <40
2023-06-19 14:30:33 -07:00
Vangelis Banos
aa8948e612
Limit dependency version cryptography>=2.3,<=39.0.0
...
cryptography 41.0.0 crashes warcprox with the following exception:
```
File "/opt/spn2/lib/python3.8/site-packages/warcprox/main.py", line 317, in main
cryptography.hazmat.backends.openssl.backend.activate_builtin_random()
AttributeError: 'Backend' object has no attribute 'activate_builtin_random'
```
Also, cryptography==40.0.0 isn't OK because when I try to use it I get:
```
pyopenssl 23.2.0 requires cryptography!=40.0.0,!=40.0.1,<42,>=38.0.0, but you have cryptography 40.0.0 which is incompatible.
```
So, the version should be <=39.0.0
2023-06-19 14:30:33 -07:00
Barbara Miller
b45e49ccec
update readme — rm travis
2023-06-19 14:30:33 -07:00
Christian Clauss
243a225537
Delete .travis.yml
2023-06-19 14:30:33 -07:00
Barbara Miller
1863551cba
Merge pull request #185 from galgeek/qa
...
qa
2023-06-06 13:30:47 -07:00
Barbara Miller
2755a10ebc
fix logging
2023-06-06 12:31:46 -07:00
Barbara Miller
1ea069ae32
fix typos
2023-06-06 12:31:46 -07:00
Barbara Miller
2acaba19df
Merge pull request #183 from galgeek/qa
...
skip revisits
2023-06-05 13:51:43 -07:00
Barbara Miller
bc143a57fe
Merge branch 'qa' into qa
2023-06-05 13:51:19 -07:00
Barbara Miller
d98896094a
bump qa version
2023-06-05 13:47:14 -07:00
Barbara Miller
1dc7de7dd8
skip duplicate revisits, per ait-job-id
2023-06-05 13:40:21 -07:00
Barbara Miller
50a4f35e5f
Merge pull request #177 from internetarchive/blocks-shrink
...
@adam-miller ok'd this elsewhere
2022-08-05 15:44:05 -07:00
Barbara Miller
9973d28de9
bump version
2022-08-04 17:28:33 -07:00
Barbara Miller
58f3e58531
bump qa version
2022-08-04 15:13:43 -07:00
Barbara Miller
460773a4a6
Merge branch 'blocks-shrink' into qa
2022-08-04 15:12:58 -07:00
Barbara Miller
ee9e375560
zlib decompression
2022-08-04 11:14:33 -07:00
Barbara Miller
8f10fce93a
resetting to Jul 1 updates
2022-08-03 19:58:54 -07:00
Barbara Miller
20789e4edb
bump qa version
2022-08-03 16:53:52 -07:00
Barbara Miller
5ed59a82d2
Merge branch 'qa' of github.com:internetarchive/warcprox into qa
2022-08-03 16:53:17 -07:00
Barbara Miller
e7fb1474ad
Merge branch 'blocks-shrink' into qa
2022-08-03 16:52:44 -07:00
Barbara Miller
1205589fae
decompress and json.loads
2022-08-03 16:52:31 -07:00
Barbara Miller
053a42a371
bump qa version
2022-08-03 16:30:49 -07:00
Barbara Miller
babbb8ca74
Merge branch 'blocks-shrink' into qa
2022-08-03 15:50:53 -07:00
Barbara Miller
2cdfceade1
decompress and split
2022-08-03 15:49:54 -07:00
Barbara Miller
09347c903e
Merge branch 'master' of github.com:internetarchive/warcprox into blocks-shrink
2022-08-03 15:46:36 -07:00
Barbara Miller
a232ffc6ba
bump qa version
2022-08-03 15:04:39 -07:00
Barbara Miller
ea5ba9b007
Merge branch 'qa' of github.com:internetarchive/warcprox into qa
2022-08-03 15:01:13 -07:00
Barbara Miller
d617b4850c
Merge branch 'blocks-shrink' into qa
2022-08-03 15:01:01 -07:00
Barbara Miller
3e8102221d
use 'compressed_blocks'
2022-08-03 14:59:36 -07:00
Barbara Miller
106cb905db
zlib decompression
2022-08-03 11:15:25 -07:00
Barbara Miller
c008c2eca7
bump version
2022-07-01 14:18:17 -07:00
Barbara Miller
7958921053
Merge pull request #175 from vbanos/random-tls-fingerprint
...
Thanks, @vbanos!
2022-07-01 14:16:05 -07:00
Barbara Miller
d3fdcbe152
bump qa version
2022-07-01 11:28:28 -07:00
Barbara Miller
6ad9a3e448
Merge branch 'qa' of github.com:internetarchive/warcprox into qa
2022-07-01 11:13:24 -07:00
Barbara Miller
ab172189fd
Merge branch 'tls-fingerprint' into qa
2022-07-01 11:11:54 -07:00
Vangelis Banos
329fef31a8
Randomize TLS fingerprint
...
Create a random TLS fingerprint per HTTPS connection to avoid TLS
fingerprinting.
2022-07-01 17:39:49 +00:00
Barbara Miller
d253ea85c3
Merge pull request #173 from internetarchive/increase_batch_sec
...
tune MIN_BATCH_SEC, MAX_BATCH_SEC for fewer dedup errors
2022-06-24 11:13:18 -07:00
Barbara Miller
8418fe10ba
add explanatory comment
2022-06-24 11:07:35 -07:00
Adam Miller
fcd9b2b3bd
Merge pull request #172 from internetarchive/adds-canonicalization-tests
...
Adding url canonicalization tests and handling of edge cases to reduc…
2022-04-27 09:57:03 -07:00
Adam Miller
aa4a550b12
Merge branch 'adds-canonicalization-tests' into qa
2022-04-26 23:49:12 +00:00
Adam Miller
731cfe80cc
Adding url canonicalization tests and handling of edge cases to reduce log noise
2022-04-26 23:48:54 +00:00
Adam Miller
9521042a23
Merge pull request #171 from internetarchive/adds-hop-path-logging
...
Adds hop path logging
2022-04-26 12:11:11 -07:00
Adam Miller
daa925db17
Bump version
2022-04-26 09:55:48 -07:00
Adam Miller
31693c5472
Merge branch 'adds-hop-path-logging' into qa
2022-04-21 18:37:39 +00:00
Adam Miller
d96dd5d842
Adjust rfc3986 package version for deployment across more versions
2022-04-21 18:37:27 +00:00
Adam Miller
0f2c94ab9e
Merge branch 'adds-hop-path-logging' into qa
2022-04-20 22:50:08 +00:00
Adam Miller
1e3d22aba4
Better handle non-ascii urls for crawl log hop info
2022-04-20 22:48:28 +00:00
Adam Miller
caf7f3b30f
Merge branch 'qa' of github.com:internetarchive/warcprox into qa
2022-03-24 21:41:19 +00:00
Adam Miller
28bec1bb14
Merge branch 'adds-hop-path-logging' into qa
2022-03-24 21:41:11 +00:00
