backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity
1,183 Commits 22 Branches 10 Tags
Commit Graph

1183 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Barbara Miller
b7d0c11547 Merge branch 'spn_subdir' into qa 2024-06-03 14:45:25 -07:00
Vangelis Banos
56e0b17dc9 New option --subdir-prefix 
Save WARCs in subdirectories equal to the current value of Warcprox-Meta['warc-prefix'].
E.g. if warc-prefix=='spn2' and --dir=/warcs, save them in /warcs/spn2/.
 2024-06-03 21:21:19 +00:00
Barbara Miller
af52dec469
bump version 2023-10-17 09:19:56 -07:00
Barbara Miller
848c089afa
Merge pull request #194 from vbanos/socksproxy 
Thank you, @vbanos!
 2023-10-17 09:18:11 -07:00
Vangelis Banos
9fd5a22502 fix typo 2023-10-17 06:12:28 +00:00
Barbara Miller
65b92c1604 fix typo, update copyright 2023-10-16 14:48:50 -07:00
Barbara Miller
7f03403531 Merge branch 'socksproxy' into qa 2023-10-16 14:04:17 -07:00
Vangelis Banos
3d653e023c Add SOCKS proxy options 
Add options `--socks-proxy`, `--socks-proxy-username,
`--socks-proxy-password`.

If enabled, all traffic is routed throught the SOCKS proxy.
 2023-10-16 18:33:42 +00:00
Barbara Miller
4cb8e0d5dc
Merge pull request #192 from internetarchive/Py311 
updates for 3.11 (and back to 3.8)
@vbanos and @avdempsey have agreed this PR is ok to merge
 2023-09-27 12:03:26 -07:00
Barbara Miller
a20ad226cb
update version to 2.5, for Python version updates 2023-09-27 11:58:39 -07:00
Barbara Miller
bc0da12c48
bump version for Py311 2023-09-20 10:57:54 -07:00
Barbara Miller
779049ea70 Merge branch 'Py311' into qa 2023-09-19 13:58:22 -07:00
Barbara Miller
8f0039de02 internetarchive/doublethink.git@Py311 2023-09-19 13:57:34 -07:00
Barbara Miller
c620d7dd19 use galgeek for now 2023-09-13 18:03:38 -07:00
Barbara Miller
4fbf523a3e get doublethink from github.com/internetarchive 2023-09-12 16:05:23 -07:00
Barbara Miller
68dd6dbb78 Merge branch 'Py311' into qa 2023-09-12 14:39:28 -07:00
Barbara Miller
3b5d9d8ef0 update rethinkdb import 2023-09-12 14:39:09 -07:00
Barbara Miller
c07efc642d Merge branch 'py311' into qa 2023-09-11 17:38:26 -07:00
Barbara Miller
5e779af2e9 trough and doublethink updates 2023-09-11 17:38:10 -07:00
Barbara Miller
0611c17fb3 Merge branch 'py311' into qa 2023-09-11 17:01:10 -07:00
Barbara Miller
a90c9c3dd4 trough 0.20 maybe 2023-09-11 17:01:02 -07:00
Barbara Miller
0bf3c24e3c Merge branch 'py311' into qa 2023-09-11 16:40:52 -07:00
Barbara Miller
99a825c055 initial commit, trying trough branch jammy+focal 2023-09-11 16:40:39 -07:00
Barbara Miller
e0b92bc901 Merge branch 'limit_revisits' into qa 2023-09-11 16:03:16 -07:00
Barbara Miller
f82eb1f6d5 minor edits post-deploy 2023-09-11 16:02:52 -07:00
Barbara Miller
93667c7f7b Merge branch 'limit_revisits' into qa 2023-08-29 13:29:24 -07:00
Barbara Miller
976ff1b20d wrapper cache_true 2023-08-20 13:11:13 -04:00
Barbara Miller
f0b69dd74e Merge branch 'limit_revisits' into qa 2023-08-15 16:12:07 -07:00
Barbara Miller
15271835f6 format in limit_revisits 2023-08-15 16:11:45 -07:00
Barbara Miller
da69503ed1 Merge branch 'limit_revisits' into qa 2023-08-15 15:50:29 -07:00
Barbara Miller
887680b0ec try iso-8859-1 2023-08-15 15:50:02 -07:00
Barbara Miller
4e88c90f4d Merge branch 'limit_revisits' into qa 2023-08-15 14:27:01 -07:00
Barbara Miller
533f5c0af2 limit_revisits wants str, not bytes 2023-08-15 14:26:14 -07:00
Barbara Miller
3023484cfc Merge branch 'limit_revisits' into qa 2023-08-03 14:10:29 -07:00
Barbara Miller
a86169c56c pep0440 version id (I think) 2023-08-03 14:09:23 -07:00
Barbara Miller
3c64ee1529 Merge branch 'limit_revisits' into qa 2023-07-21 13:37:38 -07:00
Barbara Miller
f83e82c900 limit_revisit check before dedup 2023-07-21 13:37:08 -07:00
Barbara Miller
0dc80c6044 Merge branch 'limit_revisits' into qa 2023-07-13 10:34:02 -07:00
Barbara Miller
e5b2561821 disable prepared statements: prepare_threshold=None 2023-07-13 10:19:01 -07:00
Barbara Miller
8d684b7e12 Merge branch 'limit_revisits' into qa 2023-07-12 17:57:43 -07:00
Barbara Miller
548c4e5cab initial deploy fixes 2023-07-12 17:56:39 -07:00
Barbara Miller
af4c8b071a lru_cache skip_revisit 2023-07-12 17:05:29 -07:00
Barbara Miller
47811977ef lru_cache skip_revisit 2023-07-12 17:04:07 -07:00
Barbara Miller
3de580e352 Merge branch 'limit_revisits' into qa 2023-07-11 16:37:09 -07:00
Barbara Miller
64a152ee8c lru_cache 2023-07-11 16:35:38 -07:00
Barbara Miller
c01d58df78
Merge pull request #189 from vbanos/idna-update 
Thank you, @vbanos!
 2023-07-11 14:13:47 -07:00
Barbara Miller
b5ca3b6db6 Merge branch 'idna_unpeg' into qa 2023-07-11 13:28:43 -07:00
Vangelis Banos
6eb2bd1265 Drop idna==2.10 version lock 
There is no need to use such an old `idna` version.
The latest works with py35+ and all tests pass.
Newer `idna` supports the latest Unicode standard and latest python
versions.
https://github.com/kjd/idna/blob/master/HISTORY.rst
 2023-07-09 10:02:13 +00:00
Barbara Miller
8563b95ff6 Merge branch 'limit_revisits' into qa 2023-06-28 17:35:58 -07:00
Barbara Miller
b91a7d1d89 more updates qa prototyping 2023-06-28 17:34:26 -07:00