backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Merge pull request #129 from vbanos/urllib-cache-size

Browse Source 
Increase urllib parse cache size
This commit is contained in:
Noah Levitt 2019-05-06 15:55:47 -07:00 committed by GitHub
commit 1a8c719422
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
warcprox/mitmproxy.py
View File

@ -35,6 +35,13 @@ try:
import urllib.parse as urllib_parse
except ImportError:
import urlparse as urllib_parse
# In python2/3, urllib parse caches in memory URL parsing results to avoid
# repeating the process for the same URL. The problem is that the default
# in memory cache size is just 20.
# https://github.com/python/cpython/blob/3.7/Lib/urllib/parse.py#L80
# since we do a lot of URL parsing, it makes sense to increase cache size.
urllib_parse.MAX_CACHE_SIZE = 2000
try:
import http.client as http_client
# In python3 http.client.parse_headers() enforces http_client._MAXLINE