1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 00:03:28 +01:00

10 Commits

Author SHA1 Message Date
Ilya Kreymer
af76ce9fa5 appveyor: fix appveyor builds, add py38 2020-06-08 15:03:06 -07:00
Ilya Kreymer
f0b9d5b8e8
Rewriting fix for DASH FB and document.write (#529)
* rewrite fixes:
- dash rewrite fix for fb: when rewriting, match quoted '"dash_prefetched_representation_ids"' as well as w/o quotes,
update tests to ensure rewriting both old and new formats
- wombat update to fix #527: ensure document.write() doesn't accidentally remove end-tag if end-tag was not lowercase (see webrecorder/wombat#21)

* tests: fix recorder cookie filtering test, use https://www.google.com/ for testing

* appveyor: fix appveyor builds
2020-01-11 10:44:49 -08:00
Ilya Kreymer
259f571cb9
Python 3.7 Support (#447)
* py3.7 fixes:
- add __repr__ to WBException for consistent output in py3.7
- don't raise StopIteration in generator, just return

* ci: add py3.7 builds to travis and appveyor, (don't include in integration test suite for now)
2019-02-27 08:43:33 -08:00
Ilya Kreymer
804734525c appveyor fix: use 'python -m pip' to upgrade pip (pypa/pip#5240) 2018-04-20 08:51:48 -07:00
Ilya Kreymer
9da5bd1083
Decoding and Recorder Fixes (#313)
* redisindex: use decode_resposes=True for redisindex
* recorder: close_file(): return true if closed, close_key() return filename if closed
* logging: if debug=True, log warc load failures
* appveyor build fix: add pypiwin32 as dependency for windows build
2018-03-29 13:42:00 -07:00
Ilya Kreymer
34902df80c docs work and misc:
- set depth in main toc to 3
- add info on cli apps in apps.rst
- fix typos, update links
setup: add 'pywb' cli script to be same as 'wayback'
appveyor: remove coveralls
2018-01-29 18:36:14 -08:00
Ilya Kreymer
131c5ff5da
SOCKS proxy (#281)
warcserver: SOCKS proxy:
- add support for running warcserver through a socks proxy specified via SOCKS_HOST and SOCKS_PORT
- move socks patch setup, http max_header adjustment to http module
- logging: print stack trace only if debugging
- add pysocks to extra_requirements, enable in ci
- add simple test (not actual proxy) to check that connection through proxy is attempted
- docs: add SOCKS proxy section to docs
2018-01-17 10:51:49 -08:00
Ilya Kreymer
4f340933f3
Add CodeCov (#282)
coverage: switch coverage reporting to codecov, enable in travis-ci and appveyor
coverage: update .coveragerc to include branch, exclude NotImplementedError, __repr__
README: badge update, add appveyor, codecov badges
2018-01-17 09:59:51 -08:00
Ilya Kreymer
54b265aaa8 s3 and zipnum fixes: (#253)
* s3 and zipnum fixes:
- update s3 to use boto3
- ensure zipnum indexes (.idx, .summary) are picked up automatically via DirectoryAggregator
- ensure showNumPages query always return a json object, ignoring output=
- add tests for auto-configured zipnum indexes

* reqs: add boto3 dependency, init boto Config only if avail

* s3 loader: first try with credentials, then with no-cred config
archive paths: don't add anything if path is fully qualified (contains '://')

* s3 loader: on first load, if credentialed load fails, try uncredentialed
fix typo
tests: add zinum auto collection tests

* zipnum page count query: don't add 'source' field to page count query (if 'url' key not present in dict)

* s3 loader: fix no-range load, add test, update skip check to boto3

* fix spacing

* boto -> boto3 rename error message, cleanup comments
2017-10-11 15:33:57 -07:00
Ilya Kreymer
bcb5bef39d Windows Build Fixes/Appveyor CI (#225)
windows build fixes: all tests should pass, ci with appveyor
- add appveyor.yml
- path fixes for windows, use os.path.join
- templates_dir: use '/' always for jinja2 paths
- auto colls: ensure chdir before deleting dir
- recorder: ensure warc writer is always closed
- recorder: disable locking in warcwriter on windows for now (read access not avail, shared
lock seems to not be working)
- zipnum: ensure block is closed after read!
- cached dir test: wait before adding file
- tests: adjust timeout tests to allow more leeway in timing
2017-08-05 17:12:16 -07:00