Noah Levitt
8dfb63f70d
readable stack traces, thanks py.test
2018-08-20 12:07:23 -07:00
Noah Levitt
46d5b0e82c
run trough with python 3.6 plus travis cleanup
...
docker image python:3 is now using 3.7 and building pyyaml < 3.13 fails
yaml/pyyaml#126
also filed pull request to update trough's pyyaml dependency spec
internetarchive/trough#20
2018-07-18 16:09:42 -05:00
Noah Levitt
eacf070a2a
dropping claim of support for python 2.7 (not worth hacking around tempfile.TemporaryDirectory to make tests pass)
2017-12-21 15:45:39 -08:00
Noah Levitt
ef590a2fec
py2 fix
2017-11-13 15:07:47 -08:00
Noah Levitt
46797a5dce
pypy and pypy3 are passing at the moment, so why not :)
2017-11-13 12:52:29 -08:00
Noah Levitt
b2adb778ee
Merge branch 'master' into trough-dedup
...
* master:
not gonna bother figuring out why pypy regex is not matching https://travis-ci.org/internetarchive/warcprox/jobs/299864258#L615
fix failing test just committed, which involves running "listeners" for all urls, including those not archived; make adjustments accordingly
make test_crawl_log expect HEAD request to be logged
fix crawl log handling of WARCPROX_WRITE_RECORD request
modify test_crawl_log to expect crawl log to honor --base32 setting and add tests of WARCPROX_WRITE_RECORD request and HEAD request (not written to warc)
bump dev version number
add --crawl-log-dir option to fix failing test
create crawl log dir at startup if it doesn't exist
make test pass with py27
fix crawl log test to avoid any dedup collisions
fix crawl log test
heritrix-style crawl log support
disallow slash and backslash in warc-prefix
can't see any reason to split the main() like this (anymore?)
add missing dependency warcio to tests_require
2017-11-09 15:50:18 -08:00
Noah Levitt
750a333aa6
not gonna bother figuring out why pypy regex is not matching https://travis-ci.org/internetarchive/warcprox/jobs/299864258#L615
2017-11-09 15:23:15 -08:00
Noah Levitt
dfecfc2e45
it finally works! another travis tweak though
2017-10-19 11:10:58 -07:00
Noah Levitt
0a16c0ad84
can we edit /etc/hosts in travis-ci?
2017-10-19 10:54:47 -07:00
Noah Levitt
7b1d2d8c5d
ugh fix docker command line arg
2017-10-19 10:44:53 -07:00
Noah Levitt
81497088e4
docker container for trough needs a hostname that works from outside the container (since it registers itself in the service registry)
2017-10-19 10:20:51 -07:00
Noah Levitt
7b5fe4475e
trough logs are inside the docker container now
2017-10-18 17:38:27 -07:00
Noah Levitt
158c451311
need docker to publish the rethinkdb port for --rethinkdb-dedup-url and --rethinkdb-big-table-url tests
2017-10-18 15:47:24 -07:00
Noah Levitt
1b172f37e9
apparently you can't use docker run options --rm and --detach together
2017-10-18 15:28:18 -07:00
Noah Levitt
a64a12289e
in travis-ci, run trough in another docker container, so that its version of python can be independent of the one used to run the warcprox tests
2017-10-18 15:21:53 -07:00
Noah Levitt
d4b39f3fcc
remove some debugging from .travis.yml and importantly, deactivate the trough virtualenv before installing warcprox and running tests (otherwise it uses the wrong version of python)
2017-10-18 09:45:06 -07:00
Noah Levitt
4c4f8ead09
missed an ampersand
2017-10-17 14:58:46 -07:00
Noah Levitt
73d4a19c0a
bangin (is the problem that we didn't start trough-read?
2017-10-17 14:42:54 -07:00
Noah Levitt
994eda70a8
banging
2017-10-17 14:33:36 -07:00
Noah Levitt
ddc88cda0f
more banging on travis-ci
2017-10-16 16:05:23 -07:00
Noah Levitt
0e78140d47
cryptography 2.1.1 seems to be the problem
2017-10-13 16:52:08 -07:00
Noah Levitt
166aaab3e5
banging on travis-ci
2017-10-13 16:40:08 -07:00
Noah Levitt
892960d41a
first attempt to run trough on travis-ci
2017-10-13 16:26:33 -07:00
Noah Levitt
828a2c3dcf
get all the tests to pass with ./tests/run-tests.sh
2017-10-13 15:54:05 -07:00
Noah Levitt
d177b3b80d
change rethinkdb-related command line options to use "rethinkdb urls" (parser just added to doublethink) to reduce the proliferation of rethinkdb options, and add --rethinkdb-trough-db-url option
2017-10-11 12:06:19 -07:00
Noah Levitt
5a8d1610e6
try to work around stupid travis build error, see https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch
2017-06-23 14:12:04 -07:00
Noah Levitt
95dfa54968
get rid of dbm, switch to sqlite, for easier portability, clarity around threading
2017-05-24 13:57:09 -07:00
Noah Levitt
ca7625b18d
set via header on request and response, record request via in warc (because it is sent to the remote site), do not record response via in warc (because it is not sent by the remote site)
2017-04-28 11:07:33 -07:00
Noah Levitt
35d7ccd12e
add seconds_behind to service registry and status api, which is the length of time the next url to be written to warc has been waiting in the queue
2017-03-30 15:54:19 -07:00
Noah Levitt
fed8dfa978
fix buglet
2017-03-15 12:01:34 -07:00
Noah Levitt
907e519af0
python 3.6 is out now
2017-01-23 13:53:01 -08:00
Noah Levitt
af74959864
add slack notification
2017-01-16 12:26:45 -08:00
Noah Levitt
fa1e8d3af4
allow travis-ci failures for python-nightly and also test 3.6-dev (but allow failures);
...
enable the onion site tor test because apparently travis-ci is allowing me to
install tor now, see https://travis-ci.org/internetarchive/warcprox/jobs/169101744
although https://github.com/travis-ci/apt-package-whitelist/issues/1753 is still open
2016-10-19 18:24:25 -07:00
Noah Levitt
a5d6d634d8
enable pypy and pypy3 travis-ci tests, but allow failures
2016-07-11 11:23:53 -05:00
Noah Levitt
00dc9eed84
new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites
2016-01-26 18:47:08 -08:00
Noah Levitt
93a2e4ff85
.travis.yml - disable pypy (not working because of cryptography library), require docker service
2016-01-26 18:47:08 -08:00
Noah Levitt
9f84c20274
test with rethinkdb flags too
2016-01-26 18:47:08 -08:00
Noah Levitt
4dcaedb5d9
py.test the right thing
2016-01-26 18:47:08 -08:00
Noah Levitt
7e731d40bc
try new travis docker-based infrastructure, more versions of python
2016-01-26 18:47:08 -08:00
Noah Levitt
0eb2917e50
update tox and travis config for supported python versions 2.7 and 3.4
2015-03-18 16:36:24 -07:00
Noah Levitt
111c678cee
add python3.4 to travis, tox test list; remove apt-get install python3.3-gdbm from travis configuration to fix travis error "Unable to locate package python3.3-gdbm"
2014-08-01 16:43:00 -07:00
Noah Levitt
2ab44d4adf
maybe travis build can use tox
2014-01-07 14:10:39 -08:00
Noah Levitt
62e58caea5
.travis.yml - use pip to install test dependencies
2014-01-07 14:00:46 -08:00
Noah Levitt
ad30585cbf
fix travis build?
2014-01-07 12:04:52 -08:00
Noah Levitt
8213c47969
py.test
2014-01-07 11:54:20 -08:00
Noah Levitt
f91986c1af
maybe gdbm will work
2013-12-09 18:35:40 -08:00
Noah Levitt
0d617a927c
tox (and travis ci?) were hiding the fact that the gdbm dependency was the problem
2013-12-07 00:28:56 -08:00
Noah Levitt
e9e152ca7d
tox (and travis ci?) were hiding the fact that the gdbm dependency was the problem
2013-12-07 00:27:59 -08:00
Noah Levitt
965043324a
not sure how to test travis ci without making a million commits
2013-12-06 17:14:01 -08:00
Noah Levitt
b6774da603
more fiddling trying to get test runs to work with various invocation methods, esp travis
2013-12-06 16:50:02 -08:00
