51 Commits

Author SHA1 Message Date
Noah Levitt
8dfb63f70d readable stack traces, thanks py.test 2018-08-20 12:07:23 -07:00
Noah Levitt
46d5b0e82c run trough with python 3.6 plus travis cleanup
docker image python:3 is now using 3.7 and building pyyaml < 3.13 fails
yaml/pyyaml#126

also filed pull request to update trough's pyyaml dependency spec
internetarchive/trough#20
2018-07-18 16:09:42 -05:00
Noah Levitt
eacf070a2a dropping claim of support for python 2.7 (not worth hacking around tempfile.TemporaryDirectory to make tests pass) 2017-12-21 15:45:39 -08:00
Noah Levitt
ef590a2fec py2 fix 2017-11-13 15:07:47 -08:00
Noah Levitt
46797a5dce pypy and pypy3 are passing at the moment, so why not :) 2017-11-13 12:52:29 -08:00
Noah Levitt
b2adb778ee Merge branch 'master' into trough-dedup
* master:
  not gonna bother figuring out why pypy regex is not matching https://travis-ci.org/internetarchive/warcprox/jobs/299864258#L615
  fix failing test just committed, which involves running "listeners" for all urls, including those not archived; make adjustments accordingly
  make test_crawl_log expect HEAD request to be logged
  fix crawl log handling of WARCPROX_WRITE_RECORD request
  modify test_crawl_log to expect crawl log to honor --base32 setting and add tests of WARCPROX_WRITE_RECORD request and HEAD request (not written to warc)
  bump dev version number
  add --crawl-log-dir option to fix failing test
  create crawl log dir at startup if it doesn't exist
  make test pass with py27
  fix crawl log test to avoid any dedup collisions
  fix crawl log test
  heritrix-style crawl log support
  disallow slash and backslash in warc-prefix
  can't see any reason to split the main() like this (anymore?)
  add missing dependency warcio to tests_require
2017-11-09 15:50:18 -08:00
Noah Levitt
750a333aa6 not gonna bother figuring out why pypy regex is not matching https://travis-ci.org/internetarchive/warcprox/jobs/299864258#L615 2017-11-09 15:23:15 -08:00
Noah Levitt
dfecfc2e45 it finally works! another travis tweak though 2017-10-19 11:10:58 -07:00
Noah Levitt
0a16c0ad84 can we edit /etc/hosts in travis-ci? 2017-10-19 10:54:47 -07:00
Noah Levitt
7b1d2d8c5d ugh fix docker command line arg 2017-10-19 10:44:53 -07:00
Noah Levitt
81497088e4 docker container for trough needs a hostname that works from outside the container (since it registers itself in the service registry) 2017-10-19 10:20:51 -07:00
Noah Levitt
7b5fe4475e trough logs are inside the docker container now 2017-10-18 17:38:27 -07:00
Noah Levitt
158c451311 need docker to publish the rethinkdb port for --rethinkdb-dedup-url and --rethinkdb-big-table-url tests 2017-10-18 15:47:24 -07:00
Noah Levitt
1b172f37e9 apparently you can't use docker run options --rm and --detach together 2017-10-18 15:28:18 -07:00
Noah Levitt
a64a12289e in travis-ci, run trough in another docker container, so that its version of python can be independent of the one used to run the warcprox tests 2017-10-18 15:21:53 -07:00
Noah Levitt
d4b39f3fcc remove some debugging from .travis.yml and importantly, deactivate the trough virtualenv before installing warcprox and running tests (otherwise it uses the wrong version of python) 2017-10-18 09:45:06 -07:00
Noah Levitt
4c4f8ead09 missed an ampersand 2017-10-17 14:58:46 -07:00
Noah Levitt
73d4a19c0a bangin (is the problem that we didn't start trough-read? 2017-10-17 14:42:54 -07:00
Noah Levitt
994eda70a8 banging 2017-10-17 14:33:36 -07:00
Noah Levitt
ddc88cda0f more banging on travis-ci 2017-10-16 16:05:23 -07:00
Noah Levitt
0e78140d47 cryptography 2.1.1 seems to be the problem 2017-10-13 16:52:08 -07:00
Noah Levitt
166aaab3e5 banging on travis-ci 2017-10-13 16:40:08 -07:00
Noah Levitt
892960d41a first attempt to run trough on travis-ci 2017-10-13 16:26:33 -07:00
Noah Levitt
828a2c3dcf get all the tests to pass with ./tests/run-tests.sh 2017-10-13 15:54:05 -07:00
Noah Levitt
d177b3b80d change rethinkdb-related command line options to use "rethinkdb urls" (parser just added to doublethink) to reduce the proliferation of rethinkdb options, and add --rethinkdb-trough-db-url option 2017-10-11 12:06:19 -07:00
Noah Levitt
5a8d1610e6 try to work around stupid travis build error, see https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch 2017-06-23 14:12:04 -07:00
Noah Levitt
95dfa54968 get rid of dbm, switch to sqlite, for easier portability, clarity around threading 2017-05-24 13:57:09 -07:00
Noah Levitt
ca7625b18d set via header on request and response, record request via in warc (because it is sent to the remote site), do not record response via in warc (because it is not sent by the remote site) 2017-04-28 11:07:33 -07:00
Noah Levitt
35d7ccd12e add seconds_behind to service registry and status api, which is the length of time the next url to be written to warc has been waiting in the queue 2017-03-30 15:54:19 -07:00
Noah Levitt
fed8dfa978 fix buglet 2017-03-15 12:01:34 -07:00
Noah Levitt
907e519af0 python 3.6 is out now 2017-01-23 13:53:01 -08:00
Noah Levitt
af74959864 add slack notification 2017-01-16 12:26:45 -08:00
Noah Levitt
fa1e8d3af4 allow travis-ci failures for python-nightly and also test 3.6-dev (but allow failures);
enable the onion site tor test because apparently travis-ci is allowing me to
install tor now, see https://travis-ci.org/internetarchive/warcprox/jobs/169101744
although https://github.com/travis-ci/apt-package-whitelist/issues/1753 is still open
2016-10-19 18:24:25 -07:00
Noah Levitt
a5d6d634d8 enable pypy and pypy3 travis-ci tests, but allow failures 2016-07-11 11:23:53 -05:00
Noah Levitt
00dc9eed84 new option --onion-tor-socks-proxy, host:port of tor socks proxy, used only to connect to .onion sites 2016-01-26 18:47:08 -08:00
Noah Levitt
93a2e4ff85 .travis.yml - disable pypy (not working because of cryptography library), require docker service 2016-01-26 18:47:08 -08:00
Noah Levitt
9f84c20274 test with rethinkdb flags too 2016-01-26 18:47:08 -08:00
Noah Levitt
4dcaedb5d9 py.test the right thing 2016-01-26 18:47:08 -08:00
Noah Levitt
7e731d40bc try new travis docker-based infrastructure, more versions of python 2016-01-26 18:47:08 -08:00
Noah Levitt
0eb2917e50 update tox and travis config for supported python versions 2.7 and 3.4 2015-03-18 16:36:24 -07:00
Noah Levitt
111c678cee add python3.4 to travis, tox test list; remove apt-get install python3.3-gdbm from travis configuration to fix travis error "Unable to locate package python3.3-gdbm" 2014-08-01 16:43:00 -07:00
Noah Levitt
2ab44d4adf maybe travis build can use tox 2014-01-07 14:10:39 -08:00
Noah Levitt
62e58caea5 .travis.yml - use pip to install test dependencies 2014-01-07 14:00:46 -08:00
Noah Levitt
ad30585cbf fix travis build? 2014-01-07 12:04:52 -08:00
Noah Levitt
8213c47969 py.test 2014-01-07 11:54:20 -08:00
Noah Levitt
f91986c1af maybe gdbm will work 2013-12-09 18:35:40 -08:00
Noah Levitt
0d617a927c tox (and travis ci?) were hiding the fact that the gdbm dependency was the problem 2013-12-07 00:28:56 -08:00
Noah Levitt
e9e152ca7d tox (and travis ci?) were hiding the fact that the gdbm dependency was the problem 2013-12-07 00:27:59 -08:00
Noah Levitt
965043324a not sure how to test travis ci without making a million commits 2013-12-06 17:14:01 -08:00
Noah Levitt
b6774da603 more fiddling trying to get test runs to work with various invocation methods, esp travis 2013-12-06 16:50:02 -08:00