backup/warcprox
1
0
Fork 0
mirror of https://github.com/internetarchive/warcprox.git synced 2025-01-18 13:22:09 +01:00
Code Issues Projects Releases Activity

Merge branch 'master' into qa

Browse Source 
* master:
  bump version after merge
  include warcprox host and port in filenames
  replace pencil drawing with nice diagram by James
  fix bug
  readable stack traces, thanks py.test
  --quiet means NOTICE level logging
  tweak max threads option handling
  set socket timeout for tor .onion fetching
  WARCPROX_WRITE_RECORD respect buffer size setting
  --help-hidden for help on hidden args
  half-baked readme section on warcprox architecture
  bump dev version number after merge
  restore 80 column lines
  Copy edits updated
  Copy edits
  update cryptography dep version
  use SpooledTemporaryFile for WARCPROX_WRITE_RECORD
  Apply blackout on when dedup URL equals request URL
  New --blackout-period option to skip writing redundant revisits to WARC
This commit is contained in:
Noah Levitt 2018-09-28 11:12:18 -07:00
commit eab9181129
16 changed files with 789 additions and 185 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.travis.yml
View File

@ -50,10 +50,10 @@ before_script:
- docker ps - docker ps
script: script:
- py.test -v tests - py.test -v --tb=native tests
- py.test -v --rethinkdb-dedup-url=rethinkdb://localhost/test1/dedup tests - py.test -v --tb=native --rethinkdb-dedup-url=rethinkdb://localhost/test1/dedup tests
- py.test -v --rethinkdb-big-table-url=rethinkdb://localhost/test2/captures tests - py.test -v --tb=native --rethinkdb-big-table-url=rethinkdb://localhost/test2/captures tests
- py.test -v --rethinkdb-trough-db-url=rethinkdb://localhost/trough_configuration tests - py.test -v --tb=native --rethinkdb-trough-db-url=rethinkdb://localhost/trough_configuration tests
after_script: after_script:
- ps ww -fHe - ps ww -fHe

163
README.rst
View File

@ -3,22 +3,19 @@ Warcprox - WARC writing MITM HTTP/S proxy
.. image:: https://travis-ci.org/internetarchive/warcprox.svg?branch=master .. image:: https://travis-ci.org/internetarchive/warcprox.svg?branch=master
:target: https://travis-ci.org/internetarchive/warcprox :target: https://travis-ci.org/internetarchive/warcprox
Warcprox is a tool for archiving the web. It is an http proxy that stores its Warcprox is an HTTP proxy designed for web archiving applications. When used in
traffic to disk in `WARC parallel with `brozzler <https://github.com/internetarchive/brozzler>`_ it
<https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/>`_ supports a comprehensive, modern, and distributed archival web capture system.
format. Warcprox captures encrypted https traffic by using the Warcprox stores its traffic to disk in the `Web ARChive (WARC) file format
`"man-in-the-middle" <https://en.wikipedia.org/wiki/Man-in-the-middle_attack>`_ <https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/>`_,
technique (see the `Man-in-the-middle`_ section for more info). which may then be accessed with web archival replay software like `OpenWayback
<https://github.com/iipc/openwayback>`_ and `pywb
<https://github.com/webrecorder/pywb>`_. It captures encrypted HTTPS traffic by
using the "man-in-the-middle" technique (see the `Man-in-the-middle`_ section
for more info).
The web pages that warcprox stores in WARC files can be played back using Warcprox was originally based on `pymiproxy
software like `OpenWayback <https://github.com/iipc/openwayback>`_ or `pywb <https://github.com/allfro/pymiproxy>`_ by Nadeem Douba.
<https://github.com/webrecorder/pywb>`_. Warcprox has been developed in
parallel with `brozzler <https://github.com/internetarchive/brozzler>`_ and
together they make a comprehensive modern distributed archival web crawling
system.
Warcprox was originally based on the excellent and simple pymiproxy by Nadeem
Douba. https://github.com/allfro/pymiproxy
.. contents:: .. contents::
@ -43,68 +40,72 @@ Try ``warcprox --help`` for documentation on command line options.
Man-in-the-middle Man-in-the-middle
================= =================
Normally, http proxies can't read https traffic, because it's encrypted. The Normally, HTTP proxies can't read encrypted HTTPS traffic. The browser uses the
browser uses the http ``CONNECT`` method to establish a tunnel through the HTTP ``CONNECT`` method to establish a tunnel through the proxy, and the proxy
proxy, and the proxy merely routes raw bytes between the client and server. merely routes raw bytes between the client and server. Since the bytes are
Since the bytes are encrypted, the proxy can't make sense of the information encrypted, the proxy can't make sense of the information that it proxies. This
it's proxying. This nonsensical encrypted data would not be very useful to nonsensical encrypted data is not typically useful for web archiving purposes.
archive.
In order to capture https traffic, warcprox acts as a "man-in-the-middle" In order to capture HTTPS traffic, warcprox acts as a "man-in-the-middle"
(MITM). When it receives a ``CONNECT`` directive from a client, it generates a (MITM). When it receives a ``CONNECT`` directive from a client, it generates a
public key certificate for the requested site, presents to the client, and public key certificate for the requested site, presents to the client, and
proceeds to establish an encrypted connection with the client. Then it makes a proceeds to establish an encrypted connection with the client. It then makes a
separate, normal https connection to the remote site. It decrypts, archives, separate, normal HTTPS connection to the remote site. It decrypts, archives,
and re-encrypts traffic in both directions. and re-encrypts traffic in both directions.
Although "man-in-the-middle" is often paired with "attack", there is nothing Configuring a warcprox instance as a browser’s HTTP proxy will result in
malicious about what warcprox is doing. If you configure an instance of security certificate warnings because none of the certificates will be signed
warcprox as your browser's http proxy, you will see lots of certificate by trusted authorities. However, there is nothing malicious about warcprox
warnings, since none of the certificates will be signed by trusted authorities. functions. To use warcprox effectively, the client needs to disable certificate
To use warcprox effectively the client needs to disable certificate verification or add the CA certificate generated by warcprox as a trusted
verification, or add the CA cert generated by warcprox as a trusted authority. authority. When using the latter, remember to undo this change when finished
(If you do this in your browser, make sure you undo it when you're done using using warcprox.
warcprox!)
API API
=== ===
For interacting with a running instance of warcprox. The warcprox API may be used to retrieve information from and interact with a
running warcprox instance, including:
* ``/status`` url * Retrieving status information via ``/status`` URL
* ``WARCPROX_WRITE_RECORD`` http method * Writing WARC records via ``WARCPROX_WRITE_RECORD`` HTTP method
* ``Warcprox-Meta`` http request header and response header * Controlling warcprox settings via the ``Warcprox-Meta`` HTTP header
See `<api.rst>`_. For warcprox API documentation, see: `<api.rst>`_.
Deduplication Deduplication
============= =============
Warcprox avoids archiving redundant content by "deduplicating" it. The process Warcprox avoids archiving redundant content by "deduplicating" it. The process
for deduplication works similarly to heritrix and other web archiving tools. for deduplication works similarly to deduplication by `Heritrix
<https://github.com/internetarchive/heritrix3>`_ and other web archiving tools:
1. while fetching url, calculate payload content digest (typically sha1) 1. While fetching URL, calculate payload content digest (typically SHA1
2. look up digest in deduplication database (warcprox supports a few different checksum value)
ones) 2. Look up digest in deduplication database (warcprox currently supports
3. if found, write warc ``revisit`` record referencing the url and capture time `sqlite <https://sqlite.org/>`_ by default, `rethinkdb
<https://github.com/rethinkdb/rethinkdb>`_ with two different schemas, and
`trough <https://github.com/internetarchive/trough>`_)
3. If found, write warc ``revisit`` record referencing the url and capture time
of the previous capture of the previous capture
4. else (if not found), 4. If not found,
a. write warc ``response`` record with full payload a. Write ``response`` record with full payload
b. store entry in deduplication database b. Store new entry in deduplication database
The dedup database is partitioned into different "buckets". Urls are The deduplication database is partitioned into different "buckets". URLs are
deduplicated only against other captures in the same bucket. If specified, the deduplicated only against other captures in the same bucket. If specified, the
``dedup-bucket`` field of the ``Warcprox-Meta`` http request header determines ``dedup-bucket`` field of the `Warcprox-Meta HTTP request header
the bucket, otherwise the default bucket is used. <api.rst#warcprox-meta-http-request-header>`_ determines the bucket. Otherwise,
the default bucket is used.
Deduplication can be disabled entirely by starting warcprox with the argument Deduplication can be disabled entirely by starting warcprox with the argument
``--dedup-db-file=/dev/null``. ``--dedup-db-file=/dev/null``.
Statistics Statistics
========== ==========
Warcprox keeps some crawl statistics and stores them in sqlite or rethinkdb. Warcprox stores some crawl statistics to sqlite or rethinkdb. These are
These are consulted for enforcing ``limits`` and ``soft-limits`` (see consulted for enforcing ``limits`` and ``soft-limits`` (see `Warcprox-Meta
`<api.rst#warcprox-meta-fields>`_), and can also be consulted by other fields <api.rst#warcprox-meta-fields>`_), and can also be consulted by other
processes outside of warcprox, for reporting etc. processes outside of warcprox, such as for crawl job reporting.
Statistics are grouped by "bucket". Every capture is counted as part of the Statistics are grouped by "bucket". Every capture is counted as part of the
``__all__`` bucket. Other buckets can be specified in the ``Warcprox-Meta`` ``__all__`` bucket. Other buckets can be specified in the ``Warcprox-Meta``
@ -113,21 +114,20 @@ request header. The fallback bucket in case none is specified is called
Within each bucket are three sub-buckets: Within each bucket are three sub-buckets:
* ``new`` - tallies captures for which a complete record (usually a ``response`` * ``new`` - tallies captures for which a complete record (usually a
record) was written to warc ``response`` record) was written to a WARC file
* ``revisit`` - tallies captures for which a ``revisit`` record was written to * ``revisit`` - tallies captures for which a ``revisit`` record was written to
warc a WARC file
* ``total`` - includes all urls processed, even those not written to warc (so the * ``total`` - includes all URLs processed, even those not written to a WARC
numbers may be greater than new + revisit) file, and so may be greater than the sum of new and revisit records
Within each of these sub-buckets we keep two statistics: Within each of these sub-buckets, warcprox generates two kinds of statistics:
* ``urls`` - simple count of urls * ``urls`` - simple count of URLs
* ``wire_bytes`` - sum of bytes received over the wire, including http headers, * ``wire_bytes`` - sum of bytes received over the wire from the remote server
from the remote server for each url for each URL, including HTTP headers
For historical reasons, in sqlite, the default store, statistics are kept as For historical reasons, the default sqlite store keeps statistics as JSON blobs::
json blobs::
sqlite> select * from buckets_of_stats; sqlite> select * from buckets_of_stats;
bucket stats bucket stats
@ -139,14 +139,37 @@ Plugins
======= =======
Warcprox supports a limited notion of plugins by way of the ``--plugin`` Warcprox supports a limited notion of plugins by way of the ``--plugin``
command line argument. Plugin classes are loaded from the regular python module command line argument. Plugin classes are loaded from the regular python module
search path. They will be instantiated with one argument, a search path. They are instantiated with one argument that contains the values
``warcprox.Options``, which holds the values of all the command line arguments. of all command line arguments, ``warcprox.Options``. Legacy plugins with
Legacy plugins with constructors that take no arguments are also supported. constructors that take no arguments are also supported. Plugins should either
Plugins should either have a method ``notify(self, recorded_url, records)`` or have a method ``notify(self, recorded_url, records)`` or should subclass
should subclass ``warcprox.BasePostfetchProcessor``. More than one plugin can ``warcprox.BasePostfetchProcessor``. More than one plugin can be configured by
be configured by specifying ``--plugin`` multiples times. specifying ``--plugin`` multiples times.
`A minimal example <https://github.com/internetarchive/warcprox/blob/318405e795ac0ab8760988a1a482cf0a17697148/warcprox/__init__.py#L165>`__ See a minimal example `here
<https://github.com/internetarchive/warcprox/blob/318405e795ac0ab8760988a1a482cf0a17697148/warcprox/__init__.py#L165>`__.
Architecture
============
.. image:: arch.svg
Warcprox is multithreaded. It has pool of http proxy threads (100 by default).
When handling a request, a proxy thread records data from the remote server to
an in-memory buffer that spills over to disk if necessary (after 512k by
default), while it streams the data to the proxy client. Once the HTTP
transaction is complete, it puts the recorded URL in a thread-safe queue, to be
picked up by the first processor in the postfetch chain.
The postfetch chain normally includes processors for loading deduplication
information, writing records to the WARC, saving deduplication information, and
updating statistics. The exact set of processors in the chain depends on
command line arguments; for example, plugins specified with ``--plugin`` are
processors in the postfetch chain. Each postfetch processor has its own thread
or threads. Thus the processors are able to run in parallel, independent of one
another. This design also enables them to process URLs in batch. For example,
the statistics processor gathers statistics for up to 10 seconds or 500 URLs,
whichever comes first, then updates the statistics database with just a few
queries.
License License
======= =======

433
arch.svg Normal file
View File

@ -0,0 +1,433 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="758.50537pt" height="234.7183pt" viewBox="0 0 758.50537 234.7183" version="1.1">
<defs>
<g>
<symbol overflow="visible" id="glyph0-0">
<path style="stroke:none;" d="M 0 0 L 5.15625 0 L 5.15625 -7.21875 L 0 -7.21875 Z M 2.578125 -4.078125 L 0.828125 -6.703125 L 4.328125 -6.703125 Z M 2.890625 -3.609375 L 4.640625 -6.234375 L 4.640625 -0.984375 Z M 0.828125 -0.515625 L 2.578125 -3.140625 L 4.328125 -0.515625 Z M 0.515625 -6.234375 L 2.265625 -3.609375 L 0.515625 -0.984375 Z M 0.515625 -6.234375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-1">
<path style="stroke:none;" d="M 0.78125 -6.953125 L 0.78125 -2.859375 C 0.78125 -0.703125 1.84375 0.109375 3.265625 0.109375 C 4.78125 0.109375 5.90625 -0.765625 5.90625 -2.890625 L 5.90625 -6.953125 L 4.984375 -6.953125 L 4.984375 -2.84375 C 4.984375 -1.296875 4.328125 -0.625 3.296875 -0.625 C 2.375 -0.625 1.6875 -1.28125 1.6875 -2.84375 L 1.6875 -6.953125 Z M 0.78125 -6.953125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-2">
<path style="stroke:none;" d="M 0.75 2.046875 L 1.65625 2.046875 L 1.65625 -0.671875 L 1.671875 -0.671875 C 1.96875 -0.171875 2.546875 0.109375 3.203125 0.109375 C 4.390625 0.109375 5.46875 -0.78125 5.46875 -2.5625 C 5.46875 -4.078125 4.5625 -5.109375 3.359375 -5.109375 C 2.546875 -5.109375 1.953125 -4.75 1.59375 -4.140625 L 1.5625 -4.140625 L 1.53125 -4.984375 L 0.71875 -4.984375 C 0.734375 -4.515625 0.75 -4 0.75 -3.359375 Z M 1.65625 -2.890625 C 1.65625 -3.015625 1.6875 -3.140625 1.71875 -3.265625 C 1.890625 -3.9375 2.46875 -4.390625 3.078125 -4.390625 C 4.046875 -4.390625 4.5625 -3.53125 4.5625 -2.53125 C 4.5625 -1.375 4.015625 -0.59375 3.046875 -0.59375 C 2.40625 -0.59375 1.859375 -1.03125 1.6875 -1.65625 C 1.671875 -1.78125 1.65625 -1.890625 1.65625 -2.03125 Z M 1.65625 -2.890625 "/>
</symbol>
<symbol overflow="visible" id="glyph0-3">
<path style="stroke:none;" d="M 4.15625 -7.328125 L 4.15625 -4.34375 L 4.140625 -4.34375 C 3.90625 -4.75 3.390625 -5.109375 2.625 -5.109375 C 1.40625 -5.109375 0.375 -4.078125 0.390625 -2.421875 C 0.390625 -0.90625 1.3125 0.109375 2.53125 0.109375 C 3.34375 0.109375 3.953125 -0.3125 4.234375 -0.859375 L 4.25 -0.859375 L 4.296875 0 L 5.109375 0 C 5.078125 -0.34375 5.0625 -0.84375 5.0625 -1.296875 L 5.0625 -7.328125 Z M 4.15625 -2.09375 C 4.15625 -1.953125 4.140625 -1.828125 4.109375 -1.703125 C 3.953125 -1.015625 3.390625 -0.625 2.78125 -0.625 C 1.8125 -0.625 1.3125 -1.453125 1.3125 -2.46875 C 1.3125 -3.5625 1.859375 -4.390625 2.8125 -4.390625 C 3.484375 -4.390625 3.984375 -3.921875 4.109375 -3.34375 C 4.140625 -3.234375 4.15625 -3.078125 4.15625 -2.953125 Z M 4.15625 -2.09375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-4">
<path style="stroke:none;" d="M 4.34375 0 C 4.28125 -0.34375 4.265625 -0.765625 4.265625 -1.203125 L 4.265625 -3.0625 C 4.265625 -4.0625 3.890625 -5.109375 2.359375 -5.109375 C 1.734375 -5.109375 1.140625 -4.921875 0.71875 -4.65625 L 0.921875 -4.0625 C 1.28125 -4.296875 1.765625 -4.4375 2.234375 -4.4375 C 3.25 -4.4375 3.359375 -3.6875 3.359375 -3.28125 L 3.359375 -3.171875 C 1.4375 -3.1875 0.359375 -2.53125 0.359375 -1.3125 C 0.359375 -0.59375 0.875 0.109375 1.890625 0.109375 C 2.59375 0.109375 3.140625 -0.234375 3.40625 -0.625 L 3.4375 -0.625 L 3.515625 0 Z M 3.375 -1.6875 C 3.375 -1.59375 3.359375 -1.484375 3.328125 -1.390625 C 3.1875 -0.96875 2.78125 -0.5625 2.125 -0.5625 C 1.65625 -0.5625 1.265625 -0.828125 1.265625 -1.421875 C 1.265625 -2.390625 2.390625 -2.5625 3.375 -2.546875 Z M 3.375 -1.6875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-5">
<path style="stroke:none;" d="M 0.953125 -6.1875 L 0.953125 -4.984375 L 0.1875 -4.984375 L 0.1875 -4.296875 L 0.953125 -4.296875 L 0.953125 -1.578125 C 0.953125 -0.984375 1.046875 -0.546875 1.3125 -0.28125 C 1.53125 -0.03125 1.859375 0.109375 2.296875 0.109375 C 2.640625 0.109375 2.921875 0.046875 3.09375 -0.015625 L 3.046875 -0.703125 C 2.9375 -0.671875 2.765625 -0.640625 2.53125 -0.640625 C 2.015625 -0.640625 1.84375 -0.984375 1.84375 -1.609375 L 1.84375 -4.296875 L 3.140625 -4.296875 L 3.140625 -4.984375 L 1.84375 -4.984375 L 1.84375 -6.421875 Z M 0.953125 -6.1875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-6">
<path style="stroke:none;" d="M 4.765625 -2.328125 C 4.78125 -2.421875 4.796875 -2.5625 4.796875 -2.75 C 4.796875 -3.671875 4.359375 -5.109375 2.734375 -5.109375 C 1.28125 -5.109375 0.390625 -3.921875 0.390625 -2.40625 C 0.390625 -0.90625 1.3125 0.109375 2.84375 0.109375 C 3.640625 0.109375 4.1875 -0.0625 4.5 -0.203125 L 4.359375 -0.859375 C 4.015625 -0.71875 3.625 -0.59375 2.96875 -0.59375 C 2.0625 -0.59375 1.28125 -1.109375 1.265625 -2.328125 Z M 1.265625 -2.984375 C 1.34375 -3.609375 1.75 -4.453125 2.65625 -4.453125 C 3.6875 -4.453125 3.921875 -3.5625 3.921875 -2.984375 Z M 1.265625 -2.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-7">
<path style="stroke:none;" d="M 0.4375 -0.34375 C 0.796875 -0.09375 1.53125 0.109375 2.203125 0.109375 C 3.84375 0.109375 4.640625 -0.828125 4.640625 -1.890625 C 4.640625 -2.921875 4.046875 -3.484375 2.859375 -3.9375 C 1.90625 -4.3125 1.484375 -4.625 1.484375 -5.28125 C 1.484375 -5.75 1.84375 -6.328125 2.796875 -6.328125 C 3.421875 -6.328125 3.890625 -6.109375 4.109375 -5.984375 L 4.359375 -6.71875 C 4.046875 -6.90625 3.53125 -7.0625 2.828125 -7.0625 C 1.46875 -7.0625 0.578125 -6.265625 0.578125 -5.171875 C 0.578125 -4.203125 1.28125 -3.609375 2.40625 -3.203125 C 3.359375 -2.84375 3.71875 -2.46875 3.71875 -1.828125 C 3.71875 -1.125 3.1875 -0.640625 2.265625 -0.640625 C 1.65625 -0.640625 1.0625 -0.84375 0.65625 -1.09375 Z M 0.4375 -0.34375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-8">
<path style="stroke:none;" d="M 0.40625 -0.234375 C 0.75 -0.03125 1.265625 0.109375 1.8125 0.109375 C 2.984375 0.109375 3.671875 -0.515625 3.671875 -1.390625 C 3.671875 -2.140625 3.234375 -2.5625 2.359375 -2.890625 C 1.71875 -3.140625 1.40625 -3.328125 1.40625 -3.75 C 1.40625 -4.109375 1.71875 -4.421875 2.25 -4.421875 C 2.71875 -4.421875 3.078125 -4.265625 3.265625 -4.140625 L 3.5 -4.796875 C 3.21875 -4.953125 2.78125 -5.109375 2.265625 -5.109375 C 1.203125 -5.109375 0.546875 -4.4375 0.546875 -3.640625 C 0.546875 -3.046875 0.96875 -2.546875 1.859375 -2.234375 C 2.53125 -1.984375 2.796875 -1.75 2.796875 -1.3125 C 2.796875 -0.890625 2.484375 -0.5625 1.828125 -0.5625 C 1.375 -0.5625 0.890625 -0.75 0.625 -0.921875 Z M 0.40625 -0.234375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-9">
<path style="stroke:none;" d="M 0.78125 0 L 4.65625 0 L 4.65625 -0.75 L 1.6875 -0.75 L 1.6875 -6.953125 L 0.78125 -6.953125 Z M 0.78125 0 "/>
</symbol>
<symbol overflow="visible" id="glyph0-10">
<path style="stroke:none;" d="M 2.859375 -5.109375 C 1.484375 -5.109375 0.390625 -4.125 0.390625 -2.453125 C 0.390625 -0.875 1.4375 0.109375 2.78125 0.109375 C 3.984375 0.109375 5.265625 -0.6875 5.265625 -2.53125 C 5.265625 -4.0625 4.296875 -5.109375 2.859375 -5.109375 Z M 2.84375 -4.421875 C 3.921875 -4.421875 4.34375 -3.359375 4.34375 -2.5 C 4.34375 -1.375 3.6875 -0.5625 2.828125 -0.5625 C 1.9375 -0.5625 1.3125 -1.390625 1.3125 -2.484375 C 1.3125 -3.4375 1.78125 -4.421875 2.84375 -4.421875 Z M 2.84375 -4.421875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-11">
<path style="stroke:none;" d="M 4.15625 -0.859375 C 3.890625 -0.75 3.5625 -0.625 3.046875 -0.625 C 2.046875 -0.625 1.3125 -1.328125 1.3125 -2.484375 C 1.3125 -3.53125 1.921875 -4.375 3.078125 -4.375 C 3.5625 -4.375 3.90625 -4.265625 4.125 -4.140625 L 4.328125 -4.84375 C 4.078125 -4.953125 3.609375 -5.09375 3.078125 -5.09375 C 1.4375 -5.09375 0.390625 -3.984375 0.390625 -2.4375 C 0.390625 -0.921875 1.375 0.109375 2.875 0.109375 C 3.546875 0.109375 4.078125 -0.0625 4.3125 -0.1875 Z M 4.15625 -0.859375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-12">
<path style="stroke:none;" d="M 0.75 0 L 1.65625 0 L 1.65625 -7.328125 L 0.75 -7.328125 Z M 0.75 0 "/>
</symbol>
<symbol overflow="visible" id="glyph0-13">
<path style="stroke:none;" d="M 0.78125 -0.015625 C 1.234375 0.03125 1.78125 0.0625 2.40625 0.0625 C 3.765625 0.0625 4.84375 -0.28125 5.484375 -0.9375 C 6.140625 -1.578125 6.484375 -2.5 6.484375 -3.640625 C 6.484375 -4.765625 6.125 -5.5625 5.5 -6.140625 C 4.90625 -6.703125 3.984375 -7 2.6875 -7 C 1.984375 -7 1.328125 -6.9375 0.78125 -6.859375 Z M 1.6875 -6.203125 C 1.921875 -6.25 2.265625 -6.296875 2.734375 -6.296875 C 4.625 -6.296875 5.5625 -5.25 5.546875 -3.609375 C 5.546875 -1.734375 4.5 -0.65625 2.59375 -0.65625 C 2.234375 -0.65625 1.90625 -0.671875 1.6875 -0.71875 Z M 1.6875 -6.203125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-14">
<path style="stroke:none;" d="M 1.65625 0 L 1.65625 -4.984375 L 0.75 -4.984375 L 0.75 0 Z M 1.203125 -6.96875 C 0.875 -6.96875 0.640625 -6.71875 0.640625 -6.390625 C 0.640625 -6.078125 0.859375 -5.84375 1.1875 -5.84375 C 1.546875 -5.84375 1.78125 -6.078125 1.765625 -6.390625 C 1.765625 -6.71875 1.546875 -6.96875 1.203125 -6.96875 Z M 1.203125 -6.96875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-15">
<path style="stroke:none;" d="M 1.65625 -7.328125 L 0.75 -7.328125 L 0.75 0 L 1.65625 0 L 1.65625 -1.875 L 2.109375 -2.390625 L 3.84375 0 L 4.9375 0 L 2.75 -2.9375 L 4.671875 -4.984375 L 3.578125 -4.984375 L 2.109375 -3.265625 C 1.96875 -3.09375 1.796875 -2.875 1.671875 -2.703125 L 1.65625 -2.703125 Z M 1.65625 -7.328125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-16">
<path style="stroke:none;" d="M 4.921875 -4.984375 L 4.015625 -4.984375 L 4.015625 -1.921875 C 4.015625 -1.765625 3.984375 -1.59375 3.9375 -1.46875 C 3.78125 -1.0625 3.359375 -0.640625 2.75 -0.640625 C 1.921875 -0.640625 1.625 -1.28125 1.625 -2.234375 L 1.625 -4.984375 L 0.71875 -4.984375 L 0.71875 -2.078125 C 0.71875 -0.3125 1.65625 0.109375 2.4375 0.109375 C 3.328125 0.109375 3.859375 -0.40625 4.09375 -0.8125 L 4.109375 -0.8125 L 4.171875 0 L 4.96875 0 C 4.9375 -0.390625 4.921875 -0.84375 4.921875 -1.359375 Z M 4.921875 -4.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-17">
<path style="stroke:none;" d="M 0.78125 -0.015625 C 1.078125 0.015625 1.5625 0.0625 2.171875 0.0625 C 3.3125 0.0625 4.09375 -0.140625 4.578125 -0.59375 C 4.921875 -0.921875 5.171875 -1.375 5.171875 -1.984375 C 5.171875 -3.015625 4.390625 -3.5625 3.734375 -3.71875 L 3.734375 -3.75 C 4.46875 -4.015625 4.90625 -4.59375 4.90625 -5.265625 C 4.90625 -5.8125 4.6875 -6.234375 4.328125 -6.5 C 3.890625 -6.84375 3.328125 -7 2.421875 -7 C 1.796875 -7 1.171875 -6.9375 0.78125 -6.859375 Z M 1.6875 -6.25 C 1.828125 -6.28125 2.0625 -6.3125 2.46875 -6.3125 C 3.375 -6.3125 4 -5.984375 4 -5.171875 C 4 -4.5 3.4375 -4.015625 2.5 -4.015625 L 1.6875 -4.015625 Z M 1.6875 -3.328125 L 2.421875 -3.328125 C 3.40625 -3.328125 4.21875 -2.9375 4.21875 -1.984375 C 4.21875 -0.984375 3.359375 -0.640625 2.4375 -0.640625 C 2.109375 -0.640625 1.859375 -0.65625 1.6875 -0.6875 Z M 1.6875 -3.328125 "/>
</symbol>
<symbol overflow="visible" id="glyph0-18">
<path style="stroke:none;" d="M 0.78125 0 L 1.6875 0 L 1.6875 -2.78125 C 1.890625 -2.734375 2.140625 -2.71875 2.40625 -2.71875 C 3.28125 -2.71875 4.046875 -2.984375 4.53125 -3.484375 C 4.875 -3.84375 5.0625 -4.34375 5.0625 -4.96875 C 5.0625 -5.59375 4.84375 -6.09375 4.453125 -6.421875 C 4.046875 -6.796875 3.390625 -7 2.5 -7 C 1.78125 -7 1.21875 -6.9375 0.78125 -6.875 Z M 1.6875 -6.21875 C 1.828125 -6.265625 2.140625 -6.296875 2.53125 -6.296875 C 3.515625 -6.296875 4.171875 -5.84375 4.171875 -4.921875 C 4.171875 -3.96875 3.5 -3.4375 2.421875 -3.4375 C 2.125 -3.4375 1.875 -3.46875 1.6875 -3.515625 Z M 1.6875 -6.21875 "/>
</symbol>
<symbol overflow="visible" id="glyph0-19">
<path style="stroke:none;" d="M 0.75 0 L 1.65625 0 L 1.65625 -2.65625 C 1.65625 -2.8125 1.671875 -2.953125 1.6875 -3.078125 C 1.8125 -3.765625 2.265625 -4.25 2.90625 -4.25 C 3.03125 -4.25 3.125 -4.234375 3.21875 -4.21875 L 3.21875 -5.078125 C 3.140625 -5.09375 3.0625 -5.109375 2.953125 -5.109375 C 2.34375 -5.109375 1.8125 -4.6875 1.578125 -4.015625 L 1.53125 -4.015625 L 1.5 -4.984375 L 0.71875 -4.984375 C 0.75 -4.53125 0.75 -4.015625 0.75 -3.4375 Z M 0.75 0 "/>
</symbol>
<symbol overflow="visible" id="glyph0-20">
<path style="stroke:none;" d="M 0.171875 -4.984375 L 1.859375 -2.546875 L 0.078125 0 L 1.078125 0 L 1.8125 -1.125 C 1.984375 -1.421875 2.171875 -1.6875 2.328125 -1.984375 L 2.34375 -1.984375 C 2.53125 -1.6875 2.6875 -1.40625 2.890625 -1.125 L 3.625 0 L 4.65625 0 L 2.890625 -2.578125 L 4.59375 -4.984375 L 3.625 -4.984375 L 2.921875 -3.921875 C 2.75 -3.65625 2.59375 -3.40625 2.421875 -3.109375 L 2.390625 -3.109375 C 2.234375 -3.375 2.078125 -3.640625 1.890625 -3.921875 L 1.171875 -4.984375 Z M 0.171875 -4.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-21">
<path style="stroke:none;" d="M 0.09375 -4.984375 L 1.9375 -0.390625 C 1.984375 -0.28125 2 -0.203125 2 -0.15625 C 2 -0.109375 1.96875 -0.03125 1.921875 0.0625 C 1.71875 0.53125 1.40625 0.875 1.171875 1.078125 C 0.890625 1.296875 0.59375 1.4375 0.375 1.515625 L 0.59375 2.28125 C 0.828125 2.234375 1.265625 2.078125 1.71875 1.6875 C 2.328125 1.15625 2.78125 0.28125 3.421875 -1.4375 L 4.78125 -4.984375 L 3.828125 -4.984375 L 2.84375 -2.0625 C 2.71875 -1.703125 2.609375 -1.3125 2.515625 -1.015625 L 2.5 -1.015625 C 2.40625 -1.3125 2.296875 -1.71875 2.171875 -2.046875 L 1.078125 -4.984375 Z M 0.09375 -4.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-22">
<path style="stroke:none;" d="M 6.78125 0.265625 C 6.1875 0.171875 5.4375 0 4.75 -0.171875 L 4.75 -0.21875 C 5.90625 -0.625 6.71875 -1.765625 6.71875 -3.546875 C 6.71875 -5.59375 5.5 -7.0625 3.609375 -7.0625 C 1.734375 -7.0625 0.375 -5.625 0.375 -3.40625 C 0.375 -1.171875 1.78125 0.046875 3.4375 0.109375 C 3.5625 0.109375 3.71875 0.171875 3.859375 0.21875 C 4.65625 0.5 5.578125 0.78125 6.515625 1.015625 Z M 3.53125 -0.625 C 2.140625 -0.625 1.3125 -1.953125 1.328125 -3.421875 C 1.3125 -4.921875 2.0625 -6.328125 3.578125 -6.328125 C 5.046875 -6.328125 5.78125 -4.90625 5.78125 -3.5 C 5.78125 -1.921875 4.96875 -0.625 3.53125 -0.625 Z M 3.53125 -0.625 "/>
</symbol>
<symbol overflow="visible" id="glyph0-23">
<path style="stroke:none;" d="M 2.859375 0 L 3.84375 -3.625 C 4.109375 -4.515625 4.265625 -5.203125 4.390625 -5.890625 L 4.40625 -5.890625 C 4.5 -5.1875 4.640625 -4.5 4.859375 -3.625 L 5.734375 0 L 6.671875 0 L 8.640625 -6.953125 L 7.71875 -6.953125 L 6.8125 -3.4375 C 6.59375 -2.578125 6.390625 -1.8125 6.25 -1.046875 L 6.234375 -1.046875 C 6.125 -1.78125 5.9375 -2.59375 5.75 -3.421875 L 4.90625 -6.953125 L 3.953125 -6.953125 L 3.03125 -3.4375 C 2.796875 -2.546875 2.578125 -1.71875 2.46875 -1.03125 L 2.4375 -1.03125 C 2.328125 -1.703125 2.140625 -2.5625 1.921875 -3.4375 L 1.109375 -6.953125 L 0.15625 -6.953125 L 1.921875 0 Z M 2.859375 0 "/>
</symbol>
<symbol overflow="visible" id="glyph0-24">
<path style="stroke:none;" d=""/>
</symbol>
<symbol overflow="visible" id="glyph0-25">
<path style="stroke:none;" d="M 4.375 -2.1875 L 5.109375 0 L 6.078125 0 L 3.703125 -6.953125 L 2.625 -6.953125 L 0.265625 0 L 1.1875 0 L 1.90625 -2.1875 Z M 2.09375 -2.890625 L 2.78125 -4.90625 C 2.90625 -5.328125 3.015625 -5.75 3.125 -6.15625 L 3.140625 -6.15625 C 3.25 -5.75 3.359375 -5.34375 3.5 -4.890625 L 4.1875 -2.890625 Z M 2.09375 -2.890625 "/>
</symbol>
<symbol overflow="visible" id="glyph0-26">
<path style="stroke:none;" d="M 0.78125 0 L 1.6875 0 L 1.6875 -3.015625 L 2.53125 -3.015625 C 3.34375 -2.984375 3.71875 -2.625 3.921875 -1.65625 C 4.109375 -0.796875 4.265625 -0.203125 4.390625 0 L 5.3125 0 C 5.171875 -0.265625 5 -0.9375 4.78125 -1.90625 C 4.609375 -2.625 4.296875 -3.125 3.75 -3.3125 L 3.75 -3.34375 C 4.484375 -3.59375 5.0625 -4.203125 5.0625 -5.109375 C 5.0625 -5.65625 4.875 -6.125 4.515625 -6.4375 C 4.078125 -6.828125 3.46875 -7 2.5 -7 C 1.890625 -7 1.234375 -6.953125 0.78125 -6.859375 Z M 1.6875 -6.234375 C 1.828125 -6.265625 2.140625 -6.3125 2.5625 -6.3125 C 3.515625 -6.296875 4.171875 -5.90625 4.171875 -5.015625 C 4.171875 -4.21875 3.5625 -3.6875 2.59375 -3.6875 L 1.6875 -3.6875 Z M 1.6875 -6.234375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-27">
<path style="stroke:none;" d="M 5.453125 -0.9375 C 5.09375 -0.765625 4.53125 -0.65625 3.984375 -0.65625 C 2.296875 -0.65625 1.3125 -1.75 1.3125 -3.4375 C 1.3125 -5.265625 2.40625 -6.3125 4.03125 -6.3125 C 4.609375 -6.3125 5.09375 -6.1875 5.4375 -6.015625 L 5.65625 -6.75 C 5.421875 -6.875 4.875 -7.0625 4 -7.0625 C 1.84375 -7.0625 0.375 -5.59375 0.375 -3.40625 C 0.375 -1.140625 1.84375 0.109375 3.8125 0.109375 C 4.65625 0.109375 5.3125 -0.0625 5.640625 -0.234375 Z M 5.453125 -0.9375 "/>
</symbol>
<symbol overflow="visible" id="glyph0-28">
<path style="stroke:none;" d="M 0.140625 -4.984375 L 2.03125 0 L 2.890625 0 L 4.859375 -4.984375 L 3.90625 -4.984375 L 2.9375 -2.1875 C 2.78125 -1.734375 2.625 -1.3125 2.515625 -0.90625 L 2.484375 -0.90625 C 2.375 -1.3125 2.25 -1.734375 2.078125 -2.1875 L 1.109375 -4.984375 Z M 0.140625 -4.984375 "/>
</symbol>
<symbol overflow="visible" id="glyph1-0">
<path style="stroke:none;" d="M 0 0 L 4.59375 0 L 4.59375 -6.421875 L 0 -6.421875 Z M 2.296875 -3.625 L 0.734375 -5.96875 L 3.859375 -5.96875 Z M 2.5625 -3.203125 L 4.125 -5.546875 L 4.125 -0.875 Z M 0.734375 -0.453125 L 2.296875 -2.796875 L 3.859375 -0.453125 Z M 0.453125 -5.546875 L 2.015625 -3.203125 L 0.453125 -0.875 Z M 0.453125 -5.546875 "/>
</symbol>
<symbol overflow="visible" id="glyph1-1">
<path style="stroke:none;" d="M 1.875 0 L 2.671875 0 L 2.671875 -5.5 L 4.5625 -5.5 L 4.5625 -6.1875 L -0.015625 -6.1875 L -0.015625 -5.5 L 1.875 -5.5 Z M 1.875 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-2">
<path style="stroke:none;" d="M 0.671875 0 L 1.484375 0 L 1.484375 -2.671875 C 1.484375 -2.828125 1.484375 -2.953125 1.53125 -3.0625 C 1.671875 -3.5 2.09375 -3.875 2.609375 -3.875 C 3.375 -3.875 3.640625 -3.265625 3.640625 -2.546875 L 3.640625 0 L 4.453125 0 L 4.453125 -2.640625 C 4.453125 -4.171875 3.5 -4.546875 2.890625 -4.546875 C 2.59375 -4.546875 2.3125 -4.453125 2.078125 -4.3125 C 1.828125 -4.171875 1.625 -3.96875 1.5 -3.734375 L 1.484375 -3.734375 L 1.484375 -6.515625 L 0.671875 -6.515625 Z M 0.671875 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-3">
<path style="stroke:none;" d="M 4.234375 -2.078125 C 4.25 -2.15625 4.265625 -2.28125 4.265625 -2.453125 C 4.265625 -3.265625 3.875 -4.546875 2.4375 -4.546875 C 1.140625 -4.546875 0.34375 -3.484375 0.34375 -2.140625 C 0.34375 -0.8125 1.171875 0.09375 2.53125 0.09375 C 3.234375 0.09375 3.71875 -0.0625 4.015625 -0.1875 L 3.875 -0.765625 C 3.5625 -0.640625 3.21875 -0.53125 2.640625 -0.53125 C 1.828125 -0.53125 1.140625 -0.984375 1.125 -2.078125 Z M 1.125 -2.65625 C 1.1875 -3.203125 1.546875 -3.96875 2.359375 -3.96875 C 3.28125 -3.96875 3.5 -3.171875 3.484375 -2.65625 Z M 1.125 -2.65625 "/>
</symbol>
<symbol overflow="visible" id="glyph1-4">
<path style="stroke:none;" d="M 0.703125 -6.1875 L 0.703125 0 L 1.5 0 L 1.5 -6.1875 Z M 0.703125 -6.1875 "/>
</symbol>
<symbol overflow="visible" id="glyph1-5">
<path style="stroke:none;" d="M 0.671875 0 L 1.484375 0 L 1.484375 -2.671875 C 1.484375 -2.8125 1.5 -2.9375 1.53125 -3.046875 C 1.671875 -3.5 2.078125 -3.875 2.609375 -3.875 C 3.375 -3.875 3.640625 -3.28125 3.640625 -2.5625 L 3.640625 0 L 4.453125 0 L 4.453125 -2.65625 C 4.453125 -4.171875 3.5 -4.546875 2.875 -4.546875 C 2.140625 -4.546875 1.625 -4.125 1.40625 -3.703125 L 1.390625 -3.703125 L 1.34375 -4.4375 L 0.640625 -4.4375 C 0.65625 -4.078125 0.671875 -3.703125 0.671875 -3.234375 Z M 0.671875 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-6">
<path style="stroke:none;" d="M 0.859375 -5.5 L 0.859375 -4.4375 L 0.171875 -4.4375 L 0.171875 -3.828125 L 0.859375 -3.828125 L 0.859375 -1.40625 C 0.859375 -0.875 0.9375 -0.484375 1.171875 -0.25 C 1.359375 -0.03125 1.65625 0.09375 2.03125 0.09375 C 2.34375 0.09375 2.59375 0.046875 2.75 -0.015625 L 2.71875 -0.625 C 2.609375 -0.59375 2.453125 -0.5625 2.25 -0.5625 C 1.796875 -0.5625 1.640625 -0.875 1.640625 -1.4375 L 1.640625 -3.828125 L 2.796875 -3.828125 L 2.796875 -4.4375 L 1.640625 -4.4375 L 1.640625 -5.71875 Z M 0.859375 -5.5 "/>
</symbol>
<symbol overflow="visible" id="glyph1-7">
<path style="stroke:none;" d="M 0.671875 0 L 1.46875 0 L 1.46875 -2.359375 C 1.46875 -2.5 1.484375 -2.625 1.5 -2.75 C 1.609375 -3.34375 2.015625 -3.78125 2.59375 -3.78125 C 2.703125 -3.78125 2.78125 -3.765625 2.859375 -3.75 L 2.859375 -4.515625 C 2.78125 -4.53125 2.71875 -4.546875 2.625 -4.546875 C 2.09375 -4.546875 1.609375 -4.171875 1.40625 -3.5625 L 1.359375 -3.5625 L 1.34375 -4.4375 L 0.640625 -4.4375 C 0.65625 -4.03125 0.671875 -3.578125 0.671875 -3.046875 Z M 0.671875 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-8">
<path style="stroke:none;" d="M 6.0625 0 L 6.84375 0 L 6.453125 -6.1875 L 5.4375 -6.1875 L 4.34375 -3.1875 C 4.0625 -2.40625 3.84375 -1.734375 3.6875 -1.109375 L 3.65625 -1.109375 C 3.5 -1.75 3.296875 -2.4375 3.03125 -3.1875 L 1.984375 -6.1875 L 0.96875 -6.1875 L 0.53125 0 L 1.296875 0 L 1.453125 -2.65625 C 1.515625 -3.578125 1.5625 -4.609375 1.578125 -5.390625 L 1.59375 -5.390625 C 1.765625 -4.65625 2.015625 -3.859375 2.3125 -2.984375 L 3.3125 -0.03125 L 3.921875 -0.03125 L 5.015625 -3.03125 C 5.3125 -3.890625 5.578125 -4.65625 5.78125 -5.390625 L 5.8125 -5.390625 C 5.8125 -4.609375 5.859375 -3.578125 5.90625 -2.71875 Z M 6.0625 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-9">
<path style="stroke:none;" d="M 2.546875 -4.546875 C 1.328125 -4.546875 0.34375 -3.671875 0.34375 -2.1875 C 0.34375 -0.78125 1.28125 0.09375 2.46875 0.09375 C 3.546875 0.09375 4.6875 -0.609375 4.6875 -2.25 C 4.6875 -3.609375 3.828125 -4.546875 2.546875 -4.546875 Z M 2.53125 -3.9375 C 3.484375 -3.9375 3.859375 -2.984375 3.859375 -2.234375 C 3.859375 -1.234375 3.28125 -0.5 2.515625 -0.5 C 1.71875 -0.5 1.171875 -1.234375 1.171875 -2.203125 C 1.171875 -3.046875 1.578125 -3.9375 2.53125 -3.9375 Z M 2.53125 -3.9375 "/>
</symbol>
<symbol overflow="visible" id="glyph1-10">
<path style="stroke:none;" d=""/>
</symbol>
<symbol overflow="visible" id="glyph1-11">
<path style="stroke:none;" d="M 0.390625 -0.296875 C 0.71875 -0.078125 1.359375 0.09375 1.96875 0.09375 C 3.421875 0.09375 4.125 -0.734375 4.125 -1.6875 C 4.125 -2.59375 3.59375 -3.09375 2.546875 -3.5 C 1.703125 -3.828125 1.328125 -4.125 1.328125 -4.703125 C 1.328125 -5.125 1.640625 -5.625 2.484375 -5.625 C 3.046875 -5.625 3.453125 -5.4375 3.65625 -5.328125 L 3.875 -5.984375 C 3.609375 -6.140625 3.140625 -6.28125 2.515625 -6.28125 C 1.3125 -6.28125 0.515625 -5.5625 0.515625 -4.609375 C 0.515625 -3.734375 1.140625 -3.203125 2.140625 -2.859375 C 2.984375 -2.53125 3.3125 -2.203125 3.3125 -1.625 C 3.3125 -1 2.828125 -0.5625 2.015625 -0.5625 C 1.46875 -0.5625 0.9375 -0.75 0.59375 -0.96875 Z M 0.390625 -0.296875 "/>
</symbol>
<symbol overflow="visible" id="glyph1-12">
<path style="stroke:none;" d="M 0.671875 1.8125 L 1.46875 1.8125 L 1.46875 -0.59375 L 1.484375 -0.59375 C 1.75 -0.15625 2.265625 0.09375 2.859375 0.09375 C 3.890625 0.09375 4.875 -0.6875 4.875 -2.28125 C 4.875 -3.625 4.0625 -4.546875 2.984375 -4.546875 C 2.265625 -4.546875 1.75 -4.21875 1.40625 -3.671875 L 1.390625 -3.671875 L 1.359375 -4.4375 L 0.640625 -4.4375 C 0.65625 -4.015625 0.671875 -3.5625 0.671875 -2.984375 Z M 1.46875 -2.5625 C 1.46875 -2.671875 1.5 -2.796875 1.515625 -2.890625 C 1.671875 -3.5 2.1875 -3.890625 2.75 -3.890625 C 3.59375 -3.890625 4.0625 -3.140625 4.0625 -2.25 C 4.0625 -1.234375 3.5625 -0.53125 2.71875 -0.53125 C 2.140625 -0.53125 1.65625 -0.921875 1.5 -1.484375 C 1.484375 -1.578125 1.46875 -1.6875 1.46875 -1.8125 Z M 1.46875 -2.5625 "/>
</symbol>
<symbol overflow="visible" id="glyph1-13">
<path style="stroke:none;" d="M 0.359375 -0.21875 C 0.671875 -0.03125 1.125 0.09375 1.609375 0.09375 C 2.65625 0.09375 3.265625 -0.453125 3.265625 -1.234375 C 3.265625 -1.90625 2.875 -2.28125 2.09375 -2.578125 C 1.515625 -2.796875 1.25 -2.96875 1.25 -3.328125 C 1.25 -3.65625 1.515625 -3.9375 2 -3.9375 C 2.40625 -3.9375 2.734375 -3.78125 2.90625 -3.671875 L 3.109375 -4.265625 C 2.859375 -4.40625 2.46875 -4.546875 2.015625 -4.546875 C 1.0625 -4.546875 0.484375 -3.953125 0.484375 -3.234375 C 0.484375 -2.703125 0.859375 -2.265625 1.65625 -1.984375 C 2.25 -1.765625 2.484375 -1.546875 2.484375 -1.171875 C 2.484375 -0.796875 2.203125 -0.5 1.625 -0.5 C 1.21875 -0.5 0.796875 -0.671875 0.5625 -0.828125 Z M 0.359375 -0.21875 "/>
</symbol>
<symbol overflow="visible" id="glyph1-14">
<path style="stroke:none;" d="M 1.78125 -6.359375 C 1.203125 -5.59375 0.59375 -4.421875 0.59375 -2.609375 C 0.59375 -0.828125 1.203125 0.34375 1.78125 1.109375 L 2.421875 1.109375 C 1.765625 0.1875 1.265625 -0.984375 1.265625 -2.609375 C 1.265625 -4.28125 1.75 -5.46875 2.421875 -6.359375 Z M 1.78125 -6.359375 "/>
</symbol>
<symbol overflow="visible" id="glyph1-15">
<path style="stroke:none;" d="M 3.890625 -1.9375 L 4.546875 0 L 5.40625 0 L 3.296875 -6.1875 L 2.328125 -6.1875 L 0.234375 0 L 1.0625 0 L 1.703125 -1.9375 Z M 1.859375 -2.5625 L 2.46875 -4.359375 C 2.59375 -4.734375 2.6875 -5.109375 2.78125 -5.46875 L 2.796875 -5.46875 C 2.890625 -5.125 2.984375 -4.75 3.125 -4.34375 L 3.71875 -2.5625 Z M 1.859375 -2.5625 "/>
</symbol>
<symbol overflow="visible" id="glyph1-16">
<path style="stroke:none;" d="M 1.453125 0 L 1.453125 -2.640625 C 1.453125 -3.671875 1.4375 -4.40625 1.390625 -5.1875 L 1.40625 -5.203125 C 1.71875 -4.53125 2.140625 -3.828125 2.5625 -3.140625 L 4.53125 0 L 5.34375 0 L 5.34375 -6.1875 L 4.59375 -6.1875 L 4.59375 -3.59375 C 4.59375 -2.625 4.609375 -1.875 4.671875 -1.0625 L 4.65625 -1.046875 C 4.359375 -1.671875 4.015625 -2.328125 3.546875 -3.046875 L 1.5625 -6.1875 L 0.703125 -6.1875 L 0.703125 0 Z M 1.453125 0 "/>
</symbol>
<symbol overflow="visible" id="glyph1-17">
<path style="stroke:none;" d="M 3.703125 -6.515625 L 3.703125 -3.859375 L 3.671875 -3.859375 C 3.46875 -4.21875 3.015625 -4.546875 2.34375 -4.546875 C 1.25 -4.546875 0.34375 -3.625 0.34375 -2.15625 C 0.34375 -0.8125 1.171875 0.09375 2.25 0.09375 C 2.96875 0.09375 3.515625 -0.28125 3.765625 -0.765625 L 3.78125 -0.765625 L 3.8125 0 L 4.546875 0 C 4.515625 -0.296875 4.5 -0.75 4.5 -1.140625 L 4.5 -6.515625 Z M 3.703125 -1.859375 C 3.703125 -1.734375 3.6875 -1.625 3.65625 -1.515625 C 3.515625 -0.90625 3.015625 -0.546875 2.46875 -0.546875 C 1.609375 -0.546875 1.171875 -1.296875 1.171875 -2.1875 C 1.171875 -3.171875 1.65625 -3.90625 2.5 -3.90625 C 3.09375 -3.90625 3.546875 -3.484375 3.65625 -2.96875 C 3.6875 -2.875 3.703125 -2.734375 3.703125 -2.625 Z M 3.703125 -1.859375 "/>
</symbol>
<symbol overflow="visible" id="glyph1-18">
<path style="stroke:none;" d="M 0.8125 1.109375 C 1.390625 0.328125 2.015625 -0.828125 2.015625 -2.625 C 2.015625 -4.4375 1.390625 -5.609375 0.8125 -6.359375 L 0.1875 -6.359375 C 0.84375 -5.453125 1.34375 -4.28125 1.34375 -2.640625 C 1.34375 -0.984375 0.828125 0.203125 0.1875 1.109375 Z M 0.8125 1.109375 "/>
</symbol>
</g>
<clipPath id="clip1">
<path d="M 276 159 L 352 159 L 352 234.71875 L 276 234.71875 Z M 276 159 "/>
</clipPath>
<clipPath id="clip2">
<path d="M 0 146 L 111 146 L 111 234.71875 L 0 234.71875 Z M 0 146 "/>
</clipPath>
<clipPath id="clip3">
<path d="M 648 63 L 758.503906 63 L 758.503906 159 L 648 159 Z M 648 63 "/>
</clipPath>
</defs>
<g id="surface1">
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 628.22968 287.905981 L 557.32343 287.905981 L 557.32343 331.363013 L 628.22968 331.363013 Z M 628.22968 287.905981 " transform="matrix(1,0,0,-1,-15.69843,422.3552)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-1" x="561.16777" y="109.5573"/>
<use xlink:href="#glyph0-2" x="567.844034" y="109.5573"/>
<use xlink:href="#glyph0-3" x="573.715431" y="109.5573"/>
<use xlink:href="#glyph0-4" x="579.535234" y="109.5573"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-5" x="584.46762" y="109.5573"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="587.82123" y="109.5573"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-7" x="566.62637" y="121.9399"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-5" x="571.682582" y="121.9399"/>
<use xlink:href="#glyph0-4" x="575.098105" y="121.9399"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-5" x="580.030491" y="121.9399"/>
<use xlink:href="#glyph0-8" x="583.446014" y="121.9399"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00117375 0.00166875 L -81.018358 -42.349894 L -84.178514 -36.13505 L -95.52617 -60.244425 L -69.362108 -65.295206 L -72.522264 -59.076456 L 5.048049 -18.213175 " transform="matrix(1,0,0,-1,313.65117,29.8962)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00023625 0.0011625 L -100.382576 51.657413 L -97.120857 57.821475 L -123.363045 53.208194 L -112.421639 28.911319 L -109.15992 35.075381 L -31.070076 -4.791806 " transform="matrix(1,0,0,-1,462.62867,104.2746)"/>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 291.035156 174.058594 L 291.035156 220.523438 C 291.035156 224.558594 301.429688 227.832031 314.253906 227.832031 C 327.074219 227.832031 337.46875 224.558594 337.46875 220.523438 L 337.46875 174.058594 Z M 291.035156 174.058594 "/>
<g clip-path="url(#clip1)" clip-rule="nonzero">
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00018625 0.00070625 L 0.00018625 -46.464137 C 0.00018625 -50.499294 10.394718 -53.772731 23.218936 -53.772731 C 36.039249 -53.772731 46.43378 -50.499294 46.43378 -46.464137 L 46.43378 0.00070625 Z M 0.00018625 0.00070625 " transform="matrix(1,0,0,-1,291.03497,174.0593)"/>
</g>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 337.46875 174.058594 C 337.46875 178.097656 327.074219 181.371094 314.253906 181.371094 C 301.429688 181.371094 291.035156 178.097656 291.035156 174.058594 C 291.035156 170.023438 301.429688 166.75 314.253906 166.75 C 327.074219 166.75 337.46875 170.023438 337.46875 174.058594 "/>
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00132 0.00070625 C -0.00132 -4.038356 -10.395851 -7.311794 -23.216164 -7.311794 C -36.040383 -7.311794 -46.434914 -4.038356 -46.434914 0.00070625 C -46.434914 4.035863 -36.040383 7.3093 -23.216164 7.3093 C -10.395851 7.3093 -0.00132 4.035863 -0.00132 0.00070625 Z M -0.00132 0.00070625 " transform="matrix(1,0,0,-1,337.47007,174.0593)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-9" x="303.04177" y="200.2878"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-10" x="307.76778" y="200.2878"/>
<use xlink:href="#glyph0-11" x="313.432802" y="200.2878"/>
<use xlink:href="#glyph0-4" x="318.055624" y="200.2878"/>
<use xlink:href="#glyph0-12" x="323.029286" y="200.2878"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-13" x="305.14677" y="212.6704"/>
<use xlink:href="#glyph0-14" x="312.019091" y="212.6704"/>
<use xlink:href="#glyph0-8" x="314.43369" y="212.6704"/>
<use xlink:href="#glyph0-15" x="318.519935" y="212.6704"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.001745 0.0017125 L -0.001745 -24.595944 L -8.86112 -24.595944 L 11.924036 -45.6311 L 32.709193 -24.595944 L 23.845911 -24.595944 L 23.845911 0.0017125 " transform="matrix(1,0,0,-1,302.32987,120.5564)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 233.82343 289.257544 L 162.91718 289.257544 L 162.91718 332.714575 L 233.82343 332.714575 Z M 233.82343 289.257544 " transform="matrix(1,0,0,-1,-15.69843,422.3552)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-9" x="172.08037" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-10" x="176.80638" y="109.0671"/>
<use xlink:href="#glyph0-4" x="182.471402" y="109.0671"/>
<use xlink:href="#glyph0-3" x="187.445063" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-13" x="167.94257" y="121.4497"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="174.856166" y="121.4497"/>
<use xlink:href="#glyph0-3" x="180.025885" y="121.4497"/>
<use xlink:href="#glyph0-16" x="185.845688" y="121.4497"/>
<use xlink:href="#glyph0-2" x="191.531347" y="121.4497"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00018625 -0.00169375 L 0.00018625 -46.462631 C 0.00018625 -50.497787 10.394718 -53.771225 23.218936 -53.771225 C 36.039249 -53.771225 46.43378 -50.497787 46.43378 -46.462631 L 46.43378 -0.00169375 Z M 0.00018625 -0.00169375 " transform="matrix(1,0,0,-1,291.03497,24.3069)"/>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 337.46875 24.308594 C 337.46875 28.34375 327.074219 31.617188 314.253906 31.617188 C 301.429688 31.617188 291.035156 28.34375 291.035156 24.308594 C 291.035156 20.269531 301.429688 16.996094 314.253906 16.996094 C 327.074219 16.996094 337.46875 20.269531 337.46875 24.308594 "/>
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00132 -0.00169375 C -0.00132 -4.03685 -10.395851 -7.310287 -23.216164 -7.310287 C -36.040383 -7.310287 -46.434914 -4.03685 -46.434914 -0.00169375 C -46.434914 4.037369 -36.040383 7.310806 -23.216164 7.310806 C -10.395851 7.310806 -0.00132 4.037369 -0.00132 -0.00169375 Z M -0.00132 -0.00169375 " transform="matrix(1,0,0,-1,337.47007,24.3069)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-13" x="299.52277" y="51.3943"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="306.436366" y="51.3943"/>
<use xlink:href="#glyph0-3" x="311.606085" y="51.3943"/>
<use xlink:href="#glyph0-16" x="317.425888" y="51.3943"/>
<use xlink:href="#glyph0-2" x="323.111547" y="51.3943"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-13" x="308.02547" y="63.7769"/>
<use xlink:href="#glyph0-17" x="314.897791" y="63.7769"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 102.827336 289.257544 L 31.921086 289.257544 L 31.921086 332.714575 L 102.827336 332.714575 Z M 102.827336 289.257544 " transform="matrix(1,0,0,-1,-15.69843,422.3552)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-18" x="39.72147" y="114.2253"/>
<use xlink:href="#glyph0-19" x="45.045971" y="114.2253"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-10" x="48.31703" y="114.2253"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-20" x="53.837588" y="114.2253"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-21" x="58.769975" y="114.2253"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.0011175 -0.00026875 L 24.594868 -0.00026875 L 24.594868 -8.86355 L 54.231586 11.925513 L 24.594868 32.706763 L 24.594868 23.847388 L 0.0011175 23.847388 " transform="matrix(1,0,0,-1,92.49107,123.2927)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-22" x="99.91097" y="114.2136"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-16" x="107.05158" y="114.2136"/>
<use xlink:href="#glyph0-6" x="112.737238" y="114.2136"/>
<use xlink:href="#glyph0-16" x="117.906957" y="114.2136"/>
<use xlink:href="#glyph0-6" x="123.592616" y="114.2136"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.000705 -0.00026875 L 24.594455 -0.00026875 L 24.594455 -8.86355 L 54.231174 11.925513 L 24.594455 32.706763 L 24.594455 23.847388 L 0.000705 23.847388 " transform="matrix(1,0,0,-1,223.10867,123.2927)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-22" x="229.66927" y="114.2136"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-16" x="236.80988" y="114.2136"/>
<use xlink:href="#glyph0-6" x="242.495538" y="114.2136"/>
<use xlink:href="#glyph0-16" x="247.665257" y="114.2136"/>
<use xlink:href="#glyph0-6" x="253.350916" y="114.2136"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 365.401555 289.257544 L 294.499211 289.257544 L 294.499211 332.714575 L 365.401555 332.714575 Z M 365.401555 289.257544 " transform="matrix(1,0,0,-1,-15.69843,422.3552)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-23" x="297.19077" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-19" x="305.755374" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-14" x="309.170897" y="109.0671"/>
<use xlink:href="#glyph0-5" x="311.585496" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="314.939106" y="109.0671"/>
<use xlink:href="#glyph0-24" x="320.108825" y="109.0671"/>
<use xlink:href="#glyph0-5" x="322.29641" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-10" x="325.65002" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-23" x="301.24607" y="121.4497"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-25" x="309.387603" y="121.4497"/>
<use xlink:href="#glyph0-26" x="315.702709" y="121.4497"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-27" x="321.28518" y="121.4497"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00127 -0.00124375 L 24.596386 -0.00124375 L 24.596386 -8.864525 L 54.229199 11.920631 L 24.596386 32.709694 L 24.596386 23.846413 L -0.00127 23.846413 " transform="matrix(1,0,0,-1,355.06377,124.1511)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-22" x="361.62387" y="115.074"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-16" x="368.76448" y="115.074"/>
<use xlink:href="#glyph0-6" x="374.450138" y="115.074"/>
<use xlink:href="#glyph0-16" x="379.619857" y="115.074"/>
<use xlink:href="#glyph0-6" x="385.305516" y="115.074"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 497.358586 288.398169 L 426.452336 288.398169 L 426.452336 331.8552 L 497.358586 331.8552 Z M 497.358586 288.398169 " transform="matrix(1,0,0,-1,-15.69843,422.3552)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-7" x="436.18827" y="109.0671"/>
<use xlink:href="#glyph0-4" x="441.306395" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-28" x="446.197506" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="451.057661" y="109.0671"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-13" x="431.48287" y="121.4497"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-6" x="438.396466" y="121.4497"/>
<use xlink:href="#glyph0-3" x="443.566185" y="121.4497"/>
<use xlink:href="#glyph0-16" x="449.385988" y="121.4497"/>
<use xlink:href="#glyph0-2" x="455.071647" y="121.4497"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00016125 -0.00024375 L 24.593911 -0.00024375 L 24.593911 -8.863525 L 54.23063 11.921631 L 24.593911 32.706788 L 24.593911 23.843506 L 0.00016125 23.843506 " transform="matrix(1,0,0,-1,487.39437,124.1521)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-22" x="493.95387" y="115.0749"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-16" x="501.09448" y="115.0749"/>
<use xlink:href="#glyph0-6" x="506.780138" y="115.0749"/>
<use xlink:href="#glyph0-16" x="511.949857" y="115.0749"/>
<use xlink:href="#glyph0-6" x="517.635516" y="115.0749"/>
</g>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 59.15625 39.414062 L 59.15625 35.292969 C 62.945312 32.382812 65.453125 27.332031 65.453125 21.585938 C 65.453125 12.574219 59.285156 5.261719 51.675781 5.261719 C 44.066406 5.261719 37.894531 12.574219 37.894531 21.585938 C 37.894531 27.332031 40.40625 32.382812 44.191406 35.292969 L 44.191406 39.414062 C 34.898438 41.503906 28.203125 47.378906 28.203125 54.304688 L 75.148438 54.304688 C 75.148438 47.378906 68.449219 41.503906 59.15625 39.414062 "/>
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00132 0.0016375 L -0.00132 4.122731 C 3.787743 7.032888 6.295555 12.083669 6.295555 17.829763 C 6.295555 26.841481 0.127586 34.153981 -7.481789 34.153981 C -15.091164 34.153981 -21.263039 26.841481 -21.263039 17.829763 C -21.263039 12.083669 -18.75132 7.032888 -14.966164 4.122731 L -14.966164 0.0016375 C -24.259132 -2.088206 -30.954445 -7.963206 -30.954445 -14.888987 L 15.990868 -14.888987 C 15.990868 -7.963206 9.291649 -2.088206 -0.00132 0.0016375 Z M -0.00132 0.0016375 " transform="matrix(1,0,0,-1,59.15757,39.4157)"/>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 96.824219 186.9375 C 96.824219 183.21875 95.191406 179.875 92.582031 177.507812 C 92.914062 176.382812 93.101562 175.199219 93.101562 173.972656 C 93.101562 166.808594 87.074219 161.003906 79.636719 161.003906 C 74.6875 161.003906 70.371094 163.585938 68.035156 167.417969 C 65.695312 163.585938 61.378906 161.003906 56.429688 161.003906 C 50.429688 161.003906 45.347656 164.789062 43.609375 170.011719 C 43.394531 170.003906 43.183594 169.980469 42.96875 169.980469 C 40.6875 169.980469 38.542969 170.53125 36.660156 171.496094 C 34.191406 168.738281 30.546875 166.988281 26.472656 166.988281 C 19.035156 166.988281 13.007812 172.796875 13.007812 179.957031 C 13.007812 182.835938 13.992188 185.488281 15.644531 187.640625 C 10.34375 189.382812 6.523438 194.210938 6.523438 199.902344 C 6.523438 207.066406 12.550781 212.871094 19.988281 212.871094 C 20.203125 212.871094 20.417969 212.847656 20.632812 212.839844 C 22.367188 218.0625 27.453125 221.847656 33.449219 221.847656 C 37.1875 221.847656 40.566406 220.382812 43.007812 218.015625 C 44.46875 223.652344 49.746094 227.832031 56.050781 227.832031 C 62.699219 227.832031 68.210938 223.1875 69.304688 217.085938 C 71.296875 218.203125 73.601562 218.855469 76.070312 218.855469 C 83.507812 218.855469 89.535156 213.050781 89.535156 205.890625 C 89.535156 203.453125 88.824219 201.183594 87.609375 199.234375 C 92.960938 197.519531 96.824219 192.667969 96.824219 186.9375 "/>
<g clip-path="url(#clip2)" clip-rule="nonzero">
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00175125 0.0007 C -0.00175125 3.71945 -1.634564 7.0632 -4.243939 9.430388 C -3.911907 10.555388 -3.724407 11.738981 -3.724407 12.965544 C -3.724407 20.129606 -9.751751 25.934294 -17.189251 25.934294 C -22.13847 25.934294 -26.454876 23.352263 -28.790814 19.520231 C -31.130657 23.352263 -35.447064 25.934294 -40.396282 25.934294 C -46.396282 25.934294 -51.478314 22.149138 -53.216595 16.926481 C -53.431439 16.934294 -53.642376 16.957731 -53.85722 16.957731 C -56.13847 16.957731 -58.283001 16.40695 -60.165814 15.442106 C -62.634564 18.199919 -66.279095 19.949919 -70.353314 19.949919 C -77.790814 19.949919 -83.818157 14.141325 -83.818157 6.981169 C -83.818157 4.102263 -82.833782 1.449919 -81.181439 -0.702425 C -86.48222 -2.444612 -90.302532 -7.272737 -90.302532 -12.964144 C -90.302532 -20.128206 -84.275189 -25.932894 -76.837689 -25.932894 C -76.622845 -25.932894 -76.408001 -25.909456 -76.193157 -25.901644 C -74.458782 -31.1243 -69.372845 -34.909456 -63.376751 -34.909456 C -59.63847 -34.909456 -56.259564 -33.444612 -53.818157 -31.077425 C -52.35722 -36.714144 -47.079876 -40.893831 -40.775189 -40.893831 C -34.126751 -40.893831 -28.615032 -36.2493 -27.521282 -30.147737 C -25.529095 -31.264925 -23.224407 -31.917269 -20.755657 -31.917269 C -13.318157 -31.917269 -7.290814 -26.112581 -7.290814 -18.952425 C -7.290814 -16.514925 -8.001751 -14.245394 -9.216595 -12.296175 C -3.865032 -10.581331 -0.00175125 -5.729769 -0.00175125 0.0007 Z M -0.00175125 0.0007 " transform="matrix(1,0,0,-1,96.82597,186.9382)"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-1" x="44.59157" y="190.9714"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-2" x="49.082077" y="190.9714"/>
<use xlink:href="#glyph1-3" x="54.168263" y="190.9714"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-4" x="36.35286" y="201.9686"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-5" x="38.634771" y="201.9686"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-6" x="43.6843" y="201.9686"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-3" x="46.662697" y="201.9686"/>
<use xlink:href="#glyph1-7" x="51.254012" y="201.9686"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-5" x="54.286479" y="201.9686"/>
<use xlink:href="#glyph1-3" x="59.372665" y="201.9686"/>
<use xlink:href="#glyph1-6" x="63.963979" y="201.9686"/>
</g>
<path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 752.960938 103.886719 C 752.960938 100.167969 751.324219 96.824219 748.71875 94.457031 C 749.050781 93.332031 749.234375 92.148438 749.234375 90.921875 C 749.234375 83.761719 743.210938 77.957031 735.773438 77.957031 C 730.824219 77.957031 726.507812 80.535156 724.167969 84.367188 C 721.832031 80.535156 717.515625 77.957031 712.566406 77.957031 C 706.5625 77.957031 701.484375 81.738281 699.746094 86.964844 C 699.53125 86.953125 699.320312 86.933594 699.101562 86.933594 C 696.820312 86.933594 694.675781 87.480469 692.792969 88.445312 C 690.324219 85.691406 686.679688 83.941406 682.605469 83.941406 C 675.167969 83.941406 669.140625 89.746094 669.140625 96.90625 C 669.140625 99.785156 670.128906 102.4375 671.777344 104.59375 C 666.476562 106.332031 662.660156 111.160156 662.660156 116.855469 C 662.660156 124.015625 668.683594 129.820312 676.121094 129.820312 C 676.339844 129.820312 676.550781 129.800781 676.769531 129.789062 C 678.503906 135.015625 683.585938 138.800781 689.585938 138.800781 C 693.324219 138.800781 696.699219 137.332031 699.140625 134.96875 C 700.601562 140.605469 705.878906 144.78125 712.1875 144.78125 C 718.835938 144.78125 724.347656 140.136719 725.441406 134.035156 C 727.429688 135.152344 729.734375 135.804688 732.207031 135.804688 C 739.644531 135.804688 745.671875 130 745.671875 122.839844 C 745.671875 120.402344 744.960938 118.132812 743.746094 116.183594 C 749.097656 114.46875 752.960938 109.617188 752.960938 103.886719 "/>
<g clip-path="url(#clip3)" clip-rule="nonzero">
<path style="fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.0001675 0.00118125 C 0.0001675 3.719931 -1.636551 7.063681 -4.24202 9.430869 C -3.909989 10.555869 -3.726395 11.739463 -3.726395 12.966025 C -3.726395 20.126181 -9.749832 25.930869 -17.187332 25.930869 C -22.136551 25.930869 -26.452957 23.352744 -28.792801 19.520713 C -31.128739 23.352744 -35.445145 25.930869 -40.394364 25.930869 C -46.39827 25.930869 -51.476395 22.149619 -53.214676 16.923056 C -53.42952 16.934775 -53.640457 16.954306 -53.859207 16.954306 C -56.140457 16.954306 -58.284989 16.407431 -60.167801 15.442588 C -62.636551 18.196494 -66.281082 19.946494 -70.355301 19.946494 C -77.792801 19.946494 -83.820145 14.141806 -83.820145 6.98165 C -83.820145 4.102744 -82.831864 1.4504 -81.183426 -0.70585 C -86.484207 -2.444131 -90.300614 -7.272256 -90.300614 -12.967569 C -90.300614 -20.127725 -84.277176 -25.932412 -76.839676 -25.932412 C -76.620926 -25.932412 -76.409989 -25.912881 -76.191239 -25.901162 C -74.456864 -31.127725 -69.374832 -34.912881 -63.374832 -34.912881 C -59.636551 -34.912881 -56.261551 -33.444131 -53.820145 -31.08085 C -52.359207 -36.717569 -47.081864 -40.89335 -40.77327 -40.89335 C -34.124832 -40.89335 -28.613114 -36.248819 -27.519364 -30.147256 C -25.531082 -31.264444 -23.226395 -31.916787 -20.753739 -31.916787 C -13.316239 -31.916787 -7.288895 -26.1121 -7.288895 -18.951944 C -7.288895 -16.514444 -7.999832 -14.244912 -9.214676 -12.295694 C -3.863114 -10.58085 0.0001675 -5.729287 0.0001675 0.00118125 Z M 0.0001675 0.00118125 " transform="matrix(1,0,0,-1,752.96077,103.8879)"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-8" x="686.41197" y="107.9216"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-9" x="693.835053" y="107.9216"/>
<use xlink:href="#glyph1-7" x="698.866254" y="107.9216"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-3" x="701.771337" y="107.9216"/>
<use xlink:href="#glyph1-10" x="706.362651" y="107.9216"/>
<use xlink:href="#glyph1-11" x="708.305483" y="107.9216"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-6" x="712.79599" y="107.9216"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-3" x="715.774387" y="107.9216"/>
<use xlink:href="#glyph1-12" x="720.365702" y="107.9216"/>
<use xlink:href="#glyph1-13" x="725.580188" y="107.9216"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-14" x="684.62497" y="118.9188"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-15" x="687.126824" y="118.9188"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph1-13" x="692.68039" y="118.9188"/>
<use xlink:href="#glyph1-10" x="696.309453" y="118.9188"/>
<use xlink:href="#glyph1-16" x="698.252284" y="118.9188"/>
<use xlink:href="#glyph1-3" x="704.282394" y="118.9188"/>
<use xlink:href="#glyph1-3" x="708.873708" y="118.9188"/>
<use xlink:href="#glyph1-17" x="713.465022" y="118.9188"/>
<use xlink:href="#glyph1-3" x="718.633687" y="118.9188"/>
<use xlink:href="#glyph1-17" x="723.225002" y="118.9188"/>
<use xlink:href="#glyph1-18" x="728.393667" y="118.9188"/>
</g>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00082625 0.00019375 L -0.00082625 14.367381 L 4.323393 14.367381 L -5.813326 31.679881 L -15.950045 14.367381 L -11.629732 14.367381 L -11.629732 0.00019375 " transform="matrix(1,0,0,-1,44.15317,87.8166)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M -0.00127625 -0.00075 L -0.00127625 -14.367937 L -4.321589 -14.367937 L 5.81513 -31.680437 L 15.951849 -14.367937 L 11.62763 -14.367937 L 11.62763 -0.00075 " transform="matrix(1,0,0,-1,55.99737,57.968)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.0016175 0.00165 L 0.0016175 14.368838 L 4.32193 14.368838 L -5.814789 31.681338 L -15.951507 14.368838 L -11.627289 14.368838 L -11.627289 0.00165 " transform="matrix(1,0,0,-1,42.55307,167.4704)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.0011675 0.00170625 L 0.0011675 -14.369387 L -4.323051 -14.369387 L 5.813668 -31.681887 L 15.950386 -14.369387 L 11.626168 -14.369387 L 11.626168 0.00170625 " transform="matrix(1,0,0,-1,54.39727,137.6228)"/>
<path style="fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(13.729858%,12.159729%,12.548828%);stroke-opacity:1;stroke-miterlimit:10;" d="M 0.00118 -0.00024375 L 24.59493 -0.00024375 L 24.59493 -8.863525 L 54.231649 11.921631 L 24.59493 32.706788 L 24.59493 23.843506 L 0.00118 23.843506 " transform="matrix(1,0,0,-1,619.21757,124.1521)"/>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-22" x="625.77717" y="115.0744"/>
</g>
<g style="fill:rgb(13.729858%,12.159729%,12.548828%);fill-opacity:1;">
<use xlink:href="#glyph0-16" x="632.91778" y="115.0744"/>
<use xlink:href="#glyph0-6" x="638.603438" y="115.0744"/>
<use xlink:href="#glyph0-16" x="643.773157" y="115.0744"/>
<use xlink:href="#glyph0-6" x="649.458816" y="115.0744"/>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

2
benchmarks/run-benchmarks.py
View File

@ -194,7 +194,7 @@ if __name__ == '__main__':
args = arg_parser.parse_args(args=sys.argv[1:]) args = arg_parser.parse_args(args=sys.argv[1:])
if args.trace: if args.trace:
loglevel = warcprox.TRACE loglevel = logging.TRACE
elif args.verbose: elif args.verbose:
loglevel = logging.DEBUG loglevel = logging.DEBUG
else: else:

10
setup.py
View File

@ -25,13 +25,13 @@ import setuptools
deps = [ deps = [
'certauth==1.1.6', 'certauth==1.1.6',
'warctools', 'warctools>=4.10.0',
'urlcanon>=0.1.dev16', 'urlcanon>=0.1.dev16',
'doublethink>=0.2.0.dev87', 'doublethink>=0.2.0.dev87',
'urllib3', 'urllib3>=1.23',
'requests>=2.0.1', 'requests>=2.0.1',
'PySocks', 'PySocks>=1.6.8',
'cryptography!=2.1.1', # 2.1.1 installation is failing on ubuntu 'cryptography>=2.3',
] ]
try: try:
import concurrent.futures import concurrent.futures
@ -40,7 +40,7 @@ except:
setuptools.setup( setuptools.setup(
name='warcprox', name='warcprox',
version='2.4b3.dev180', version='2.4b3.dev184',
description='WARC writing MITM HTTP/S proxy', description='WARC writing MITM HTTP/S proxy',
url='https://github.com/internetarchive/warcprox', url='https://github.com/internetarchive/warcprox',
author='Noah Levitt', author='Noah Levitt',

2
tests/test_ensure_rethinkdb_tables.py
View File

@ -30,7 +30,7 @@ import logging
import sys import sys
logging.basicConfig( logging.basicConfig(
stream=sys.stdout, level=warcprox.TRACE, stream=sys.stdout, level=logging.TRACE,
format='%(asctime)s %(process)d %(levelname)s %(threadName)s ' format='%(asctime)s %(process)d %(levelname)s %(threadName)s '
'%(name)s.%(funcName)s(%(filename)s:%(lineno)d) %(message)s') '%(name)s.%(funcName)s(%(filename)s:%(lineno)d) %(message)s')

36
tests/test_warcprox.py
View File

@ -90,8 +90,7 @@ def _send(self, data):
# http_client.HTTPConnection.send = _send # http_client.HTTPConnection.send = _send
logging.basicConfig( logging.basicConfig(
# stream=sys.stdout, level=logging.DEBUG, # level=warcprox.TRACE, stream=sys.stdout, level=logging.TRACE,
stream=sys.stdout, level=warcprox.TRACE,
format='%(asctime)s %(process)d %(levelname)s %(threadName)s ' format='%(asctime)s %(process)d %(levelname)s %(threadName)s '
'%(name)s.%(funcName)s(%(filename)s:%(lineno)d) %(message)s') '%(name)s.%(funcName)s(%(filename)s:%(lineno)d) %(message)s')
logging.getLogger("requests.packages.urllib3").setLevel(logging.WARN) logging.getLogger("requests.packages.urllib3").setLevel(logging.WARN)
@ -1718,8 +1717,14 @@ def test_slash_in_warc_prefix(warcprox_, http_daemon, archiving_proxies):
def test_crawl_log(warcprox_, http_daemon, archiving_proxies): def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
urls_before = warcprox_.proxy.running_stats.urls urls_before = warcprox_.proxy.running_stats.urls
hostname = socket.gethostname().split('.', 1)[0]
port = warcprox_.proxy.server_port
default_crawl_log_path = os.path.join(
warcprox_.options.crawl_log_dir,
'crawl-%s-%s.log' % (hostname, port))
try: try:
os.unlink(os.path.join(warcprox_.options.crawl_log_dir, 'crawl.log')) os.unlink(default_crawl_log_path)
except: except:
pass pass
@ -1740,14 +1745,14 @@ def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
# wait for postfetch chain # wait for postfetch chain
wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 2) wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 2)
file = os.path.join(warcprox_.options.crawl_log_dir, 'test_crawl_log_1.log') file = os.path.join(
warcprox_.options.crawl_log_dir,
'test_crawl_log_1-%s-%s.log' % (hostname, port))
assert os.path.exists(file) assert os.path.exists(file)
assert os.stat(file).st_size > 0 assert os.stat(file).st_size > 0
assert os.path.exists(os.path.join( assert os.path.exists(default_crawl_log_path)
warcprox_.options.crawl_log_dir, 'crawl.log'))
crawl_log = open(os.path.join( crawl_log = open(default_crawl_log_path, 'rb').read()
warcprox_.options.crawl_log_dir, 'crawl.log'), 'rb').read()
# tests will fail in year 3000 :) # tests will fail in year 3000 :)
assert re.match(b'\A2[^\n]+\n\Z', crawl_log) assert re.match(b'\A2[^\n]+\n\Z', crawl_log)
assert crawl_log[24:31] == b' 200 ' assert crawl_log[24:31] == b' 200 '
@ -1768,8 +1773,7 @@ def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
'contentSize', 'warcFilename', 'warcFileOffset'} 'contentSize', 'warcFilename', 'warcFileOffset'}
assert extra_info['contentSize'] == 145 assert extra_info['contentSize'] == 145
crawl_log_1 = open(os.path.join( crawl_log_1 = open(file, 'rb').read()
warcprox_.options.crawl_log_dir, 'test_crawl_log_1.log'), 'rb').read()
assert re.match(b'\A2[^\n]+\n\Z', crawl_log_1) assert re.match(b'\A2[^\n]+\n\Z', crawl_log_1)
assert crawl_log_1[24:31] == b' 200 ' assert crawl_log_1[24:31] == b' 200 '
assert crawl_log_1[31:42] == b' 54 ' assert crawl_log_1[31:42] == b' 54 '
@ -1800,7 +1804,9 @@ def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
# wait for postfetch chain # wait for postfetch chain
wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 3) wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 3)
file = os.path.join(warcprox_.options.crawl_log_dir, 'test_crawl_log_2.log') file = os.path.join(
warcprox_.options.crawl_log_dir,
'test_crawl_log_2-%s-%s.log' % (hostname, port))
assert os.path.exists(file) assert os.path.exists(file)
assert os.stat(file).st_size > 0 assert os.stat(file).st_size > 0
@ -1833,7 +1839,9 @@ def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
# wait for postfetch chain # wait for postfetch chain
wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 4) wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 4)
file = os.path.join(warcprox_.options.crawl_log_dir, 'test_crawl_log_3.log') file = os.path.join(
warcprox_.options.crawl_log_dir,
'test_crawl_log_3-%s-%s.log' % (hostname, port))
assert os.path.exists(file) assert os.path.exists(file)
crawl_log_3 = open(file, 'rb').read() crawl_log_3 = open(file, 'rb').read()
@ -1871,7 +1879,9 @@ def test_crawl_log(warcprox_, http_daemon, archiving_proxies):
# wait for postfetch chain # wait for postfetch chain
wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 5) wait(lambda: warcprox_.proxy.running_stats.urls - urls_before == 5)
file = os.path.join(warcprox_.options.crawl_log_dir, 'test_crawl_log_4.log') file = os.path.join(
warcprox_.options.crawl_log_dir,
'test_crawl_log_4-%s-%s.log' % (hostname, port))
assert os.path.exists(file) assert os.path.exists(file)
crawl_log_4 = open(file, 'rb').read() crawl_log_4 = open(file, 'rb').read()

44
tests/test_writer.py
View File

@ -22,7 +22,7 @@ USA.
import os import os
import fcntl import fcntl
from multiprocessing import Process, Queue from multiprocessing import Process, Queue
from datetime import datetime from datetime import datetime, timedelta
import pytest import pytest
import re import re
from warcprox.mitmproxy import ProxyingRecorder from warcprox.mitmproxy import ProxyingRecorder
@ -34,6 +34,7 @@ import warcprox
import io import io
import tempfile import tempfile
import logging import logging
import hashlib
def lock_file(queue, filename): def lock_file(queue, filename):
"""Try to lock file and return 1 if successful, else return 0. """Try to lock file and return 1 if successful, else return 0.
@ -58,7 +59,7 @@ def test_warc_writer_locking(tmpdir):
url='http://example.com', content_type='text/plain', status=200, url='http://example.com', content_type='text/plain', status=200,
client_ip='127.0.0.2', request_data=b'abc', client_ip='127.0.0.2', request_data=b'abc',
response_recorder=recorder, remote_ip='127.0.0.3', response_recorder=recorder, remote_ip='127.0.0.3',
timestamp=datetime.utcnow()) timestamp=datetime.utcnow(), payload_digest=hashlib.sha1())
dirname = os.path.dirname(str(tmpdir.mkdir('test-warc-writer'))) dirname = os.path.dirname(str(tmpdir.mkdir('test-warc-writer')))
wwriter = WarcWriter(Options( wwriter = WarcWriter(Options(
@ -129,7 +130,7 @@ def test_special_dont_write_prefix():
wwt.join() wwt.join()
wwt = warcprox.writerthread.WarcWriterProcessor( wwt = warcprox.writerthread.WarcWriterProcessor(
Options(writer_threads=1)) Options(writer_threads=1, blackout_period=60, prefix='foo'))
wwt.inq = warcprox.TimestampedQueue(maxsize=1) wwt.inq = warcprox.TimestampedQueue(maxsize=1)
wwt.outq = warcprox.TimestampedQueue(maxsize=1) wwt.outq = warcprox.TimestampedQueue(maxsize=1)
try: try:
@ -158,6 +159,41 @@ def test_special_dont_write_prefix():
recorded_url = wwt.outq.get(timeout=10) recorded_url = wwt.outq.get(timeout=10)
assert not recorded_url.warc_records assert not recorded_url.warc_records
assert wwt.outq.empty() assert wwt.outq.empty()
# test blackout_period option. Write first revisit record because
# its outside the blackout_period (60). Do not write the second
# because its inside the blackout_period.
recorder = ProxyingRecorder(io.BytesIO(b'test1'), None)
recorder.read()
old = datetime.utcnow() - timedelta(0, 3600)
ru = RecordedUrl(
url='http://example.com/dup',
content_type='text/plain',
status=200, client_ip='127.0.0.2', request_data=b'abc',
response_recorder=recorder, remote_ip='127.0.0.3',
timestamp=datetime.utcnow(),
payload_digest=recorder.block_digest)
ru.dedup_info = dict(id=b'1', url=b'http://example.com/dup',
date=old.strftime('%Y-%m-%dT%H:%M:%SZ').encode('utf-8'))
wwt.inq.put(ru)
recorded_url = wwt.outq.get(timeout=10)
recorder = ProxyingRecorder(io.BytesIO(b'test2'), None)
recorder.read()
recent = datetime.utcnow() - timedelta(0, 5)
ru = RecordedUrl(
url='http://example.com/dup', content_type='text/plain',
status=200, client_ip='127.0.0.2', request_data=b'abc',
response_recorder=recorder, remote_ip='127.0.0.3',
timestamp=datetime.utcnow(),
payload_digest=recorder.block_digest)
ru.dedup_info = dict(id=b'2', url=b'http://example.com/dup',
date=recent.strftime('%Y-%m-%dT%H:%M:%SZ').encode('utf-8'))
wwt.inq.put(ru)
assert recorded_url.warc_records
recorded_url = wwt.outq.get(timeout=10)
assert not recorded_url.warc_records
assert wwt.outq.empty()
finally: finally:
wwt.stop.set() wwt.stop.set()
wwt.join() wwt.join()
@ -212,7 +248,7 @@ def test_warc_writer_filename(tmpdir):
url='http://example.com', content_type='text/plain', status=200, url='http://example.com', content_type='text/plain', status=200,
client_ip='127.0.0.2', request_data=b'abc', client_ip='127.0.0.2', request_data=b'abc',
response_recorder=recorder, remote_ip='127.0.0.3', response_recorder=recorder, remote_ip='127.0.0.3',
timestamp=datetime.utcnow()) timestamp=datetime.utcnow(), payload_digest=hashlib.sha1())
dirname = os.path.dirname(str(tmpdir.mkdir('test-warc-writer'))) dirname = os.path.dirname(str(tmpdir.mkdir('test-warc-writer')))
wwriter = WarcWriter(Options(directory=dirname, prefix='foo', wwriter = WarcWriter(Options(directory=dirname, prefix='foo',

16
warcprox/__init__.py
View File

@ -266,21 +266,21 @@ def timestamp14():
return '{:%Y%m%d%H%M%S}'.format(now) return '{:%Y%m%d%H%M%S}'.format(now)
# monkey-patch log levels TRACE and NOTICE # monkey-patch log levels TRACE and NOTICE
TRACE = 5 logging.TRACE = (logging.NOTSET + logging.DEBUG) // 2
def _logger_trace(self, msg, *args, **kwargs): def _logger_trace(self, msg, *args, **kwargs):
if self.isEnabledFor(TRACE): if self.isEnabledFor(logging.TRACE):
self._log(TRACE, msg, args, **kwargs) self._log(logging.TRACE, msg, args, **kwargs)
logging.Logger.trace = _logger_trace logging.Logger.trace = _logger_trace
logging.trace = logging.root.trace logging.trace = logging.root.trace
logging.addLevelName(TRACE, 'TRACE') logging.addLevelName(logging.TRACE, 'TRACE')
NOTICE = (logging.INFO + logging.WARN) // 2 logging.NOTICE = (logging.INFO + logging.WARN) // 2
def _logger_notice(self, msg, *args, **kwargs): def _logger_notice(self, msg, *args, **kwargs):
if self.isEnabledFor(NOTICE): if self.isEnabledFor(logging.NOTICE):
self._log(NOTICE, msg, args, **kwargs) self._log(logging.NOTICE, msg, args, **kwargs)
logging.Logger.notice = _logger_notice logging.Logger.notice = _logger_notice
logging.notice = logging.root.notice logging.notice = logging.root.notice
logging.addLevelName(NOTICE, 'NOTICE') logging.addLevelName(logging.NOTICE, 'NOTICE')
import warcprox.controller as controller import warcprox.controller as controller
import warcprox.playback as playback import warcprox.playback as playback

4
warcprox/controller.py
View File

@ -299,9 +299,7 @@ class WarcproxController(object):
status_info.update(self.proxy.status()) status_info.update(self.proxy.status())
self.status_info = self.service_registry.heartbeat(status_info) self.status_info = self.service_registry.heartbeat(status_info)
self.logger.log( self.logger.trace('status in service registry: %s', self.status_info)
warcprox.TRACE, "status in service registry: %s",
self.status_info)
def start(self): def start(self):
with self._start_stop_lock: with self._start_stop_lock:

15
warcprox/crawl_log.py
View File

@ -24,11 +24,15 @@ import datetime
import json import json
import os import os
import warcprox import warcprox
import socket
class CrawlLogger(object): class CrawlLogger(object):
def __init__(self, dir_, options=warcprox.Options()): def __init__(self, dir_, options=warcprox.Options()):
self.dir = dir_ self.dir = dir_
self.options = options self.options = options
self.hostname = socket.gethostname().split('.', 1)[0]
def start(self):
if not os.path.exists(self.dir): if not os.path.exists(self.dir):
logging.info('creating directory %r', self.dir) logging.info('creating directory %r', self.dir)
os.mkdir(self.dir) os.mkdir(self.dir)
@ -49,7 +53,7 @@ class CrawlLogger(object):
self.options.base32) self.options.base32)
else: else:
# WARCPROX_WRITE_RECORD request # WARCPROX_WRITE_RECORD request
content_length = len(recorded_url.request_data) content_length = int(records[0].get_header(b'Content-Length'))
payload_digest = records[0].get_header(b'WARC-Payload-Digest') payload_digest = records[0].get_header(b'WARC-Payload-Digest')
fields = [ fields = [
'{:%Y-%m-%dT%H:%M:%S}.{:03d}Z'.format(now, now.microsecond//1000), '{:%Y-%m-%dT%H:%M:%S}.{:03d}Z'.format(now, now.microsecond//1000),
@ -77,12 +81,11 @@ class CrawlLogger(object):
pass pass
line = b' '.join(fields) + b'\n' line = b' '.join(fields) + b'\n'
if 'warc-prefix' in recorded_url.warcprox_meta: prefix = recorded_url.warcprox_meta.get('warc-prefix', 'crawl')
filename = '%s.log' % recorded_url.warcprox_meta['warc-prefix'] filename = '%s-%s-%s.log' % (
else: prefix, self.hostname, self.options.server_port)
filename = 'crawl.log'
crawl_log_path = os.path.join(self.dir, filename) crawl_log_path = os.path.join(self.dir, filename)
with open(crawl_log_path, 'ab') as f: with open(crawl_log_path, 'ab') as f:
f.write(line) f.write(line)

110
warcprox/main.py
View File

@ -60,10 +60,23 @@ class BetterArgumentDefaultsHelpFormatter(
else: else:
return argparse.ArgumentDefaultsHelpFormatter._get_help_string(self, action) return argparse.ArgumentDefaultsHelpFormatter._get_help_string(self, action)
def _build_arg_parser(prog='warcprox'): def _build_arg_parser(prog='warcprox', show_hidden=False):
if show_hidden:
def suppress(msg):
return msg
else:
def suppress(msg):
return argparse.SUPPRESS
arg_parser = argparse.ArgumentParser(prog=prog, arg_parser = argparse.ArgumentParser(prog=prog,
description='warcprox - WARC writing MITM HTTP/S proxy', description='warcprox - WARC writing MITM HTTP/S proxy',
formatter_class=BetterArgumentDefaultsHelpFormatter) formatter_class=BetterArgumentDefaultsHelpFormatter)
hidden = arg_parser.add_argument_group('hidden options')
arg_parser.add_argument(
'--help-hidden', action='help', default=argparse.SUPPRESS,
help='show help message, including help on hidden options, and exit')
arg_parser.add_argument('-p', '--port', dest='port', default='8000', arg_parser.add_argument('-p', '--port', dest='port', default='8000',
type=int, help='port to listen on') type=int, help='port to listen on')
arg_parser.add_argument('-b', '--address', dest='address', arg_parser.add_argument('-b', '--address', dest='address',
@ -81,8 +94,12 @@ def _build_arg_parser(prog='warcprox'):
help='define custom WARC filename with variables {prefix}, {timestamp14}, {timestamp17}, {serialno}, {randomtoken}, {hostname}, {shorthostname}') help='define custom WARC filename with variables {prefix}, {timestamp14}, {timestamp17}, {serialno}, {randomtoken}, {hostname}, {shorthostname}')
arg_parser.add_argument('-z', '--gzip', dest='gzip', action='store_true', arg_parser.add_argument('-z', '--gzip', dest='gzip', action='store_true',
help='write gzip-compressed warc records') help='write gzip-compressed warc records')
arg_parser.add_argument('--no-warc-open-suffix', dest='no_warc_open_suffix', hidden.add_argument(
default=False, action='store_true', help=argparse.SUPPRESS) '--no-warc-open-suffix', dest='no_warc_open_suffix',
default=False, action='store_true',
help=suppress(
'do not name warc files with suffix ".open" while writing to '
'them, but lock them with lockf(3) intead'))
# not mentioned in --help: special value for '-' for --prefix means don't # not mentioned in --help: special value for '-' for --prefix means don't
# archive the capture, unless prefix set in warcprox-meta header # archive the capture, unless prefix set in warcprox-meta header
arg_parser.add_argument( arg_parser.add_argument(
@ -146,40 +163,60 @@ def _build_arg_parser(prog='warcprox'):
'rethinkdb service registry table url; if provided, warcprox ' 'rethinkdb service registry table url; if provided, warcprox '
'will create and heartbeat entry for itself')) 'will create and heartbeat entry for itself'))
# optional cookie values to pass to CDX Server; e.g. "cookie1=val1;cookie2=val2" # optional cookie values to pass to CDX Server; e.g. "cookie1=val1;cookie2=val2"
arg_parser.add_argument('--cdxserver-dedup-cookies', dest='cdxserver_dedup_cookies', hidden.add_argument(
help=argparse.SUPPRESS) '--cdxserver-dedup-cookies', dest='cdxserver_dedup_cookies',
help=suppress(
'value of Cookie header to include in requests to the cdx '
'server, when using --cdxserver-dedup'))
arg_parser.add_argument('--dedup-min-text-size', dest='dedup_min_text_size', arg_parser.add_argument('--dedup-min-text-size', dest='dedup_min_text_size',
type=int, default=0, type=int, default=0,
help=('try to dedup text resources with payload size over this limit in bytes')) help=('try to dedup text resources with payload size over this limit in bytes'))
arg_parser.add_argument('--dedup-min-binary-size', dest='dedup_min_binary_size', arg_parser.add_argument('--dedup-min-binary-size', dest='dedup_min_binary_size',
type=int, default=0, help=( type=int, default=0, help=(
'try to dedup binary resources with payload size over this limit in bytes')) 'try to dedup binary resources with payload size over this limit in bytes'))
# optionally, dedup request only when `dedup-bucket` is available in hidden.add_argument(
# Warcprox-Meta HTTP header. By default, we dedup all requests. '--dedup-only-with-bucket', dest='dedup_only_with_bucket',
arg_parser.add_argument('--dedup-only-with-bucket', dest='dedup_only_with_bucket', action='store_true', default=False, help=suppress(
action='store_true', default=False, help=argparse.SUPPRESS) 'only deduplicate captures if "dedup-bucket" is set in '
arg_parser.add_argument('--queue-size', dest='queue_size', type=int, 'the Warcprox-Meta request header'))
default=500, help=argparse.SUPPRESS) arg_parser.add_argument('--blackout-period', dest='blackout_period',
arg_parser.add_argument('--max-threads', dest='max_threads', type=int, type=int, default=0,
help=argparse.SUPPRESS) help='skip writing a revisit record if its too close to the original capture')
arg_parser.add_argument('--profile', action='store_true', default=False, hidden.add_argument(
help=argparse.SUPPRESS) '--queue-size', dest='queue_size', type=int, default=500,
arg_parser.add_argument( help=suppress(
'--writer-threads', dest='writer_threads', type=int, default=None, 'maximum number of urls that can be queued at each '
help=argparse.SUPPRESS) 'step of the processing chain (see the section on warcprox '
'architecture in README.rst)'))
hidden.add_argument(
'--max-threads', dest='max_threads', type=int, default=100,
help=suppress('maximum number of http worker threads'))
hidden.add_argument(
'--profile', action='store_true', default=False,
help=suppress(
'turn on performance profiling; summary statistics are dumped '
'every 10 minutes and at shutdown'))
hidden.add_argument(
'--writer-threads', dest='writer_threads', type=int, default=1,
help=suppress(
'number of warc writer threads; caution, see '
'https://github.com/internetarchive/warcprox/issues/101'))
arg_parser.add_argument( arg_parser.add_argument(
'--onion-tor-socks-proxy', dest='onion_tor_socks_proxy', '--onion-tor-socks-proxy', dest='onion_tor_socks_proxy',
default=None, help=( default=None, help=(
'host:port of tor socks proxy, used only to connect to ' 'host:port of tor socks proxy, used only to connect to '
'.onion sites')) '.onion sites'))
# Configurable connection socket timeout, default is 60 sec. hidden.add_argument(
arg_parser.add_argument( '--socket-timeout', dest='socket_timeout', type=float, default=60,
'--socket-timeout', dest='socket_timeout', type=float, help=suppress(
default=None, help=argparse.SUPPRESS) 'socket timeout, used for proxy client connection and for '
'connection to remote server'))
# Increasing this value increases memory usage but reduces /tmp disk I/O. # Increasing this value increases memory usage but reduces /tmp disk I/O.
arg_parser.add_argument( hidden.add_argument(
'--tmp-file-max-memory-size', dest='tmp_file_max_memory_size', '--tmp-file-max-memory-size', dest='tmp_file_max_memory_size',
type=int, default=512*1024, help=argparse.SUPPRESS) type=int, default=512*1024, help=suppress(
'size of in-memory buffer for each url being processed '
'(spills over to temp space on disk if exceeded)'))
arg_parser.add_argument( arg_parser.add_argument(
'--max-resource-size', dest='max_resource_size', type=int, '--max-resource-size', dest='max_resource_size', type=int,
default=None, help='maximum resource size limit in bytes') default=None, help='maximum resource size limit in bytes')
@ -194,11 +231,18 @@ def _build_arg_parser(prog='warcprox'):
'Qualified name of plugin class, e.g. "mypkg.mymod.MyClass". ' 'Qualified name of plugin class, e.g. "mypkg.mymod.MyClass". '
'May be used multiple times to register multiple plugins. ' 'May be used multiple times to register multiple plugins. '
'See README.rst for more information.')) 'See README.rst for more information.'))
arg_parser.add_argument('--version', action='version', arg_parser.add_argument(
'-q', '--quiet', dest='quiet', action='store_true',
help='less verbose logging')
arg_parser.add_argument(
'-v', '--verbose', dest='verbose', action='store_true',
help='verbose logging')
arg_parser.add_argument(
'--trace', dest='trace', action='store_true',
help='very verbose logging')
arg_parser.add_argument(
'--version', action='version',
version="warcprox {}".format(warcprox.__version__)) version="warcprox {}".format(warcprox.__version__))
arg_parser.add_argument('-v', '--verbose', dest='verbose', action='store_true')
arg_parser.add_argument('--trace', dest='trace', action='store_true')
arg_parser.add_argument('-q', '--quiet', dest='quiet', action='store_true')
return arg_parser return arg_parser
@ -224,7 +268,11 @@ def parse_args(argv):
''' '''
Parses command line arguments with argparse. Parses command line arguments with argparse.
''' '''
arg_parser = _build_arg_parser(prog=os.path.basename(argv[0])) show_hidden = False
if '--help-hidden' in argv:
show_hidden = True
argv = [argv[0], '--help-hidden']
arg_parser = _build_arg_parser(os.path.basename(argv[0]), show_hidden)
args = arg_parser.parse_args(args=argv[1:]) args = arg_parser.parse_args(args=argv[1:])
try: try:
@ -242,11 +290,11 @@ def main(argv=None):
args = parse_args(argv or sys.argv) args = parse_args(argv or sys.argv)
if args.trace: if args.trace:
loglevel = warcprox.TRACE loglevel = logging.TRACE
elif args.verbose: elif args.verbose:
loglevel = logging.DEBUG loglevel = logging.DEBUG
elif args.quiet: elif args.quiet:
loglevel = logging.WARNING loglevel = logging.NOTICE
else: else:
loglevel = logging.INFO loglevel = logging.INFO

27
warcprox/mitmproxy.py
View File

@ -250,7 +250,7 @@ class MitmProxyHandler(http_server.BaseHTTPRequestHandler):
''' '''
self._conn_pool = self.server.remote_connection_pool.connection_from_host( self._conn_pool = self.server.remote_connection_pool.connection_from_host(
host=self.hostname, port=int(self.port), scheme='http', host=self.hostname, port=int(self.port), scheme='http',
pool_kwargs={'maxsize': 6}) pool_kwargs={'maxsize': 6, 'timeout': self._socket_timeout})
self._remote_server_conn = self._conn_pool._get_conn() self._remote_server_conn = self._conn_pool._get_conn()
if is_connection_dropped(self._remote_server_conn): if is_connection_dropped(self._remote_server_conn):
@ -263,10 +263,9 @@ class MitmProxyHandler(http_server.BaseHTTPRequestHandler):
self._remote_server_conn.sock.set_proxy( self._remote_server_conn.sock.set_proxy(
socks.SOCKS5, addr=self.onion_tor_socks_proxy_host, socks.SOCKS5, addr=self.onion_tor_socks_proxy_host,
port=self.onion_tor_socks_proxy_port, rdns=True) port=self.onion_tor_socks_proxy_port, rdns=True)
self._remote_server_conn.timeout = self._socket_timeout self._remote_server_conn.sock.settimeout(self._socket_timeout)
self._remote_server_conn.sock.connect((self.hostname, int(self.port))) self._remote_server_conn.sock.connect((self.hostname, int(self.port)))
else: else:
self._remote_server_conn.timeout = self._socket_timeout
self._remote_server_conn.connect() self._remote_server_conn.connect()
# Wrap socket if SSL is required # Wrap socket if SSL is required
@ -276,16 +275,17 @@ class MitmProxyHandler(http_server.BaseHTTPRequestHandler):
context.check_hostname = False context.check_hostname = False
context.verify_mode = ssl.CERT_NONE context.verify_mode = ssl.CERT_NONE
self._remote_server_conn.sock = context.wrap_socket( self._remote_server_conn.sock = context.wrap_socket(
self._remote_server_conn.sock, server_hostname=self.hostname) self._remote_server_conn.sock,
server_hostname=self.hostname)
except AttributeError: except AttributeError:
try: try:
self._remote_server_conn.sock = ssl.wrap_socket( self._remote_server_conn.sock = ssl.wrap_socket(
self._remote_server_conn.sock) self._remote_server_conn.sock)
except ssl.SSLError: except ssl.SSLError:
self.logger.warn( self.logger.warn(
"failed to establish ssl connection to %s; python " "failed to establish ssl connection to %s; "
"ssl library does not support SNI, considering " "python ssl library does not support SNI, "
"upgrading to python >= 2.7.9 or python 3.4", "consider upgrading to python 2.7.9+ or 3.4+",
self.hostname) self.hostname)
raise raise
return self._remote_server_conn.sock return self._remote_server_conn.sock
@ -424,8 +424,7 @@ class MitmProxyHandler(http_server.BaseHTTPRequestHandler):
self.command, self.path, self.request_version) self.command, self.path, self.request_version)
# Swallow headers that don't make sense to forward on, i.e. most # Swallow headers that don't make sense to forward on, i.e. most
# hop-by-hop headers, see # hop-by-hop headers. http://tools.ietf.org/html/rfc2616#section-13.5.
# http://tools.ietf.org/html/rfc2616#section-13.5.
# self.headers is an email.message.Message, which is case-insensitive # self.headers is an email.message.Message, which is case-insensitive
# and doesn't throw KeyError in __delitem__ # and doesn't throw KeyError in __delitem__
for key in ( for key in (
@ -503,10 +502,7 @@ class PooledMixIn(socketserver.ThreadingMixIn):
def __init__(self, max_threads=None): def __init__(self, max_threads=None):
self.active_requests = set() self.active_requests = set()
self.unaccepted_requests = 0 self.unaccepted_requests = 0
if max_threads: self.max_threads = max_threads or 100
self.max_threads = max_threads
else:
self.max_threads = 100
self.pool = concurrent.futures.ThreadPoolExecutor(self.max_threads) self.pool = concurrent.futures.ThreadPoolExecutor(self.max_threads)
self.logger.info("%s proxy threads", self.max_threads) self.logger.info("%s proxy threads", self.max_threads)
@ -596,11 +592,6 @@ class PooledMitmProxy(PooledMixIn, MitmProxy):
request_queue_size = 4096 request_queue_size = 4096
def __init__(self, options=warcprox.Options()): def __init__(self, options=warcprox.Options()):
if options.max_threads:
self.logger.info(
'max_threads=%s set by command line option',
options.max_threads)
PooledMixIn.__init__(self, options.max_threads) PooledMixIn.__init__(self, options.max_threads)
self.profilers = collections.defaultdict(cProfile.Profile) self.profilers = collections.defaultdict(cProfile.Profile)

51
warcprox/warc.py
View File

@ -19,8 +19,6 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
USA. USA.
''' '''
from __future__ import absolute_import
import logging import logging
import warcprox import warcprox
import hashlib import hashlib
@ -83,16 +81,21 @@ class WarcRecordBuilder:
concurrent_to=principal_record.id) concurrent_to=principal_record.id)
return principal_record, request_record return principal_record, request_record
else: else:
principal_record = self.build_warc_record(url=recorded_url.url, principal_record = self.build_warc_record(
url=recorded_url.url,
warc_date=warc_date, data=recorded_url.request_data, warc_date=warc_date, data=recorded_url.request_data,
warc_type=recorded_url.custom_type, warc_type=recorded_url.custom_type,
content_type=recorded_url.content_type.encode("latin1")) content_type=recorded_url.content_type.encode("latin1"),
payload_digest=warcprox.digest_str(
recorded_url.payload_digest, self.base32),
content_length=recorded_url.size)
return (principal_record,) return (principal_record,)
def build_warc_record(self, url, warc_date=None, recorder=None, data=None, def build_warc_record(self, url, warc_date=None, recorder=None, data=None,
concurrent_to=None, warc_type=None, content_type=None, remote_ip=None, concurrent_to=None, warc_type=None, content_type=None, remote_ip=None,
profile=None, refers_to=None, refers_to_target_uri=None, profile=None, refers_to=None, refers_to_target_uri=None,
refers_to_date=None, payload_digest=None, truncated=None): refers_to_date=None, payload_digest=None, truncated=None,
content_length=None):
if warc_date is None: if warc_date is None:
warc_date = warctools.warc.warc_datetime_str(datetime.datetime.utcnow()) warc_date = warctools.warc.warc_datetime_str(datetime.datetime.utcnow())
@ -126,21 +129,41 @@ class WarcRecordBuilder:
headers.append((b'WARC-Truncated', truncated)) headers.append((b'WARC-Truncated', truncated))
if recorder is not None: if recorder is not None:
headers.append((warctools.WarcRecord.CONTENT_LENGTH, str(len(recorder)).encode('latin1'))) if content_length is not None:
headers.append((
warctools.WarcRecord.CONTENT_LENGTH,
str(content_length).encode('latin1')))
else:
headers.append((
warctools.WarcRecord.CONTENT_LENGTH,
str(len(recorder)).encode('latin1')))
headers.append((warctools.WarcRecord.BLOCK_DIGEST, headers.append((warctools.WarcRecord.BLOCK_DIGEST,
warcprox.digest_str(recorder.block_digest, self.base32))) warcprox.digest_str(recorder.block_digest, self.base32)))
recorder.tempfile.seek(0) recorder.tempfile.seek(0)
record = warctools.WarcRecord(headers=headers, content_file=recorder.tempfile) record = warctools.WarcRecord(headers=headers, content_file=recorder.tempfile)
else: else:
headers.append((warctools.WarcRecord.CONTENT_LENGTH, str(len(data)).encode('latin1'))) if content_length is not None:
digest = hashlib.new(self.digest_algorithm, data) headers.append((
headers.append((warctools.WarcRecord.BLOCK_DIGEST, warctools.WarcRecord.CONTENT_LENGTH,
warcprox.digest_str(digest, self.base32))) str(content_length).encode('latin1')))
else:
headers.append((
warctools.WarcRecord.CONTENT_LENGTH,
str(len(data)).encode('latin1')))
# no http headers so block digest == payload digest
if not payload_digest: if not payload_digest:
headers.append((warctools.WarcRecord.PAYLOAD_DIGEST, payload_digest = warcprox.digest_str(
warcprox.digest_str(digest, self.base32))) hashlib.new(self.digest_algorithm, data), self.base32)
content_tuple = content_type, data headers.append((
record = warctools.WarcRecord(headers=headers, content=content_tuple) warctools.WarcRecord.PAYLOAD_DIGEST, payload_digest))
headers.append((warctools.WarcRecord.BLOCK_DIGEST, payload_digest))
if hasattr(data, 'read'):
record = warctools.WarcRecord(
headers=headers, content_file=data)
else:
content_tuple = content_type, data
record = warctools.WarcRecord(
headers=headers, content=content_tuple)
return record return record

30
warcprox/warcproxy.py
View File

@ -44,6 +44,8 @@ import datetime
import urlcanon import urlcanon
import os import os
from urllib3 import PoolManager from urllib3 import PoolManager
import tempfile
import hashlib
class WarcProxyHandler(warcprox.mitmproxy.MitmProxyHandler): class WarcProxyHandler(warcprox.mitmproxy.MitmProxyHandler):
''' '''
@ -285,8 +287,18 @@ class WarcProxyHandler(warcprox.mitmproxy.MitmProxyHandler):
and (warc_type or 'WARC-Type' in self.headers)): and (warc_type or 'WARC-Type' in self.headers)):
timestamp = datetime.datetime.utcnow() timestamp = datetime.datetime.utcnow()
# stream this? request_data = tempfile.SpooledTemporaryFile(
request_data = self.rfile.read(int(self.headers['Content-Length'])) max_size=self._tmp_file_max_memory_size)
payload_digest = hashlib.new(self.server.digest_algorithm)
# XXX we don't support chunked uploads for now
length = int(self.headers['Content-Length'])
buf = self.rfile.read(min(65536, length - request_data.tell()))
while buf != b'':
request_data.write(buf)
payload_digest.update(buf)
buf = self.rfile.read(
min(65536, length - request_data.tell()))
warcprox_meta = None warcprox_meta = None
raw_warcprox_meta = self.headers.get('Warcprox-Meta') raw_warcprox_meta = self.headers.get('Warcprox-Meta')
@ -301,11 +313,14 @@ class WarcProxyHandler(warcprox.mitmproxy.MitmProxyHandler):
warcprox_meta=warcprox_meta, warcprox_meta=warcprox_meta,
content_type=self.headers['Content-Type'], content_type=self.headers['Content-Type'],
custom_type=warc_type or self.headers['WARC-Type'].encode('utf-8'), custom_type=warc_type or self.headers['WARC-Type'].encode('utf-8'),
status=204, size=len(request_data), status=204,
size=request_data.tell(),
client_ip=self.client_address[0], client_ip=self.client_address[0],
method=self.command, method=self.command,
timestamp=timestamp, timestamp=timestamp,
duration=datetime.datetime.utcnow()-timestamp) duration=datetime.datetime.utcnow()-timestamp,
payload_digest=payload_digest)
request_data.seek(0)
self.server.recorded_url_q.put(rec_custom) self.server.recorded_url_q.put(rec_custom)
self.send_response(204, 'OK') self.send_response(204, 'OK')
@ -492,12 +507,15 @@ class WarcProxy(SingleThreadedWarcProxy, warcprox.mitmproxy.PooledMitmProxy):
def server_activate(self): def server_activate(self):
http_server.HTTPServer.server_activate(self) http_server.HTTPServer.server_activate(self)
self.logger.info( self.logger.notice(
'listening on %s:%s', self.server_address[0], 'listening on %s:%s', self.server_address[0],
self.server_address[1]) self.server_address[1])
# take note of actual port in case running with --port=0 so that other
# parts of warcprox have easy access to it
self.options.server_port = self.server_address[1]
def server_close(self): def server_close(self):
self.logger.info('shutting down') self.logger.notice('shutting down')
http_server.HTTPServer.server_close(self) http_server.HTTPServer.server_close(self)
self.remote_connection_pool.clear() self.remote_connection_pool.clear()

23
warcprox/writerthread.py
View File

@ -31,6 +31,7 @@ import logging
import time import time
import warcprox import warcprox
from concurrent import futures from concurrent import futures
from datetime import datetime
import threading import threading
class WarcWriterProcessor(warcprox.BaseStandardPostfetchProcessor): class WarcWriterProcessor(warcprox.BaseStandardPostfetchProcessor):
@ -52,6 +53,7 @@ class WarcWriterProcessor(warcprox.BaseStandardPostfetchProcessor):
max_workers=options.writer_threads or 1, max_workers=options.writer_threads or 1,
max_queued=10 * (options.writer_threads or 1)) max_queued=10 * (options.writer_threads or 1))
self.batch = set() self.batch = set()
self.blackout_period = options.blackout_period or 0
def _startup(self): def _startup(self):
self.logger.info('%s warc writer threads', self.pool._max_workers) self.logger.info('%s warc writer threads', self.pool._max_workers)
@ -114,7 +116,26 @@ class WarcWriterProcessor(warcprox.BaseStandardPostfetchProcessor):
else self.options.prefix) else self.options.prefix)
# special warc name prefix '-' means "don't archive" # special warc name prefix '-' means "don't archive"
return (prefix != '-' and not recorded_url.do_not_archive return (prefix != '-' and not recorded_url.do_not_archive
and self._filter_accepts(recorded_url)) and self._filter_accepts(recorded_url)
and not self._in_blackout(recorded_url))
def _in_blackout(self, recorded_url):
"""If --blackout-period=N (sec) is set, check if duplicate record
datetime is close to the original. If yes, we don't write it to WARC.
The aim is to avoid having unnecessary `revisit` records.
Return Boolean
"""
if self.blackout_period and hasattr(recorded_url, "dedup_info") and \
recorded_url.dedup_info:
dedup_date = recorded_url.dedup_info.get('date')
if dedup_date and recorded_url.dedup_info.get('url') == recorded_url.url:
try:
dt = datetime.strptime(dedup_date.decode('utf-8'),
'%Y-%m-%dT%H:%M:%SZ')
return (datetime.utcnow() - dt).total_seconds() <= self.blackout_period
except ValueError:
return False
return False
def _log(self, recorded_url, records): def _log(self, recorded_url, records):
# 2015-07-17T22:32:23.672Z 1 58 dns:www.dhss.delaware.gov P http://www.dhss.delaware.gov/dhss/ text/dns #045 20150717223214881+316 sha1:63UTPB7GTWIHAGIK3WWL76E57BBTJGAK http://www.dhss.delaware.gov/dhss/ - {"warcFileOffset":2964,"warcFilename":"ARCHIVEIT-1303-WEEKLY-JOB165158-20150717223222113-00000.warc.gz"} # 2015-07-17T22:32:23.672Z 1 58 dns:www.dhss.delaware.gov P http://www.dhss.delaware.gov/dhss/ text/dns #045 20150717223214881+316 sha1:63UTPB7GTWIHAGIK3WWL76E57BBTJGAK http://www.dhss.delaware.gov/dhss/ - {"warcFileOffset":2964,"warcFilename":"ARCHIVEIT-1303-WEEKLY-JOB165158-20150717223222113-00000.warc.gz"}