1
0
mirror of https://github.com/webrecorder/pywb.git synced 2025-03-15 08:04:49 +01:00

265 Commits

Author SHA1 Message Date
Ilya Kreymer
06a22c845b ensure cdx loading happens lazily
add perms test to ensure 'short-circuiting' in case of
permission exception
2014-03-01 18:40:16 -08:00
Kenji Nagahashi
1f65eff828 Merge remote-tracking branch 'origin/master' into cdx-server
Conflicts:
	pywb/cdx/cdxdomainspecific.py
	pywb/cdx/cdxserver.py
	pywb/cdx/test/cdxserver_test.py
	setup.py
	tests/test_integration.py
2014-02-28 19:47:24 +00:00
Ilya Kreymer
c084b45298 Merge master into pkg-reorg 2014-02-28 10:25:36 -08:00
Ilya Kreymer
304a33aa5b add coverage badge 2014-02-27 18:52:41 -08:00
Ilya Kreymer
921b2eb2e1 improve testing and a few fixes:
archivalrouter: support empty collection, with and without SCRIPT_NAME
cdx: remove cdx source test, including access denied
replay: when content-type present, limit the decompressed stream to content-length
(this ensures last 4 bytes in warc/arc record are not read)
integration tests for identity replay
2014-02-27 18:43:55 -08:00
Kenji Nagahashi
9eda5ad97e address test cases broken by previous commit.
move py.test fixture and fixture classes (TestExclusionPerms, PrintReporter)
  to tests.fixture module. update test_config.yaml accordingly.
2014-02-28 01:39:04 +00:00
Kenji Nagahashi
2c40c9b112 refactor cdxserver, add tests focused on wsgi_cdxserver, add docstrings.
align cdxops function interfaces - all cdx_iter.
  move module functions / common ops to class methods
  support both 0/1 and true/false for boolean parameters
  move CDXObject to text conversion to wsgi_cdxserver (may have broken
    embedded cdxserver mode).
  pass config object as function arg rather than as global var.
2014-02-27 01:58:07 +00:00
Ilya Kreymer
51d61a8738 package reorg!
split up remaining parts of pywb root pkg
into core, dispatch and bootstrap
2014-02-24 03:00:01 -08:00
Ilya Kreymer
9194e867ea - add referrer self-redirect check and test case
- dispatching: cleanup wbrequestresponse, move tests to a seperate file
- wbrequest: store both rel_prefix and host_prefix, with wb_prefix either full
or rel path as needed, so that full and relative paths are
both available in wbrequest
- create WbUrlHandler to differentiate handlers which
support WbUrl (timestamp[mod]/url) semantic vs other request handlers.
2014-02-23 23:31:54 -08:00
Ilya Kreymer
922917a631 rename BufferedReader -> DecompressingBufferedReader
remove max_len from DecompressingBufferedReader as it applied to
the compressed size, not original size.
Add integration test for verifying content length of larger file
2014-02-20 11:53:08 -08:00
Ilya Kreymer
ff428ed43e exclusions: add AllAllowPerms and refactor exclusions interface
add TestExclusionPerms and a sample exclusion integration test
refactor cdx server init params into **kwargs
convert all cdx params to use camelCase
2014-02-19 20:20:31 -08:00
Ilya Kreymer
a09dec4b3e cdx: add domain-specific rules at cdx layer for custom canonicalization!
and 'fuzzy' matching when not found
handled via cdxdomainspecific.py
BaseCDXServer contains a canonicalizer object and a fuzzy query
canonicalizer abstracted to seperate class (in canonicalizer.py)
clean up cdx related exceptions
default rules read from cdx/rules.yaml
filename configurable via 'domain_specific_rules' setting in config.yaml
fix typo in pywb/rewrite
2014-02-18 14:56:13 -08:00
Ilya Kreymer
94f1dc3be5 cleanup wbexceptions, remove unused 2014-02-17 10:23:37 -08:00
Ilya Kreymer
5345459298 pywb 0.2!
move to distinct packages: pywb.utils, pywb.cdx, pywb.warc, pywb.util, pywb.rewrite!
each package will have its own README and tests
shared sample_data and install
2014-02-17 10:01:09 -08:00
Ilya Kreymer
2528ee0a7c refactoring of binsearch and cdxserver into seperate packages
also move complicated doctests and integration tests to tests/
2014-02-12 13:16:07 -08:00