q S r SSKJr SSKJr SSKJr SSKJr SSKJr SSKrSSK r SSK
r
SSK r SSK r SSK
r
SSKrSSKrSSKrSSKrSSKJr SSKrSS KJr SS
KJr SS KJr SS KJr SS
K J r SSK J r SSK J!r! SSK J"r" SSK J#r# SSK$J%r% SSK$J&r& SSK'J(r( SSK)r)\RV ( a \,r-Sr.Sr/Sr0Sr1S\1- S\.- - r2\" SS5 r3S r4S r5S r6S r7S r8 S$S! jr9 " S" S#\5 r:g! \* a Sr) NZf = f)%&Implementation of gsutil test command. )absolute_import)print_function)division)unicode_literals)
namedtupleN)range)ProjectIdException)Command)ResetFailureCount)CommandException)PopulateProjectId) GetTestNames)InvokedFromParFile)unittest)NO_MAX)UTF8)
IS_WINDOWS 2 sequential_onlyz3
gsutil test [-l] [-u] [-f] [command command...]
z
SYNOPSIS
a
DESCRIPTION
The gsutil test command runs the gsutil unit tests and integration tests.
The unit tests use an in-memory mock storage service implementation, while
the integration tests send requests to the production service using the
`preferred API
`_ set in the
boto configuration file.
CAUTION: The ``test`` command creates test buckets and objects in your project.
Force quitting the ``test`` command can leave behind stale buckets, objects,
and HMAC keys in your project.
To run both the unit tests and integration tests, run the command with no
arguments:
gsutil test
To run the unit tests only (which run quickly):
gsutil test -u
Tests run in parallel regardless of whether the top-level -m flag is
present. To limit the number of tests run in parallel to 10 at a time:
gsutil test -p 10
To force tests to run sequentially:
gsutil test -p 1
To have sequentially-run tests stop running immediately when an error occurs:
gsutil test -f
To run tests for one or more individual commands add those commands as
arguments. For example, the following command will run the cp and mv command
tests:
gsutil test cp mv
To list available tests, run the test command with the -l argument:
gsutil test -l
The tests are defined in the code under the gslib/tests module. Each test
file is of the format test_[name].py where [name] is the test name you can
pass to this command. For example, running "gsutil test ls" would run the
tests in "gslib/tests/test_ls.py".
You can also run an individual test class or function name by passing the
test module followed by the class name and optionally a test name. For
example, to run the an entire test class by name:
gsutil test naming.GsutilNamingTests
or an individual test function:
gsutil test cp.TestCp.test_streaming
You can list the available tests under a module or class by passing arguments
with the -l option. For example, to list all available test functions in the
cp module:
gsutil test -l cp
To output test coverage:
gsutil test -c -p 500
coverage html
This will output an HTML report to a directory named 'htmlcov'.
Test coverage is compatible with v4.1 of the coverage module
(https://pypi.python.org/pypi/coverage).
OPTIONS
-b Run tests against multi-regional US buckets. By default,
tests run against regional buckets in us-central1.
-c Output coverage information.
-f Exit on first sequential test failure.
-l List available tests.
-p N Run at most N tests in parallel. The default value is %d.
-s Run tests against S3 instead of GS.
-u Only run unit tests.
TestProcessDataz name return_code stdout stderrc D ^ ^ " UU 4S jS[ R 5 mT$ )zCreates a closure of CustomTestResult.
Args:
total_tests: The total number of tests being run.
Returns:
An instance of CustomTestResult.
c 2 >^ \ rS rSrSrUU U4S jrSrU =r$ )3MakeCustomTestResultClass..CustomTestResult zDA subclass of unittest.TextTestResult that prints a progress report.c > [ TU ] U5 U R ( a SR UR 5 R S5 SS 5 nSU R T[ U R 5 [ U R 5 [ U R 5 U4- nUS S nUR S5 nU R R SU- 5 g g )N.z(
%d/%d finished - E[%d] F[%d] s[%d] - %sI z%s - )super startTestdotsjoinidsplittestsRunlenerrorsfailuresskippedljuststreamwrite)selftesttest_idmessageCustomTestResult __class__ total_testss &platform/gsutil/gslib/commands/test.pyr" =MakeCustomTestResultClass..CustomTestResult.startTest s d-d3
((4779??3/45>MM;DKK0@
&DLL(97DD #2,--# 'G+,
)__name__
__module__ __qualname____firstlineno____doc__r" __static_attributes__
__classcell__)r4 r3 r5 s @r6 r3 r s N - -r8 r3 )r TextTestResult)r5 r3 s `@r6 MakeCustomTestResultClassrB s" - -00 -
r8 c U /n/ nU( a UR 5 nU H` n[ U[ R 5 ( a UR U5 M5 UR UR 5 [
S5 S 5 Mb U( a M U$ )zGTakes a list of test suites and returns a list of contained test names.gslib.tests.test_N)pop
isinstancer TestSuiteappendr% r( )
test_suitesuites
test_namessuiter0 s r6 GetTestNamesFromSuitesrM sv <&* JJLE D(,, - -
d$'')C(;$<$=>?
r8 c p [ U R 5 R S5 S S- U R - $ )z%>EK{ 0%88 N9 56
["5t
<
< nY 78
Y 3T
:
: nY 78 I 2 2 &&~i'@A&--nY.GH % !
"F>$:
*
+
1
2
4 4r8 c 8 SnU H nU( a M US-
nM U$ )z*Counts number of falses in the input list.r rP r9 )
input_list num_falsere s r6 CountFalseInListrk s' )d 41ni
r8 c r Un[ 5 ( d [ R /O/ n[ R R
( a S/O/ n[ R R ( a S/O/ n / n
SS[ 5 - /n
[ R " 5 n U n [ U5 U: Ga U[ U 5 : Ga [ R R 5 n
U( a X]S' [ 5 n[! U[" R$ /- U
- S/- U- U - S[& - /- X [ S5 S /- 5 Vs/ s H n[( R* " U5 PM nn[( R, " U
5 H3 u nn[( R* " U5 U[( R* " U5 ' M5 UR/ [0 R2 " U[0 R4 [0 R4 US
95 US -
nUR/ S 5 [ R " 5 U -
S
: a: [7 SX-
[ U5 [ U 5 4- 5 [ R " 5 n [ U5 U: a U[ U 5 : a GM U[ U 5 :X a" [7 S[ U5 [ U 5 4- 5 U$ ! [ a GN1f = fs snf )aO Creates test processes to run tests in parallel.
Args:
parallel_tests: List of all parallel tests.
test_index: List index of last created test before this function call.
process_list: List of running subprocesses. Created processes are appended
to this list.
process_done: List of booleans indicating process completion. One 'False'
will be added per process created.
max_parallel_tests: Maximum number of tests to run in parallel.
root_coverage_file: The root .coverage filename if coverage is requested.
Returns:
Index of last created test.
-s-bz-oz GSUtil:default_project_id=%sGSUTIL_COVERAGE_OUTPUT_FILEr0 --rD N)stdoutstderrenvrP Fr z.Created %d new processes (total %d/%d created)z.Test process creation finished (%d/%d created)) r sys
executabletestsutil RUN_S3_TESTSUSE_MULTIREGIONAL_BUCKETSr r
timerk r( osenvironcopydictlistgslib GSUTIL_PATH_SEQUENTIAL_ISOLATION_FLAGsix
ensure_str iteritemsrH
subprocessPopenPIPEprint)parallel_tests
test_index process_list process_donemax_parallel_testsroot_coverage_fileorig_test_indexexecutable_prefix s3_argumentmultiregional_bucketsproject_id_argprocess_create_start_time
last_log_timers envstrpartcmdkvs r6 CreateTestProcessesr s * / /A.B.Bs~~& **11r+$)JJ$H$H4&b. ,/@/BBN #iik+- , '*< < c.) )
**//
C+=
'(
VF *.
H
"
" .
. /
0
',?(@(A
B C
D *
*
t *
c"1"%.."3fS^^A
# * * # %&
!OJ yy{]"Q&
=*C ,=.!## % iikmA
, '*< < c.) )@ 3~&& ; s> 2 3 4 6 U
s 0J# 2 J4#
J10J1c
\ rS rSrSr\R " S/ \S\SSSS\ /S9
r
\R " S/ S S
\ 0 S 9r
S rS
r\" 5 \" 5 4S jrS rSrg) TestCommandih r r0 r zbuflp:scTF) command_name_aliasesusage_synopsismin_argsmax_argssupported_sub_args file_url_okprovider_url_okurls_start_argsupported_private_args command_helpz2Run gsutil unit/integration tests (for developers)) help_namehelp_name_aliases help_typehelp_one_line_summary help_textsubcommand_help_textc
/ n/ n/ nSnSnSn [ U5 n
[ R " 5 =p[ USUUUUS9n
[ U5 U
: Ga [ [ U5 5 H nX^ ( d XN R 5 c M! SX^' XN R 5 u nnXN R R 5 XN R R 5 XN R S:w a US-
nUR [ X XN R UUS95 M [ U5 U
: a [ UU
UUUUS9n
[ U5 U
: Ga [ R " 5 U -
S: a [ S[ U5 X4- 5 [ U5 U :X a US-
nO
[ U5 n SnUS: a~ / n[ [ U5 5 H! nX^ ( a M UR X 5 M# [ R " 5 U -
n[ S R [ [ U5 5 US
95 [ R " 5 n [ R " S5 [ U5 U
: a GM [ R " 5 nU( a U H} nUR" S:w d M UR R% S 5 n[ S UR& - 5 U H/ n[ UR) [* 5 R- 5 5 M1 M UUU -
4$ )
a> Executes the parallel/isolated portion of the test suite.
Args:
parallel_integration_tests: List of tests to execute.
max_parallel_tests: Maximum number of parallel tests to run at once.
coverage_filename: If not None, filename for coverage output.
Returns:
(int number of test failures, float elapsed time)
r )r TrP )name return_coderq rr r z %d/%d finished - %d failures zL{sec} seconds elapsed since beginning parallel tests.
Still running: {procs})secprocs
zResults for failed test %s:)r( rz r r poll communicaterq closerr
returncoderH r r formatrQ intsleepr r& r decoder strip)r/ parallel_integration_testsr coverage_filenamer r process_resultsnum_parallel_failures progress_less_logging_cyclescompleted_as_of_last_lognum_parallel_testsparallel_start_timer r proc_numrq rr
still_runningelapsedprocess_run_finish_timeresult
new_stderrlines r6 RunParallelTests TestCommand.RunParallelTests s LLO #$ 78*.))+5$%?%&%1%1%78I KJ o
!3
3C -.( !\%;%@%@%B%J
!% %/;;= %%++- %%++- ! , , 1
1
$
!;!E(4(>(I(I#)#)
+ , /
\ / /()C)3)5)5); ? u%< = U#8 9 ; ;= -. G %7 Dk
'(
)* r8 c [ U5 S: d [ U5 S: ah [ U5 n[ U5 nUR U5 n[ S5 U H/ u pE[ SUR 5 - 5 [ SU- 5 M1 gg)a Prints all skipped tests, and the reasons they were skipped.
Takes the union of sequentual_skipped and parallel_skipped,
and pretty-prints the resulting methods and reasons. Note that these two
arguments are lists of tuples from TestResult.skipped as described here:
https://docs.python.org/2/library/unittest.html#unittest.TestResult.skipped
Args:
sequentual_skipped: An instance of TestResult.skipped.
parallel_skipped: An instance of TestResult.skipped.
r zTests skipped:z z Reason: N)r( setunionr r% )r/ r parallel_skipped all_skippedmethodreasons r6 r TestCommand.PrintSkippedTests s "c*:&;a&? 12 -.&,,-=>k '.&
dVYY[ !
nv%& (
'@r8 c SnSn[ nSnSnU R ( a U R H u pgUS:X a S[ R l M" US:X a SnM, US:X a SnM6 US:X a SnM@ US[
- :X a SnMQ US:X a
[
U5 nMd US :X aA [ R R ( d [ S
5 eS[ R l M US :X d M S[ R l
M U( a [ ( d [ S 5 e[ R R ( a0 U[ : a& U R R S
[ 5 [ n[ [! 5 5 nU( aM U R" ( d< [% S[' U5 - 5 [% SSR) [ U5 5 5 gU R" ( aZ / n U R" HG n
X; d U
R+ S5 S U; a U R- SU
- 5 M6 U R- U
5 MI OU V
s/ s H n
SU
- PM
n n
[. R0 " 5 [. R2 " 5 n U ( aA [. R4 " 5 n U H& n
U R7 U
5 nU R9 U5 M( U( aG [M W 5 n[% S[' U5 - 5 [% SSR) [ U5 5 5 g[N RP " 5 RS 5 [N RT :: a SnO&Sn[N RV " [N RX 5 U( aO [ R " S /S S [Z R\ " 5 S - /S 9nUR_ 5 URa 5 SnSn[c W 5 u nnnn[ R R ( d / n[N Rd " S!U5 [N Rd " S"U5 [N Rd " S#U5 [N Rd " S$U5 U=( d [' U5 S:* =( a U( + nS%[f Rh S&' U( aT U Rk 5 n [m U 5 n [. Rn " UU US'9n U Rq U 5 n U Rs 5 nGOUS:X a UU-
n/ n[t Rt " 5 n [' U5 [' U5 - ( a [% S([' U5 [' U5 - - 5 UU- n!U Rw [ U! V"s/ s H n"U"PM sn"5 5 n U Rk 5 n#[m U#5 n [. Rn " UU US'9n U Rq U 5 n U Rs 5 nU Rx n$OSn#Sn[t Rt " 5 U -
n%UU-
n[' U5 n&U&( d O/ n$[{ UU&5 n'U&S: aK US: aE S)n(U'[ : a U(S*-
n([% S+R) [| R~ " U(U&U'4- 5 5 5 O[% S,U&- 5 U R UUU( a WR R OS5 u nn)U R U#UU$U%U&UU)5 U( aR WR 5 UR 5 UR 5 [% S-UR R - 5 S.[f Rh S&' U( a U( d [ 5 ggs sn
f ! [: [< 4 a| nSU
< S[? U5 < 3n [A U
5 OL! [B a? n[D RF " 5 n[H RJ " SSU5 nUSU- -
n SnAOSnAff = f[ U5 eSnAff = fs sn"f )/z)Command entry point for the test command.Frn Tz-cz-fz-lrp z-prm zbS3 tests require S3 credentials. Please add appropriate credentials to your .boto file and re-run.z-uzrCoverage has been requested but the coverage module was not found. You can install it with "pip install coverage".zCReducing parallel tests to %d due to S3 maximum bucket limitations.zFound %d test names: z
r r zgslib.tests.test_%sz%Failed to import test code from file z#. TestLoader provided this error:
z\nz
z
Additional traceback:
%sNrP r zgslib/third_party/*z
gslib/tests/**)sourceomitzSequential tests to run: %szIsolated tests to run: %sz Parallel unit tests to run: %sz%Parallel integration tests to run: %s1GSUTIL_TEST_ANALYTICS) verbosity resultclassfailfastz Running %d tests sequentially.z1Running %d tests in parallel mode (%d processes).z Please be patient while your CPU is incinerated. If your machine becomes unresponsive, consider reducing the amount of parallel test processes by running 'gsutil test -p '.
z4Running %d tests sequentially in isolated processes.z%Coverage information was saved to: %s0)H _DEFAULT_TEST_PARALLEL_PROCESSESsub_optsrv rw ry r long HAS_S3_CREDSr
rx RUN_INTEGRATION_TESTScoverage#_DEFAULT_S3_TEST_PARALLEL_PROCESSESloggerwarnr] r argsr r( r$ r& rH r installHandler
TestLoaderrG loadTestsFromNameaddTests ImportErrorAttributeErrorrQ
__import__ Exception traceback
format_excresubrM logging getLoggergetEffectiveLevelINFOdisableERRORtempfile
gettempdirerasestartrg debugr{ r| countTestCasesrB TextTestRunnerrun
wasSuccessfulrz loadTestsFromNamesr+ mintextwrapwrapr
data_filesfilenamer stopcombinesaver )*r/ r
list_testsr perform_coverager oarK commands_to_testr loaderrL command_namesuite_for_current_commandemsg stack_traceerrr coverage_controllerr r r_ r^ parallel_unit_testsr run_tests_sequentiallyr5 r runnerretsequential_start_timesequential_tests_to_run test_namer r r r
num_processesr2 r s* r6
RunCommandTestCommand.RunCommand
s HJ9 O }}--$! 915%**
.
$Y !
$Y(
$Y*
444
5 !/
$Y #Aw
$Y((" $6 7
7 %)%**
!
$Y-2%**
*- 0 <
= =
@@
kk =? ? 'J$)) "S_ 45 CVJ /01
yy))$ C !3z!A
!
!"7$">
?
!
!$
' DNN:4/$6:N
"F "e*, &&,&>&>|&L
#
..2
3 +( )%0j "S_ 45 CVJ /01
,,.',,>ii
oogmm$
% - -gY2G2A2:2E2E2G#2M40 1 ! ! %A$G!~': :: + +#% MM/1AB MM-~> MM24GH MM9,. . 3 "#= >! C 2#1 1 +.BJJ&' ((*k-k:k&&3>08:f
JJu c ,,. q 66%'" "iik
%8!9 9
.#$s+>'??A B"25H"H )) /FG/F)I/FG HJ $ 3 3 5 /0DE ((95@2:< jj . . 0 [[ ! $ .C C !N2 9:
.0BC
!&81&<G'
=
= 6
7G
mmG'9=&I IJL N G#$ &9=9N9N &(:
* * 3 3%):+6 5
24F 02I 02G 3 5
!!#
4
+
+
4
45 7 +.BJJ&'"7
Q O ^, &(4c!f># = | $ =#..0K&&+6C 4< r CreateCommandSpec _SYNOPSISr r command_specHelpSpec_DETAILED_HELP_TEXT help_specr r r r r3 r? r9 r8 r6 r r h s~ . ** # 8 9 ,
># )X=t+4 25 ',c
r8 r )N);r>
__future__r r r r collectionsr r r{ r r rt r r rz r r six.movesr r gslib.cloud_apir
gslib.commandr r gslib.exceptionr
gslib.project_idr gslib.testsrv gslib.tests.utilr r r gslib.utils.constantsr r gslib.utils.system_utilr r r PY3r r r r r r6 r9 r rB rM rT rg rk r r r9 r8 r6 rG s - & % ' "
. ! + , . ) / % ( & . 77 $#$ &( # .
^| '}^'`' D . =?6 %24j ,0 K\H
' H
g
(s C0 0C;:C;