NSrSSKJr SSKJr SSKJr SSKJr SSKrSSKJr SSK r SSK r SSK r SSK r SSK r SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKrSS KJr SS KJr SS KJr SS KJr SS KJr SSKJ r SSK!J"r" SSK#J$r$ SSK%J&r& SSK%J'r' SSK(J)r) SSK(J*r* SSK(J+r+ SSK(J,r, SSK(J-r- SSK.J/r/ SSK0J1r1 SSK2J3r3 SSK2J4r4 SSK2J5r5 SSK2J6r6 SSK2J7r7 SSK8J9r9 SS K8J:r: SS!K8J;r; SS"K8JJ?r@ SS%KAJBrB SS&KAJCrC SS'KAJDrD SS(KAJErE SS)KFJGrG SS*KFJHrH SS+KFJIrI SS,KJJKrK SS-KJJLrL SS.KMJNrN SS/KMJOrO SSKPrSS0KPJQrQ SS1KPJRrR SS2KPJSrS SS3KPJTrT SS4KPJUrU SS5KPJVrV SS6KPJWrW SS7KPJXrX SS8KPJYrY SS9KZJ[r[ SS:K\J]r] SS;K^J_r_ SSKaJcrc SS?KaJdrd SS@KeJfrf SSAKgJhri SSBKjJkrl SCrnSDroSErpSFrqSGrrSHrsS}SIjrtSJru/rvSKrwSLrxSMrySNrzSOr{SPr|SCr}SQSR/SQSS/SQST/SUSS/SUST/SVSR/SVSS/SVST//SWQ/SXQSYSS/SZSS/SZST/S[SS/S[ST/S\.r~S]rS^rS_r\"S`/SaQ5r"SbSc\$\]5r"SdSe\5r"SfSg\"SgSh55rSir"SjSk\GR5r"SlSm\5r"SnSo\GR5r"SpSq\5rSrr"SsSt\5rSurSvrSwrSxrSyrSzrS{rS|rg!\ma SrlGNf=f)~axBase class for gsutil commands. In addition to base class code, this file contains helpers that depend on base class state (such as GetAndPrintAcl) In general, functions that depend on class state and that are used by multiple commands belong in this file. Functions that don't depend on class state belong in util.py, and non-shared helpers belong in individual subclasses. )absolute_import)print_function)division)unicode_literalsN) namedtuple) StorageUri)AccessDeniedException)ArgumentException)ServiceException)CloudApiDelegator) ApiSelector)GsutilApiMapFactory)CommandException) HelpProvider)CaptureThreadStatException)LogPerformanceSummaryParams)CopyObjectInfo)CopyObjectsIterator)NameExpansionIterator)NameExpansionResult)SeekAheadNameExpansionIterator)PluralityCheckableIterator)SeekAheadThread)ChildProcessSignalHandler)GetCaughtSignals) KillProcess)MultithreadedMainSignalHandler)RegisterSignalHandler) HaveFileUrls)HaveProviderUrls)StorageUrlFromString)UrlsAreForSingleProvider)UrlsAreMixOfBucketsAndObjects)storage_v1_messages) FinalMessage)MetadataMessage)PerformanceSummaryMessage)ProducerThreadMessage)MainThreadUIQueue) UIController)UIThread)GetFriendlyConfigFilePaths)!GetMaxConcurrentCompressedUploads)NO_MAX)UTF8) AtomicDict)$CheckMultiprocessingAvailableAndInit)multiprocessing_context)ProcessAndThreadSafeInt)PutToQueueWithTimeout)SEEK_AHEAD_JOIN_TIMEOUT)ShouldProhibitMultiprocessing)UI_THREAD_JOIN_TIMEOUT)ZERO_TASKS_TO_DO_ARGUMENT)RsyncDiffToApply)GcloudStorageCommandMixin) GetTermLines) IS_WINDOWS)AclTranslation)GetNonMetadataHeaders)PRIVATE_DEFAULT_OBJ_ACL)CreateWildcardIterator)queue)RandomicJ[R"U5nUR(d{SUlUR [R R 5 [R"5nUR[R"S55 URU5 U$)aFetches a logger with the given name that resembles 'print' output. Initial Logger Configuration: The logger abides by gsutil -d/-D/-DD/-q options. If none of those options were specified at invocation, the returned logger will display all messages logged with level INFO or above. Log propagation is disabled. If a logger with the specified name has already been created and configured, it is not reconfigured, e.g.: foo = CreateOrGetGsutilLogger('foo') # Creates and configures Logger "foo". foo.setLevel(logging.DEBUG) # Change level from INFO to DEBUG foo = CreateOrGetGsutilLogger('foo') # Does not reset level to INFO. Args: command_name: (str) Command name to create logger for. Returns: A logging.Logger object. Fz %(message)s) logging getLoggerhandlers propagatesetLevelrootlevel StreamHandler setFormatter Formatter addHandler) command_namelog log_handlers platform/gsutil/gslib/command.pyCreateOrGetGsutilLoggerrTtsr, ,'# CMLL##$'')KW..}=>NN; *c:URRU5 gN)logger exceptionclses rS_DefaultExceptionHandlerr]s**qrUcUR(dgURnUR5(aK[RR UR 5(aURRSU5 gg)NTzSkipping symbolic link %s...F) exclude_symlinksexpanded_storage_url IsFileUrlospathislink object_namerXinfo)command_instanceurl exp_src_urls rS_UrlArgCheckerrjs]  * * ((+ 0G0G!H!H  !?M  rUcg)NT) unused_argss rSDummyArgCheckerrns rUc URXS9$)N thread_state) SetAclFunc)r[name_expansion_resultrqs rSSetAclFuncWrapperrts - IIrUcZURR[U55 SUlg)zDException handler that maintains state about post-completion status.FN)rXerrorstreverything_set_okayrZs rSSetAclExceptionHandlerrys **3q6!#rUct[(a-[[SS5(a[R"5 ggg)Natfork) CryptoRandomgetattrr{rlrUrS_CryptoRandomAtForkr~s*\glHd;;  <\rUcd[R"[5n[R U5 U$rW)r2QueueMAX_QUEUE_SIZEqueuesappend new_queues rS_NewMultiprocessingQueuers$%++N;)--  rUcd[R"[5n[RU5 U$rW)rrrrrs rS_NewThreadsafeQueuers"kk.))--  rUi~i0ucL[RRSS[5$)NGSUtiltask_estimation_threshold)botoconfiggetint!DEFAULT_TASK_ESTIMATION_THRESHOLDrlrUrS_GetTaskEstimationThresholdrs!   H&A= ??rUaclchgetsetcorsdefacl)rEroff)rEronrE versioningweb)chaclgetaclsetaclgetcorssetcorschdefacl getdefacl setdefacldisablelogging enablelogging getlogging getversioning setversioning getwebcfg setwebcfgcP[R"5q/q/q[R 5q[S5q[[S9q [[S9q [[S9q [R5q[R 5q[[S9q[S5q[S5q[[S9q[[S9q[[S9q[R-5q[S5q[R3[45q[R9[;55qg)a Initializes module-level variables that will be inherited by subprocesses. On Windows, a multiprocessing.Manager object should only be created within an "if __name__ == '__main__':" block. This function must be called, otherwise every command that calls Command.Apply will fail. While multiprocessing variables are initialized at the beginning of gsutil execution, new processes and threads are created only by calls to Command.Apply. When multiple processes and threads are used, the flow of startup/teardown looks like this: 1. __main__: initializes multiprocessing variables, including any necessary Manager processes (here and in gslib.utils.parallelism_framework_util). 2. __main__: Registers signal handlers for terminating signals responsible for cleaning up multiprocessing variables and manager processes upon exit. 3. Command.Apply registers signal handlers for the main process to kill itself after the cleanup handlers registered by __main__ have executed. 4. If worker processes have not been created for the current level of recursive calls, Command.Apply creates those processes. ---- Parallel operations start here, so steps are no longer numbered. ---- - Command.Apply in the main thread starts the ProducerThread. - The Producer thread adds task arguments to the global task queue. - It optionally starts the SeekAheadThread which estimates total work for the Apply call. - Command.Apply in the main thread starts the UIThread, which will consume messages from the global status queue, process them, and display them to the user. - Each worker process creates a thread pool to perform work. - The worker process registers signal handlers to kill itself in response to a terminating signal. - The main thread of the worker process moves items from the global task queue to the process-local task queue. - Worker threads retrieve items from the process-local task queue, perform the work, and post messages to the global status queue. - Worker threads may themselves call Command.Apply. - This creates a new pool of worker subprocesses with the same size as the main pool. This pool is shared amongst all Command.Apply calls at the given recursion depth. - This reuses the global UIThread, global status queue, and global task queue. - This starts a new ProducerThread. - A SeekAheadThread is not started at this level; only one such thread exists at the top level, and it provides estimates for top-level work only. - The ProducerThread runs out of tasks, or the user signals cancellation. - The ProducerThread cancels the SeekAheadThread (if it is running) via an event. - The ProducerThread enqueues special terminating messages on the global task queue and global status queue, signaling the UI Thread to shut down and the main thread to continue operation. - In the termination case, existing processes exit in response to terminating signals from the main process. ---- Parallel operations end here. ---- 5. Further top-level calls to Command.Apply can be made, which will repeat all of the steps made in #4, except that worker processes will be reused. T)managerN)r2Managerrconsumer_pools task_queuesLockcaller_id_lockr3caller_id_counterr0 total_taskscall_completed_mapglobal_return_values_map Conditionneed_pool_or_done_condworker_checking_level_lockcaller_id_finished_countnew_pool_neededcurrent_max_recursive_levelshared_vars_mapshared_vars_list_map thread_statsdict class_map failure_countrrglob_status_queueBoundedSemaphorer-!concurrent_compressed_upload_lockrlrUrS"InitializeMultiprocessingVariablesr sR $ + + -'.+<<>.-d37++"'2(8#,,. '||~(8 ,D1/ 7 =w//#G4G,,lln)*$/- mmN3'.&>&>')'+#rUc[5 [R5 [RR R R5 g)z7Should be called by signal handlers prior to shut down.N)ShutDownGsutilrshutdowngslibutilsparallelism_framework_utiltop_level_managerrlrUrS TeardownMultiprocessingProcessesrs4  ++((::CCErUc[S5q[5q[R "5q[5q[5q[S5q [S5q [R"[5q [5q[R"5q[5q[5q[5q/q[5q[R "5q[R."[155qg)a:Initializes module-level variables used when running multi-threaded. When multiprocessing is not available (or on Windows where only 1 process is used), thread-safe analogs to the multiprocessing global variables must be initialized. This function is the thread-safe analog to InitializeMultiprocessingVariables. FN)r3rr0r threadingrrrrrrrrrrrrrrrrrrrr-rrlrUrSInitializeThreadingVariablesrs.e4'\>>#.!|l) 7 >)%0-kk.1'\$..0#L/,+ +(~~/&/&@&@')'+#rU CommandSpec rPusage_synopsiscommand_name_aliasesmin_argsmax_argssupported_sub_args file_url_okprovider_url_okurls_start_arggs_api_supportgs_default_apisupported_private_argsargparse_argumentsc ^\rSrSrSrSrSrSrSr\ SSS\ SSSSSSSS4 S j5r S r \ "\ 5rS rS rS-U4S jjrSrSrS.SjrSrS/SjrSrSrS0SjrS0SjrSrSrSrSrSrSr Sr!S0Sjr"S0Sjr#S r$S1S!jr%S"r&S#r'"S$S%\(5r)S\*SSSSSS4S&jr+S'r,S(r-S2S)jr.S*r/S+r0S,r1U=r2$)3Commandiz#Base class for all gsutil commands.N) rriamkmslabelrE notification retentionrrFc [UUU=(d /UUUUUUU =(d [R/U =(d [RU U =(d /S9 $)z2Creates an instance of CommandSpec, with defaults.r)rr XMLrs rSCreateCommandSpecCommand.CreateCommandSpecsX L&4,@,FB ( (*<#.'6&4&4&I8I&4&G .D*<*B D DrUc.URR$rW) command_specrPselfs rS_GetDefaultCommandNameCommand._GetDefaultCommandNames    ) ))rUc.URR$)zzCalculate the index in args of the first URL arg. Returns: Index of the first URL arg (according to the command spec). )rrrs rS_CalculateUrlsStartArgCommand._CalculateUrlsStartArg$s    + ++rUc [RURS5nU(a`USSU-nURR SR [ R"SURURS.-555 U$)zBMap deprecated aliases to the corresponding new command, and warn.N zYou are using a deprecated alias, "%(used_alias)s", for the "%(command_name)s" command. This will stop working on 9/9/2014. Please use "%(command_name)s" with the appropriate sub-command in the future. See "gsutil help %(command_name)s" for details.) used_aliasrP) OLD_ALIAS_MAPrcommand_alias_usedrXwarnjointextwrapwraprP)rargsnew_command_argss rS_TranslateDeprecatedAliases#Command._TranslateDeprecatedAliases,s$(()@)@$Gab !D (d kktyy --M !% 7 7"&"3"3     KrUc >[TU]5 XlX lX0lX@lXPlXlX`lXl Xpl Xl SUl SUl SUlXlSUl[#UR$5UlU (a$U Hn UR&R)U 5 M URb[+UR5UlOSUlUR.c[1SUR$-5eUR&R3[4R65(+Ul[;UR8[<R>RASSS5S9q!UREU5Ul#URI5 UR.RKURM5S9Ul[OURF5UR.RP:d-[OURF5UR.RR:aURU5 UR$URV;aURY5 UR.RZ[\R^/S.nUR.R`[\R^S.n[bRd"URX5Ul3SUl4[kURURfUR&[m[nRp[B5UR UR,UR URURS 9 Ul9S Ul:[v(aU=RtS - sl:[xRzUl>U=RtUR|- sl:UR~(a+UR~HunnUS :XdUS :XdMSUl O [5RUlBg)aInstantiates a Command. Args: command_runner: CommandRunner (for commands built atop other commands). args: Command-line args (arg0 = actual arg, not command name ala bash). headers: Dictionary containing optional HTTP headers to pass to boto. debug: Debug level to pass in to boto connection (range 0..3). trace_token: Trace token to pass to the API implementation. parallel_operations: Should command operations be executed in parallel? bucket_storage_uri_class: Class to instantiate for cloud StorageUris. Settable for testing/mocking. gsutil_api_class_map_factory: Creates map of cloud storage interfaces. Settable for testing/mocking. logging_filters: Optional list of logging. Filters to apply to this command's logger. command_alias_used: The alias that was actually used when running this command (as opposed to the "official" command name, which will always correspond to the file name). perf_trace_token: Performance measurement trace token to use when making API calls. user_project: Project to be billed for this request. Implementation note: subclasses shouldn't need to define an __init__ method, and instead depend on the shared initialization that happens here. If you do define an __init__ method in a subclass you'll need to explicitly call super().__init__(). But you're encouraged not to do this, because it will make changing the __init__ interface more painful. FNzA"%s" command implementation is missing a command_spec definition.rdump_status_messages_file) quiet_moder)r)gss3debug http_headers trace_tokenperf_trace_token user_projectrzpython z-rz-RT)Csuper__init__command_runner unparsed_argsheadersrr r parallel_operationsr bucket_storage_uri_classgsutil_api_class_map_factoryr_recursion_requested all_versionsrseek_ahead_gsutil_apirTrPrX addFilterr>non_metadata_headersrr isEnabledForrEINFOrr*rrr ui_controllerrr ParseSubOpts_replacerlenrr$RaiseWrongNumberOfArgumentsException&_commands_with_subcommands_and_suboptsCheckArgumentsrr rrr GetApiMapgsutil_api_map project_idr r)sysstderr gsutil_api gsutil_cmdr<r GSUTIL_PATH gsutil_pathsub_optsr1 is_availablemultiprocessing_is_available)rrrrrr rrrlogging_filtersrr r  log_filter support_map default_mapounused_a __class__s rSrCommand.__init__?sFT G(LJ",2$$<!(D%!D$DD0!%D*$*;*;:K:KL MM++227<<@@DO DOO;?;;??%-/J%)<+,M 006DI))2222436D DII**333 DII**333 //1  K KK ..K ..ooK.77 ));EDDO'(E(E(,(;(;(, (9,/JJ )G.2jj595N5N484D4D9=9N9N595F5F HDODOz oo"o((DOOt'''O }}+!X 9T %)$ " ' -.;; %rUc[UR5URR:aEURRS:aSOSnSURURRU4-nO%SURURR 4-nUSURR <SUR<3- n[U5e)zCRaises exception for wrong number of arguments supplied to command.rsrz/The %s command requires at least %d argument%s.z,The %s command accepts at most %d arguments.z Usage: ( For additional help run: gsutil help )rrrrrPrrr)rtail_strmessages rSr ,Command.RaiseWrongNumberOfArgumentsExceptions 499~))222))22Q6BhB##T%6%6%?%?JKg@##T%6%6%?%?@Ag  (($*;*;==G 7 ##rUchSURR<SUR<3n[U5e)z?Raises exception for specifying an invalid argument to command.z&Incorrect option(s) specified. Usage: r9)rrrPr)rr;s rSRaiseInvalidArgumentException%Command.RaiseInvalidArgumentExceptions0!!00$2C2CEG 7 ##rUcUc URnOUn[R"X@RRURR=(d /5upVU(a(WWsUlUlU(aUR5 ggU(a [S5eWW4$![R a UR 5 Nkf=f)a9Parses sub-opt args. Args: check_args: True to have CheckArguments() called after parsing. args: List of args. If None, self.args will be used. should_update_sub_opts_and_args: True if self.sub_opts and self.args should be updated with the values returned after parsing. Else return a tuple of sub_opts, args returned by getopt.getopt. This is done to allow this method to be called from get_gcloud_storage_args in which case we do not want to update self.sub_opts and self.args. Raises: RaiseInvalidArgumentException: Invalid args specified. NzIRequested to check arguments but sub_opts and args have not been updated.) rgetoptrrr GetoptErrorr>r,r" TypeError)r check_argsrshould_update_sub_opts_and_argsrparsed_sub_opts parsed_argss rSrCommand.ParseSubOptss$ |iimm+%+]] **==    2 2 8b&:"o '!0+dmTY    HI I k ))   + ((*+sA B##$C  C cURR(dI[URURRS5(a[ SUR -5eURR(dJ[URURRS5(a[ SUR -5egg)aChecks that command line arguments match the command_spec. Any commands in self._commands_with_subcommands_and_subopts are responsible for calling this method after handling initial parsing of their arguments. This prevents commands with sub-commands as well as options from breaking the parsing of getopt. TODO: Provide a function to parse commands and sub-commands more intelligently once we stop allowing the deprecated command versions. Raises: CommandException if the arguments don't match. NzN"%s" command does not support "file://" URLs. Did you mean to use a gs:// URL?z1"%s" command does not support provider-only URLs.) rrrrrrrPrr rs rSr"Command.CheckArguments s    ) )TYYt00??@ABB @!../ 00    - -4#4#4#C#C#DEFF %'+'8'89 :: G .rUcX[UURUURURS9$)aHelper to instantiate gslib.WildcardIterator. Args are same as gslib.WildcardIterator interface, but this method fills in most of the values from instance state. Args: url_string: URL string naming wildcard objects to iterate. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. Returns: WildcardIterator for use by caller. )rr%rX)r@r(r%rX)r url_stringrs rSWildcardIteratorCommand.WildcardIterator$s+ "*"&///;-1__)-  66rUc $UR(dt[URUR[R "S5[ URURURURURS9 UlUR$)a?Helper to instantiate a Cloud API instance for a seek-ahead iterator. This must be separate from the core command.gsutil_api instance for thread-safety, since other iterators typically use that instance and the SeekAheadIterator operates in parallel. Returns: Cloud API instance for use by the seek-ahead iterator. dummyr) rr rr$rErFrrrr r r rs rSGetSeekAheadGsutilApiCommand.GetSeekAheadGsutilApi9sv  % %#4  ' '      G $  00&&00(( $*d   % %%rUc2[SUR-5e)zAbstract function in base class. Subclasses must implement this. The return value of this function will be used as the exit status of the process, so subclass commands should return an integer exit code (0 for success, a value in [1,255] for failure). z5Command %s is missing its RunCommand() implementation)rrPrs rS RunCommandCommand.RunCommandVs' ,.2.?.?@ AArUc V/n[[[U55n[U5(aUR(d [ S5eUHnUR 5(aUR5(aUR(a$SUlURUR5 MbURUR5RS/S9H#n[USSSURSS9n U"X 5 M% MURUR5 M [U5S:a[!UR"UR$UR&UR(UURUR*UR,=(d UR.US 9 n [1UR"UR$UR35UURUR*S 9n UR5UU UUR,(+U S 9 UR6(dUR,(d [ S 5egg) aSets the standard or default object ACL depending on self.command_name. Args: acl_func: ACL function to be passed to Apply. acl_excep_handler: ACL exception handler to be passed to Apply. url_strs: URL strings on which to set ACL. object_fields: If present, list of object metadata fields to retrieve; if None, default name expansion iterator fields will be used. Raises: CommandException if an ACL could not be set. z/Cannot operate on a mix of buckets and objects.*id bucket_fieldsFN)source_storage_urlis_multi_source_request!is_multi_top_level_source_requestnames_containerr`expanded_resultr)rcontinue_on_errorbucket_listing_fields)r) fail_on_errorseek_ahead_iterator'ACLs for some objects could not be set.)listmapr!r#rr IsCloudUrlIsBucketrerrLrM IterBucketsr storage_urlrrrPrrXr(rr`rrrQApplyrx) racl_funcacl_excep_handlerurl_strs object_fieldsmulti_threaded_url_argsurlsrhblrname_expansion_for_urlname_expansion_iteratorrcs rS ApplyAclFuncCommand.ApplyAclFuncgs"! ((3 4D%d++D4L4L N OO    cllnn  # # #/ ! ( ( 8**nn)kk?@c%8#&(-27 %%(__ $ && " T 2@ &&s~~6-0 "#q( 5    ** ++ // !  " "(( 22Nd6N6N - !/;    **  $ $ & !  " "(( * jj("#'#9#99%8 :  # #D,B,B F GG-C #rUcU(aUR(aeUnO URnUR(aSOSnURnURR SXE5 UR UR 5[R:Xa"UR S:waURXS5 OURXS5 [UR[[R"5S95 g)zSets the object ACL for the name_expansion_result provided. Args: name_expansion_result: NameExpansionResult describing the target object. thread_state: If present, use this gsutil Cloud API instance for the set. zdefault object ACLACLzSetting %s on %s...r) message_timeN)def_aclr(r`rXrfGetApiSelectorschemer r_SetAclXmlPassthrough_SetAclGsutilApir4 status_queuer&time)rrsrqr( op_stringrhs rSrrCommand.SetAclFuncs j??j(, $%I  4 4CKK*I;!!#**-@ d   1 C,*11)tyy{CErUcvURn[RUlURURUUR UR URS9 X2lg![a>nUR(a'SUl URRU5 SnANIeSnAff=f!X2lf=f)aSets the ACL for the URL provided using the XML passthrough functions. This function assumes that self.def_acl, self.canned, and self.continue_on_error are initialized, and that self.acl_arg is either an XML string or a canned ACL string. Args: url: CloudURL to set the ACL on. gsutil_api: gsutil Cloud API to use for the ACL set. Must support XML passthrough functions. )canned def_obj_aclproviderFN) prefer_apir rXmlPassThroughSetAclacl_argrrzr|r r`rxrXrv)rrhr(orig_prefer_apir\s rSr}Command._SetAclXmlPassthroughs!++O.)ooj%%dll&)-1[[26,,/2zz &;.   #(  ! .s0AA%% B-/3B("B0'B((B--B00B8c UR5(GaUR(aUR(aFURUR[ R "5URURS/S9 g [R"UR[ R5nU(dUR[5 [ R "US9nURURUURS/S9 g UR(aFURUR[ R "5URURS/S9 g [R"UR[ R5n[ R "US9nURURUURS/S9 g UR(aZURURUR [ R""5URUR$URS9 g [R"UR[ R5n[ R""US9nURURUR UURUR$S9 g ![&aneS nAf[(a>nUR*(a'S UlUR.R1U5 S nAg eS nAff=f) aHSets the ACL for the URL provided using the gsutil Cloud API. This function assumes that self.def_acl, self.canned, and self.continue_on_error are initialized, and that self.acl_arg is either a JSON string or a canned ACL string. Args: url: CloudURL to set the ACL on. gsutil_api: gsutil Cloud API to use for the ACL set. rX)canned_def_aclrfields)defaultObjectAcl)rr) canned_aclrr)r)r generationr)rrNF)rhrzr PatchBucket bucket_nameapitools_messagesBucketrr|r= JsonToMessageObjectAccessControlrr?BucketAccessControlPatchObjectMetadatareObjectrr r r`rxrXrv) rrhr(rbucket_metadata bucket_acl object_aclobject_metadatar\s rSr~Command._SetAclGsutilApis]=  << [[  " "3??#4#;#;#=26,,,/JJ+/& # 2 )66 /CCEK  !89/66!,.O  " "3??#2,/JJ+/& # 2 [[  " "3??#4#;#;#=.2ll,/JJ+/& # 2 (55 /CCEJ/66:FO  " "3??#2,/JJ+/& # 2 ;;  ( (),):)A)A)C25**47NN48LL ) B&33ll-AAC*-44D/  ( (),)825**47NN ) D     #(  ! sMA=J3BJ3AJ3A*J3 A*J34A>J33 L =J>> L  3LLL cURSnURSSn[U5(d[SUR-5e[R R U5(a=[R"US[5nUR5nSSS5 SUl OX[R"USURSURS9nUR!5nX7;a[SU-5eS Ul S UlX0lUR'XU5 UR"(dUR((d [S 5egg!,(df  N=f) zSets ACLs on the self.args using the passed-in acl function. Args: acl_func: ACL function to be passed to Apply. acl_excep_handler: ACL exception handler to be passed to Apply. rrNz,"%s" command spanning providers not allowed.rF)rvalidaterzInvalid canned ACL "%s".Trd)rr"rrPrbrcisfilecodecsopenr/readrr storage_urirr canned_aclsrxrrur`)rrlrmrurl_argsfrrs rSSetAclCommandHelperCommand.SetAclCommandHelper<s2iilGyy}H $H - - K!../ 00  ww~~g ;;wT *a&&( +dk$$ 1+ #'#@#@ Bk  ++-k  #9GCDDdk $DLh8<  # #D,B,B F GG-C #- + *s E EcSSKJn U(a?URRSR [ R "S555 gg)zWarns service account users who have received an AccessDenied error. When one of the metadata-related commands fails due to AccessDenied, user must ensure that they are listed as an Owner in the API console. r)IS_SERVICE_ACCOUNTraIt appears that your service account has been denied access while attempting to perform a metadata operation. If you believe that you should have access to this metadata (i.e., if it is associated with your account), please make sure that your service accounts email address is listed as an Owner in the Permissions tab of the API console. See "gsutil help creds" for further information. N)$gcs_oauth2_boto_plugin.oauth2_pluginrrXwarningrrr)rrs rS_WarnServiceAccountsCommand._WarnServiceAccountsgsDH kk$)) --L MNOrUcURU5n[U5nURRUR5[ R :XaZURS:waJURRUURURS9n[UR55 gURS:Xa;URRnU(dUR R#SXU5 O.URR$nU(dUR'U5 [[(R*"U55 g![anUR5 eSnAff=f)zwPrints the standard or default object ACL depending on self.command_name. Args: url_str: URL string to get ACL for. r)rrNraNo default object ACL present for %s. This could occur if the default object ACL is private, in which case objects created in this bucket will be readable only by their creators. It could also mean you do not have OWNER permission on %s and therefore do not have permission to read the default object ACL. It could also mean that %s has Bucket Policy Only enabled and therefore object ACLs and default object ACLs are disabled (see https://cloud.google.com/storage/docs/bucket-policy-only).)#GetAclCommandBucketListingReferencer!r(r{r|r rXmlPassThroughGetAclrzprintto_xmlr rrP root_objectrrXrr_ListAccessControlsAclr=JsonFromMessage)rurl_strrrrhr_s rSGetAndPrintAclCommand.GetAndPrintAcls"  2 27 ;C w 'C &&szz2kooE doo223?C||,>6@@      __ 5 5  ! !  ! !%%6'' rUc[U5nUR5(a)[URU5R S/S95nO>UR S:XaS/nOS/n[URU5R US95nUR5(a [S5eUR5(a[U<SUR <S35e[U5S $) a Gets a single bucket listing reference for an acl get command. Args: url_str: URL string to get the bucket listing reference for. Returns: BucketListingReference for the URL string. Raises: CommandException if string did not result in exactly one reference. r)rarrrYNo URLs matchedz8 matched more than one URL, which is not allowed by the  commandr) r!rrrM IterObjectsrPriIsEmptyr HasPluralityre)rr wildcard_urlplurality_iterrZs rSr+Command.GetAclCommandBucketListingReferences(0L1    ( 4 4%*G 5 -.n   h &+,  1    ( 4 4) 5 +,n . //""$$  1 1 3 44   ""rUcURXS9nUR5(a[U<SUR<S35e[ U5Sn[ UR 5UR4$)a0Gets a single bucket URL based on the command arguments. Args: arg: String argument to get bucket URL for. bucket_fields: Fields to populate for the bucket. Returns: (StorageUrl referring to a single bucket, Bucket metadata). Raises: CommandException if args did not match exactly one bucket. rYz8 matched more than one URL, which is not allowed by the rr)GetBucketUrlIterFromArgrrrPrer!rLr)rargrZplurality_checkable_iteratorrrs rSGetSingleBucketUrlFromArg!Command.GetSingleBucketUrlFromArgst$(#?#? $@$* #0022 !4#4#46 77 + ,Q /C  / @@rUc.[U5nUR5(aUR5(a[SUR-5e[ UR U5RUS95nUR5(a [S5eU$)a Gets a single bucket URL based on the command arguments. Args: arg: String argument to iterate over. bucket_fields: Fields to populate for the bucket. Returns: PluralityCheckableIterator over buckets. Raises: CommandException if iterator matched no buckets. z""%s" command must specify a bucketrYr) r!rgrrrPrrMrir)rrrZarg_urlrs rSrCommand.GetBucketUrlIterFromArgs#3'G     7#3#3#5#5 A!../ 00$> c"..].K$M #++-- . // ''rUc[RnU(a%UHnXRR5 M! 0[lS[lgrW)r provider_pool connectionclose)rconnection_poolis rS_ResetConnectionPoolCommand._ResetConnectionPoolsC !..O!%%++- "J JrUc UR(dU(aU(dC[RRSS[R RR 5nUS:a[SU-5eU(dC[RRSS[R RR5nUS:a[SU-5eOSnSn[5upVU(aTUS:aN[SR[R"SU<S S R[55<S 3555eURS :HnU(a.US :Xa(US:a"U(aURR!S5 URR#SU5 URR#SU5 X4$)aDetermines the values of process_count and thread_count. These values are used for parallel operations. If we're not performing operations in parallel, then ignore existing values and use process_count = thread_count = 1. Args: process_count: A positive integer or None. In the latter case, we read the value from the .boto config file. thread_count: A positive integer or None. In the latter case, we read the value from the .boto config file. parallel_operations_override: Used to override self.parallel_operations. This allows the caller to safely override the top-level flag for a single call. print_macos_warning: Print a warning about parallel processing on MacOS if true. Returns: (process_count, thread_count): The number of processes and threads to use, respectively. rparallel_process_countrz$Invalid parallel_process_count "%d".parallel_thread_countz#Invalid parallel_thread_count "%d".rz/It is not possible to set process_count > 1 on z0. Please update your config file(s) (located at z, z') and set "parallel_process_count = 1".rmacOSa^If you experience problems with multiprocessing on MacOS, they might be related to https://bugs.python.org/issue33725. You can disable multiprocessing by editing your .boto config or by adding the following flag to your command: `-o "GSUtil:parallel_process_count=1"`. Note that multithreading is still available even if you disable multiprocessing. zprocess count: %dzthread count: %d)rrrrrcommandsDEFAULT_PARALLEL_PROCESS_COUNTrDEFAULT_PARALLEL_THREAD_COUNTr6rrrr,recursive_apply_levelrXrfr)r process_count thread_countparallel_operations_overrideprint_macos_warningshould_prohibit_multiprocessingos_nameis_main_threads rS_GetProcessAndThreadCount!Command._GetProcessAndThreadCount(s8 #?  ** . NN ! ! @ @B   E, -. . {{)) - NN ! ! ? ?A   D+ ,- -  ml/L/N,#&=1+< TYY -- "<">?A BC DD //14Nw'1ma6GN kk CD KK)=9KK(,7  ((rUcp[ [R5 [R5nSSS5 [R"U5nU=R S- slSUlSUlSUlU[W'S[U'S[U'S[U'/[U'U$!,(df  N}=f)zBSet up the state for a caller id, corresponding to one Apply call.NrrFr)rr IncrementGetValuecopyrrXr(rrrrrr)r caller_idr[s rS_SetUpPerCallerStateCommand._SetUpPerCallerStatens !!##,,.i  ))D/C" CJCN $CIiK $)y!*+Y'*,Y' 3 s )B'' B5c/n[5n[RU5 [R 5 [R 5[ :a [S5e[U5Hfn[[5n[R"URX!UU4S9nSUlURU5 [5 UR!5 Mh [#XE5n [RU 5 g)z7Create a new pool of processes that call _ApplyThreads.z,Recursion depth of Apply calls is too great.)targetrTN)rrrrrrMAX_RECURSIVE_DEPTHrrangerrr2Process _ApplyThreadsdaemonr~start _ConsumerPool) r num_processes num_threadsr processes task_queuerrp consumer_pools rS_CreateNewConsumerPoolCommand._CreateNewConsumerPoolsI)+Jz"))+"++-0CC K LL = !!.1 ! ) )1C1C0;0E0<0> ?aahqggi"")8M-(rUc$\rSrSrSrSrSrSrSrg)Command.ParallelOverrideReasonizAEnum class to describe purpose of overriding parallel operations.slicespeedperfdiagrlN) __name__ __module__ __qualname____firstlineno____doc__SLICESPEEDPERFDIAG__static_attributes__rlrUrSParallelOverrideReasonrsK E EHrUrc 8U(a$0n UHn [X 5X'[X S5 M URXxU5upxURS:H=(a URS:HnU(a [ UUS9 U =(a Xx-S:Hn U(a*UR (dUS:a[URS9 UR5nU(aU[U'UH n S[X4'M UR (aUOSnUU-S:a+URUUUUUUUU U U US9 U(a [5 OURXX?XYU 5 U(a1UH+n W U [RX45-n[X U5 M- U (a[ RU5$g)aCalls _Parallel/SequentialApply based on multiprocessing availability. Args: func: Function to call to process each argument. args_iterator: Iterable collection of arguments to be put into the work queue. exception_handler: Exception handler for WorkerThread class. shared_attrs: List of attributes to manage across sub-processes. arg_checker: Used to determine whether we should process the current argument or simply skip it. Also handles any logging that is specific to a particular type of argument. parallel_operations_override: A string (see ParallelOverrideReason) describing the reason to override self.parallel_operations. This allows the caller to safely override the top-level flag for a single call. process_count: The number of processes to use. If not specified, then the configured default will be used. thread_count: The number of threads per process. If not specified, then the configured default will be used.. should_return_results: If true, then return the results of all successful calls to func in a list. fail_on_error: If true, then raise any exceptions encountered when executing func. This is only applicable in the case of process_count == thread_count == 1. seek_ahead_iterator: If present, a seek-ahead iterator that will provide an approximation of the total number of tasks and bytes that will be iterated by the ProducerThread. Used only if multiple processes and/or threads are used. Returns: Results from spawned threads. rr)rrr)rX)rcrN)r}setattrrrsequential_caller_idrr.r1rXrrr_ParallelApply_AggregateThreadStats_SequentialApplyrr)rfunc args_iteratorexception_handler shared_attrs arg_checkerrrrshould_return_resultsrbrcoriginal_shared_vars_valuesnamerrusable_processes_count final_values rSrk Command.Applysf$&!$,3D,?#) A  %)$B$B%A%C!]00A56//25! .:< "I}'Cq'HM t@@ +$++>))+I (49%$-.)* "&!B!B,HI,,q0         1'C E  D1B' O$348?;N;N  < K(  % ) )) 44rUc2[RRSSS5S:Xa%[RRSSS5S:XdMURR S[ R "SURR-5-S-5 gg) z2Outputs a suggestion to the user to use gsutil -m.rrrrrrz==> NOTE: You are performing a sequence of gsutil operations that may run significantly faster if you instead use gsutil -m %s ... Please see the -m section under "gsutil help options" for further information about when gsutil -m can be advantageous.N) rrrrXrfrfillrrPrs rS_MaybeSuggestGsutilDashM Command._MaybeSuggestGsutilDashM6s KK  x)A1 E J KK  x)@! D I kkthmm B    ( (  )** -1 12 JrUc [SSURURURURS9n[ U5nSn [ U5n U S- n U [:Xd U [-S:XaUR!5 U"X 5(a[#XXCXeU5n UR%X5 Mf![a n Sn A O~Sn A f[ann [5 U(aeU"X 5 OG![a:n URRSU[R"55 Sn A OSn A ff=fSn A MSn A ff=fU [-nU['5:aUR!5 [)UR*R,[/[0R0"555 UR2R5X@5 UR7U5 g)zPerforms all function calls sequentially in the current thread. No other threads or processes will be spawned. This degraded functionality is used when the multiprocessing module is not available or the user requests only one thread and one process. NF)rr r r r4Caught exception while handling exception for %s: %sr) WorkerThreadrr r r iternext StopIteration Exception_IncrementFailureCountrXr traceback format_exc#OFFER_GSUTIL_M_SUGGESTION_THRESHOLD#OFFER_GSUTIL_M_SUGGESTION_FREQUENCYr)Task PerformTaskr;r4r(rr%rshared_vars_updaterUpdate_ProcessSourceUrlTypes)rrrrrr r!rb worker_threadsequential_call_countrr\rtask#lines_since_suggestion_last_printeds rSrCommand._SequentialApplyBs!!&)-)B)B262G2G-1-=-=.2.?.? AM 'M M" q  #F F "E E J %%' T D ) G!!$-?       ( d & ( KK  G$$& ( ( (  4,A+N,O'*ln< ##%$//66&tyy{35 %%,,Y=  .sH B++ D5: D5D0C#"D0# D'-0D"D0"D''D00D5c  URS:Hn XRR:XaBURS::a2[R [ [ R "5S55 [(d=U (a6[R[R4Hn [U [SS9 M [(dUS:a[R[55 O[!5n[RU5 [#UUR$UUR&UR(UR*[UR,UR.UR0UR2S9 US:aU (d[4R75 UR[8R;5:ah[< U (aUR?Xg[5 O;[@RCSS9 [<RE5 [<RG5 SSS5 U (d[4RI5 OU (d[4R75 UR[K5:a[M5 [!5n[RU5 [#UUR$UUR&UR(UR*[UR,UR.UR0UR2S9 [4RI5 [URnU (a U (dSn [O[PRP"U5UUUUUUUU U U (a[OSS9 nSnU (a#[S[[TRV[X5n[< [ZU(a SSS5 OUS:a^U (aW[@R;5(a>[@RC5 UR?Xg[5 [<RE5 [<RG5 SSS5 MU (aB[][[^5 URa[bS 9 UReURf5 URh(a URheURj(aU (a URjeU (a5U (d-[][[m[ R "555 ggg!,(df  GN=f!U (d[4RI5 ff=f![4RI5 f=f!,(df  GN$=f) acDispatches input arguments across a thread/process pool. Pools are composed of parallel OS processes and/or Python threads, based on options (-m or not) and settings in the user's config file. If only one OS process is requested/available, dispatch requests across threads in the current OS process. In the multi-process case, we will create one pool of worker processes for each level of the tree of recursive calls to Apply. E.g., if A calls Apply(B), and B ultimately calls Apply(C) followed by Apply(D), then we will only create two sets of worker processes - B will execute in the first, and C and D will execute in the second. If C is then changed to call Apply(E) and D is changed to call Apply(F), then we will automatically create a third set of processes (lazily, when needed) that will be used to execute calls to E and F. This might look something like: Pool1 Executes: B / \ Pool2 Executes: C D / \ Pool3 Executes: E F Apply's parallelism is generally broken up into 4 cases: - If process_count == thread_count == 1, then all tasks will be executed by _SequentialApply. - If process_count > 1 and thread_count == 1, then the main thread will create a new pool of processes (if they don't already exist) and each of those processes will execute the tasks in a single thread. - If process_count == 1 and thread_count > 1, then this process will create a new pool of threads to execute the tasks. - If process_count > 1 and thread_count > 1, then the main thread will create a new pool of processes (if they don't already exist) and each of those processes will, upon creation, create a pool of threads to execute the tasks. Args: caller_id: The caller ID unique to this call to command.Apply. See command.Apply for description of other arguments. rrT)is_final_handler) rrr$rrrr r r ) reset_valueN)rcrtimeout)7rrrrputr'rr<signalSIGINTSIGTERMrrrrrr WorkerPoolrXrr$rrr r r racquirerrrrrReset notify_allwaitrelease_GetCurrentMaxRecursiveLevel"_IncrementCurrentMaxRecursiveLevelProducerThreadrr+r&r'rrr4r8rr7r;runknown_exceptioniterator_exceptionr%)rrrrrr rrr!rbrcrr signal_numrproducer_thread ui_threads rSrCommand._ParallelApplyst//14N$(C(C(I(II ""a'5diik4HI :. 7*j % $ /+9't FOI,cjj-Hi  ! i (  " !aN&&((    !  % %m&7 9 + + - ##% " $-/HInn3n4 !!/"?"?@ ((  - --))m  . ..:-|DIIK/HI;~k&% $ , , .0 % , , .Z " !sF/AT2AS4 T8BTT$'T= Uc[U[5(dg[URURUR UR S9 g)z6Logs the URL type information to analytics collection.N)is_daisy_chain has_file_src has_cloud_srcprovider_types) isinstancerrrZr[r\r])rrs rSr;Command._ProcessSourceUrlTypests? m%8 9 9 }/K/K-:-G-G.;.I.I/s rSrCommand._ApplyThreads}sV 1 OMO  )*  mmL";<+ !623J !11,? )!%!>!>**jj!))..$$&& (KL  ((%(8 4!((%(88^^ d / / D!    ") rU)rrrrrrrrrrxr_r(rr$r)r+rrXr.rrr r%rrrrr,r rr )NNNN)FNT)FrW)TNN)3r r rrrrr!rr staticmethodr.rrpropertyrPrrrr r>rr"rMrQrTrurrr}r~rrrrrrrrrrrobjectrrjrkr)rrr;rr __classcell__r5s@rSrrs~+,,@( '+-1!"!'+-$)(-'('+'+/3+/DD:*01,,8 $"& $ M=^ $$$37'*R:26*&:A*"& RHhE4.:HT)HVO0)1V0$#LA,(8 !$59 D)L#J),v&)-"' $A5F 2E/d*.26hJTMJ#J#rUrc \rSrSrSrSrSrg)ricXlX lgrWrr)rrrs rSr_ConsumerPool.__init__s N OrUcTURHn[UR5 M grW)rrpid)rprocesss rSShutDown_ConsumerPool.ShutDowns>>'++"rUrrN)r r rrrrwrrlrUrSrrs !rUrc\rSrSrSrSrg)r7ia!Task class representing work to be completed. Args: func: The function to be executed. args: The arguments to func. caller_id: The globally-unique caller ID corresponding to the Apply call. exception_handler: The exception handler to use if the call to func fails. should_return_results: True iff the results of this function should be returned from the Apply call. arg_checker: Used to determine whether we should process the current argument or simply skip it. Also handles any logging that is specific to a particular type of argument. fail_on_error: If true, then raise any exceptions encountered when executing func. This is only applicable in the case of process_count == thread_count == 1. rlN)r r rrrrrlrUrSr7r7s  rUr7zUfunc args caller_id exception_handler should_return_results arg_checker fail_on_errorc"[X[5$)aInitializes and runs the seek-ahead thread. We defer starting this thread until it is needed, since it is only useful when the ProducerThread iterates more results than it can store on the global task queue. Args: seek_ahead_iterator: Iterator that yields SeekAheadResults. seek_ahead_thread_cancel_event: threading.Event for signaling the seek-ahead thread to terminate. Returns: The thread object for the initialized thread. )rr)rcseek_ahead_thread_cancel_events rS_StartSeekAheadThreadr|s* ,* ,,rUc:^\rSrSrSrSU4SjjrSrSrU=r$)rRi z[[U] 5 X@lXlX lX0lXPlXlXpl X`l Xl [5Ul SUlSUlSUlXlXlUR'5 g)aInitializes the producer thread. Args: cls: Instance of Command for which this ProducerThread was created. args_iterator: Iterable collection of arguments to be put into the work queue. caller_id: Globally-unique caller ID corresponding to this call to Apply. func: The function to be called on each element of args_iterator. task_queue: The queue into which tasks will be put, to later be consumed by Command._ApplyThreads. should_return_results: True iff the results for this call to command.Apply were requested. exception_handler: The exception handler to use when errors are encountered during calls to func. arg_checker: Used to determine whether we should process the current argument or simply skip it. Also handles any logging that is specific to a particular type of argument. fail_on_error: If true, then raise any exceptions encountered when executing func. This is only applicable in the case of process_count == thread_count == 1. seek_ahead_iterator: If present, a seek-ahead iterator that will provide an approximation of the total number of tasks and bytes that will be iterated by the ProducerThread. status_queue: status_queue to inform task_queue estimation. Only valid when calling from the main thread, else None. Even if this is the main thread, the status_queue will only properly work if args is a collection of NameExpansionResults, which is the type that gives us initial information about files to be processed. Otherwise, nothing will be added to the queue. TN)r rRrrr[rrrr rr!rb_SharedVariablesUpdatershared_variables_updaterrrSrTrcrr) rr[rrrrr!rr rbrcrr5s rSrProducerThread.__init__ s|T .$(*IH&N O".!6&$;$=D!DK!D"D2$JJLrUc  SnSnSnSnSnSnSnSnSn [UR5Ul[UR5nUR%URU5(GaMUS- nUR&(Ga-US-(dr[)U[*5(d*[)U[,5(d[)U[.5(a3[1UR&[3X[4R4"555 [)U[*5(d[)U[,5(aIUR6(a7[8R:"UR65n SU ;aU [=U S5- n O>[)U[.5(a)UR>(aU [=UR>5- n U(dUc [A5nUS::aSnO|X:awURB(ad[DRF"5n[IURBU5n[JRLROSS S5(aURQ[RS 9 SnUn[UURXR"URURVUR$UR 5nU(aURXR[U5 GM![a n Sn A GOSn A f[an [ 5 UR (aXleURURU 5 O[![aNn URRRSUR[R"55 Sn A OSn A ff=fURR!UR"UR5 Sn A GMxSn A ff=f![a!n UR(dXl.Sn A OSn A ff=fU[^UR"'U(d[US[`UR"SSSS5nURXR[U5 Ub#URc5 URQ[RS 9 UR&(ar[)U[*5(d*[)U[,5(d[)U[.5(a3[1UR&[3UW [4R4"5SS 95 [eUR"[fROUR"55 g!U[^UR"'U(d[US[`UR"SSSS5nURXR[U5 Ub#URc5 URQ[RS 9 UR&(ar[)U[*5(d*[)U[,5(d[)U[.5(a3[1UR&[3UW [4R4"5SS 95 [eUR"[fROUR"55 f=f) NrFTr,rdsizertask_estimation_forcerD)finished)4r.rr/r0r1r2rbrTrr[rXrrr3r4rr:rr rr^rrr9r4r(rr_jsonloadsint copy_sizerrcrEventr|rrrrr5r7r!rrFrSrr8r _NotifyIfDoner) r num_taskscur_task last_taskrseek_ahead_threadr{seek_ahead_thread_consideredr total_sizer\rjson_expanded_results rSrunProducerThread.runIsIHI $%)"#( DpBj 2 23d  d(()$$   DHHd + + q.)    s?T#677T>22T#344%%%))MO4!455400%%'+zz$2F2F'G$11$8$@ AA*D"233c$..11 -(0*E*G'(A--1*7))1:1B.$9,,.L%N!;;??8-DdKK#((1H(I-1*)$))T>>00$2L2L**D,>,>@( OO   *A     "   &' #  5$$TXXq1 5hhoo##I))Y11355 5  ) ) 0 0 J  x # $ $!" #%.k$..! 7dD* oo(#  &&**, '>?    d/ 0 0JN5$5$'1$8H'I'I    !)","&))++/ 1 2DNN,00@B?%.k$..! 7dD* oo(#  &&**, '>?    d/ 0 0JN5$5$'1$8H'I'I    !)","&))++/ 1 2DNN,00@BsN.J7I2N.7 N+N.T  N+"N&7LN& M,AM'"N&'M,,3N&N.&N++N.. O8OT OT D2X?)r rrr[rrrbrrTrcrr!rrrSrj) r r rrrrrrrnros@rSrRrR s"D$( :xyByBrUrRc<\rSrSrSrSSjrSrSrg)rJiz3Pool of worker threads to which tasks can be added.Nc USLUSL:wdeXlXlXlXlU=(d [ 5Ul/Ul[U5Hsn [UR UUUUUUURURURURS9 nUR RU5 UR5 Mu g)Nra) rr r r rrthreadsrr-rr)rrrXrbrrr$rrrr r r rr<s rSrWorkerPool.__init__s,  $**< == =L,"$ 9$7$9DODL < " // +#;'#,,00&&(( *m ll-(!rUc:URRU5 g)zCAdds a task to the task queue; used only in the multi-process case.N)rrF)rr>s rSreWorkerPool.AddTasksOOrU)rr rrr r ) NNNNrNNNNN)r r rrrrrerrlrUrSrJrJs0; !%(,"  $ ,\rUrJcn^\rSrSrSrS U4Sjjr\S5r\S5rSr Sr Sr U=r $) r-ia.Thread where all the work will be performed. This makes the function calls for Apply and takes care of all error handling, return value propagation, and shared_vars. Note that this thread is NOT started upon instantiation because the function- calling logic is also used in the single-threaded case. c >[[U] 5 [R"5Ul[ R "5UlXlX0l SUl 0Ul [5Ul XlXlXlXlSUlU(aGU(a?['UUUUUURURUR UR"S9 Ulggg)a6Initializes the worker thread. Args: task_queue: The thread-safe queue from which this thread should obtain its work. logger: Logger to use for this thread. worker_semaphore: threading.BoundedSemaphore to be released each time a task is completed, or None for single-threaded execution. bucket_storage_uri_class: Class to instantiate for cloud StorageUris. Settable for testing/mocking. gsutil_api_map: Map of providers and API selector tuples to api classes which can be used to communicate with those providers. Used for the instantiating CloudApiDelegator class. debug: debug level for the CloudApiDelegator class. status_queue: Queue for reporting status updates. user_project: Project to be billed for this request. TN)rr r r r )r r-rrbgetpidrur init_timerrbrcached_classesrr9rr r r thread_gsutil_apir ) rrrXrbrr$rrrr r r r5s rSrWorkerThread.__init__ s: ,&(yy{DHYY[DN O,DKD68DL,"$ "DN0 "   ||00&&(( *d%3rUc\[RURUR45c2[ UR 5[URUR4'[URUR4nUR 5 U[URUR4'g)zEUpdate the thread_stats AtomicDict before task_queue.get() is called.N)rrruident _ThreadStatrStartBlockedTimer thread_stats rS_StartBlockedTimeWorkerThread._StartBlockedTimeG s~4::./7-8-HlDHHdjj)*$**56K  "+6L$((DJJ'(rUc[URUR4nUR5 U[URUR4'g)zDUpdate the thread_stats AtomicDict after task_queue.get() is called.N)rrurEndBlockedTimers rS_EndBlockedTimeWorkerThread._EndBlockedTimeS s>$**56K +6L$((DJJ'(rUcxURnURX!RURS9nUR(a[ R X4//S9 UR(aURR!5 UR"R%X25 [&R US5n[)X75 g![an[5 UR(aeURX%5 SnAN![a>nURRSU[R"55 SnASnANSnAff=fSnAff=f!UR(aURR!5 UR"R%X25 [&R US5n[)X75 f=f)zMakes the function call for a task. Args: task: The Task to perform. cls: The instance of a class which gives context to the functions called by the Task's function. E.g., see SetAclFuncWrapper. rp) default_valuer,Nr)rrrrr!rrr1r2rbrrXrr3r4rbrOr9r:rr)rr>r[rresultsr\rnum_dones rSr8WorkerThread.PerformTaskZ s_I) #yyt7M7M Ng # # **9i9; + =    %%' %%i5 *33IqAhI(-  &    &   ( ( & **  Et""$ & & & &    %%' %%i5 *33IqAhI(sUA C E  E'C=8E= E0E7E;EEEE  EA)F9cUR5 URR5nUR5 UR[ :XaMQUR nURRUS5nU(dE[R"[U5n[UR5Ul X0RU'URX5 MrW)rrrrrr8rrrrrTrPrXr8)rr>rr[s rSrWorkerThread.run s   __ "d  / / ..i    # #It 4c ii ),-,S-=-=> ),I& t!# rU) rrrrr rur9rrr r rb) NNNrNNNNN) r r rrrrrrrr8rrrnros@rSr-r-sc&!%(,"  $ 9*v 7 777 $)L""rUr-c0\rSrSrSrSrSrSrSrSr g) ri z1Stores thread idle and execution time statistics.c`SUlXl[R"5UlSUlg)Nr)total_idle_timeend_block_timerstart_block_timetotal_execution_time)rrs rSr_ThreadStat.__init__ s*D$!IIKD !DrUc[R"5UlURUR- nU=RU- slgrW)rrrr)r exec_times rSr_ThreadStat.StartBlockedTime s7 IIKD%%(;(;;I*rUc[R"5UlURUR- nU=RU- slgrW)rrrr)r idle_times rSr_ThreadStat.EndBlockedTime s7))+D##d&;&;;II%rUcURUR:a"U=RXR- - slgU=RXR- - slg)z)Decide final stats upon Apply completion.Nrrrr)rend_times rS AggregateStat_ThreadStat.AggregateStat sN T222 8.A.A#AA  h)>)>>>rUrN) r r rrrrrrrrrlrUrSrr s9 "+ & ?rUrc[R"5nS=p[R5H0nURU5 XR- nX#R - nM2 [ UUS9 g)zAt the end of the top-level Apply call, aggregate the thread stats dict. This should only be called in the main process and thread because it logs to the MetricsCollector. r)thread_idle_timethread_execution_timeN)rrvaluesrrrr)cur_timerrrs rSrr se YY[(+,,/!((*kh'222O<<<+4HJrUc$\rSrSrSrSrSrSrg)ri aBUsed to update shared variable for a class in the global map. Note that each thread will have its own instance of the calling class for context, and it will also have its own instance of a _SharedVariablesUpdater. This is used in the following way: 1. Before any tasks are performed, each thread will get a copy of the calling class, and the globally-consistent value of this shared variable will be initialized to whatever it was before the call to Apply began. 2. After each time a thread performs a task, it will look at the current values of the shared variables in its instance of the calling class. 2.A. For each such variable, it computes the delta of this variable between the last known value for this class (which is stored in a dict local to this class) and the current value of the variable in the class. 2.B. Using this delta, we update the last known value locally as well as the globally-consistent value shared across all classes (the globally consistent value is simply increased by the computed delta). c0UlgrWlast_shared_var_valuesrs rSr _SharedVariablesUpdater.__init__ s "$DrUc[RUS5nU(a]UHVnX4nURRUS5n[X$5U- nXv-URU'[R XW5 MX gg)z.Update any shared variables with their deltas.Nr)rrrr}rr)rrr[ shared_varsr#key last_valuedeltas rSr:_SharedVariablesUpdater.Update sz&**9d;K$0044S!< "Z/+0+=##C( !!#-rUrN)r r rrrrr:rrlrUrSrr s0%.rUrc[UnX!:Xa4US:a-[ S[U'[R5 SSS5 ggg!,(df  g=f)aNotify any threads waiting for results that something has finished. Each waiting thread will then need to check the call_completed_map to see if its work is done. Note that num_done could be calculated here, but it is passed in as an optimization so that we have one less call to a globally-locked data structure. Args: caller_id: The caller_id of the function whose progress we're checking. num_done: The number of tasks currently completed for that caller_id. rTN)rrrrM)rr num_to_dos rSrr sK)$)yA~ &*#'')  .  s A Ac[HnUR5 M [HnUR5 M [R5 g! MP=f! g=f)zEShut down all processes in consumer pools in preparation for exiting.N)rcancel_join_threadrrwr)qrs rSrr s[ a   &m& ((*    sAAAAc*[R5$rW)rrrlrUrSrPrP! s $ - - //rUc,[R5 grW)rrrlrUrSrQrQ& s'')rUc,[R5 grW)rrrlrUrSr2r2+ rUc,[R5 grW)r DecrementrlrUrSDecrementFailureCountr0 rrUc*[R5$)zAReturns the number of failures processed during calls to Apply().)rrrlrUrSGetFailureCountr5 s    !!rUc,[R5 g)zEResets the failure_count variable to 0 - useful if error is expected.N)rrLrlrUrSResetFailureCountr; srUrW)r __future__rrrrr collectionsrrrArrErbrGr&rrrr3rboto.storage_urirrgslib.cloud_apir r r gslib.cloud_api_delegatorr gslib.cs_api_mapr rgslib.exceptionrgslib.help_providerr gslib.metricsrrgslib.name_expansionrrrrr"gslib.plurality_checkable_iteratorrgslib.seek_ahead_threadrgslib.sig_handlingrrrrrgslib.storage_urlrr r!r"r#"gslib.third_party.storage_apitoolsr$rgslib.thread_messager%r&r'r(gslib.ui_controllerr)r*r+gslib.utils.boto_utilr,r-gslib.utils.constantsr.r/&gslib.utils.parallelism_framework_utilr0r1r2r3r4r5r6r7r8gslib.utils.rsync_utilr9gslib.utils.shim_utilr:gslib.utils.system_utilr;r<gslib.utils.translation_helperr=r>r?gslib.wildcard_iteratorr@ six.movesrArCryptorBr| ImportErrorr5r6rTr]rjrnrtryrr~rrrrrrrrrrrrrmrr7r|ThreadrRrJr-rrrrrrPrQr2rrrrlrUrSrs'%' "    ' 1-,7(0,,45/464?I38/*=4*.26;W-0:61,(<C(&-=WJJHJPIL3;0.9@B:$+ '(#&*#" JJ"   %*!? T]enen4 E"E"/-e$"E*"E* W#l5W#x.Fv 4,2xBY%%xBv33lW"9##W"t$?&$?N J +.f+.\*.  0 *   " _N,s:KKK