SrSSKJr SSKJr SSKJr SSKJr SSKrSSKJr SSK r SSK r SSK r SSK r SSK r SSKrSSKrSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSS KJ r SS KJ r SS K!J"r" SS K#J$r$ SSK%r%SSK&r&SS K'J(r( SS K'J)r) SSK'J*r* SSK'J+r+ SSK'J,r, SSK'J-r- SSK'J.r. SSK'J/r/ SSK'J0r0 SSK'J1r1 SSK'J2r2 SSK'J3r3 SSK4J5r5 SSK6J7r7 SSK6J8r8 SSK6J9r9 SSK6J:r: SSK6J;r; SSK6Jr> SS K?J@r@ SS!KAJBrB SS"KAJCrC SS#KAJDrD SS$KEJFrF SS%KGJHrH SS&KGJIrI SS'KGJJrJ SS(KGJKrK SS)KGJLrL SS*KMJNrN SS+KMJOrO SS,KPJQrQ SS-K&JRrR SS.KSJTrT SS/KSJUrU SS0KSJVrV SS1KSJWrW SS2KXJYrZ SS3K[J\r\ SS4K[J]r] SS5K^J_r_ SS6K^J`r` SS7K^Jara SS8K^Jbrb SS9K^Jcrc SS:K^Jdrd SS;K^Jere SSK^Jhrh SS?K^Jiri SS@K^Jjrj SSAKkJlrl SSBKkJmrm SSCKkJnrn SSDKkJoro SSEKpJqrq SSFKpJrrr SSGKpJsrs SSHKpJtrt SSIKpJuru SSJKvJwrw SSKKvJxrx SSLKyJzrz SSMKyJ{r{ SSNKyJ|r| SSOK}J~r~ SSPK}Jr SSQK}Jr SSRK}Jr SSSKJr SSTKJr SSUKJr SSVKJr SSWKJr SSXKJr SSYKJr SSZKJr SS[KJr SS\KJr SS]KJr SS^KJr SS_KJr SS`KJr SSaKJr SSbKJr SScKJr SSdKJr SSeKJr SSfKJr SSgKJr SShKJr SSiKJr SSjKJr SSkKJr SSlKJr SSmKJr SSnKJr SSoKJr SSpKJr SSqKJr SSrKJr SSsKJr SStKJr SSuKJr SSvKJr SSwKJr \GR^(a\r\(aSSKr\"\"5GRf(a \lGRhOSSx9q\lGRl"5qSyrSzrS{rS|r/S}Qr\"S~S5r\"SS5r\"SS5rSrSr\rSrSr\"\"5GRf(a \lGRhOSSx9q\lGRl"5qSSS.r"SS\5rSrSrSSjr\"S/SQ5rSSjrSrSrSrSrSrSrSSjrSrSrSrSSjrSrSrSSjrSrSSjrSSjrSSjrSrSrSrSrSSjrSrSSjrSSjrSSjrSrSrSSjrSrSrSSjrSr"SS\5rSSjrSSjrSSjrSSjrSSjrSSjrSSjrSrSSjrSSjrS\-S\-S\-S.rSrSrSGrSSjGr"SS\5Gr"SS\5Gr"SS\5Gr"SSG\5GrSGrSGrSGrSSjGr SGr g)z(Helper functions for copy functionality.)absolute_import)print_function)division)unicode_literalsN) namedtuple) attrgetter)range) protojson)config)AccessDeniedException)ArgumentException)CloudApi)EncryptionException)NotFoundException)PreconditionException) Preconditions)ResumableDownloadException)ResumableUploadAbortException)ResumableUploadException)!ResumableUploadStartOverException)ServiceException)MAX_COMPOSE_ARITY)0DEFAULT_PARALLEL_COMPOSITE_UPLOAD_COMPONENT_SIZE)+DEFAULT_PARALLEL_COMPOSITE_UPLOAD_THRESHOLD)-DEFAULT_SLICED_OBJECT_DOWNLOAD_COMPONENT_SIZE)-DEFAULT_SLICED_OBJECT_DOWNLOAD_MAX_COMPONENTS)(DEFAULT_SLICED_OBJECT_DOWNLOAD_THRESHOLD)DEFAULT_GZIP_COMPRESSION_LEVEL) ApiSelector)DaisyChainWrapper)CommandException)HashMismatchException)InvalidUrlError)FilePart)GenerateComponentObjectPrefix)ReadParallelUploadTrackerFile)$ValidateParallelCompositeTrackerData))WriteComponentToParallelUploadTrackerFile)WriteParallelUploadTrackerFile)FileProgressCallbackHandler)ProgressCallbackWithTimeout)#ResumableStreamingJsonUploadWrapper) storage_url)ContainsWildcard)GenerationFromUrlAndString)IsCloudSubdirPlaceholder)StorageUrlFromString)storage_v1_messages) FileMessage)RetryableErrorMessage)DeleteDownloadTrackerFiles)DeleteTrackerFile)ENCRYPTION_UPLOAD_TRACKER_ENTRY)GetDownloadStartByte)GetTrackerFilePath)GetUploadTrackerData)#RaiseUnwritableTrackerFileException)ReadOrCreateDownloadTrackerFile)"SERIALIZATION_UPLOAD_TRACKER_ENTRY)TrackerFileType)!WriteDownloadComponentTrackerFile)WriteJsonDataToTrackerFile)parallelism_framework_util) stet_util)temporary_file_util) text_util)GetJsonResumableChunkSize)GetMaxRetryDelay) GetNumRetries)ResumableThreshold)UsingCrcmodExtension)GetCloudApiInstance)GetDownloadSerializationData)DEFAULT_FILE_BUFFER_SIZE)MIN_SIZE_COMPUTE_LOGGING)UTF8) CryptoKeyType)CryptoKeyWrapperFromKey)FindMatchingCSEKInBotoConfig)GetEncryptionKeyWrapper)Base64EncodeHash)"CalculateB64EncodedMd5FromContents)CalculateHashesFromContents)CHECK_HASH_IF_FAST_ELSE_FAIL)CHECK_HASH_NEVER) ConcatCrc32c)GetDownloadHashAlgs)GetMd5)GetUploadHashAlgs)HashingFileUploadWrapper)ObjectIsGzipEncoded) AtomicDict)$CheckMultiprocessingAvailableAndInit)PutToQueueWithTimeout) ATIME_ATTR)ConvertDatetimeToPOSIX)GID_ATTR) MODE_ATTR) MTIME_ATTR)ParseAndSetPOSIXAttributes)UID_ATTR)CheckFreeSpace) GetFileSize)GetStreamFromFileUrl) IS_WINDOWS)AddS3MarkerAclToObjectMetadata)CopyObjectMetadata)DEFAULT_CONTENT_TYPE)ObjectMetadataFromHeaders)PreconditionsFromHeaders)S3MarkerAclFromObjectMetadata) DivideAndCeil)HumanReadableToBytes)MakeHumanReadable)SECONDS_PER_DAY)TEN_MIB)CreateWildcardIterator)managerF_corruptzF/gsutil/tmp/parallel_composite_uploads/for_details_see/gsutil_help_cp/ad PARALLEL_UPLOAD_SALT_TO_PREVENT_COLLISIONS. The theory is that no user will have prepended this to the front of one of their object names and then done an MD5 hash of the name, and then prepended PARALLEL_UPLOAD_TEMP_NAMESPACE to the front of their object name. Note that there will be no problems with object name length since we hash the original name. )crc32ccustomerEncryptionetag generationmd5Hashsize%PerformParallelUploadFileToObjectArgszfilename file_start file_length src_url dst_url canned_acl content_type storage_class tracker_file tracker_file_lock encryption_key_sha256 gzip_encoded%PerformSlicedDownloadObjectToFileArgszicomponent_num src_url src_obj_metadata_json dst_url download_file_name start_byte end_byte decryption_key!PerformSlicedDownloadReturnValueszKcomponent_num crc32c bytes_transferred component_total_size server_encodingi GZIP_ALL_FILESi` lz text/markdownzapplication/gzip)mdtgzc\rSrSrSrSrg)FileConcurrencySkipErrori+z@Raised when skipping a file due to a concurrent, duplicate copy.N__name__ __module__ __qualname____firstlineno____doc____static_attributes__r*platform/gsutil/gslib/utils/copy_helper.pyrr+sHrrcLURR[U55 g)9Simple exception handler to allow post-completion status.N)loggererrorstrclses r_RmExceptionHandlerr/s**3q6rcURR[U55 U=RS- slURR S[ R "55 g)rz* Encountered exception while copying: %s N)rrrop_failure_countdebug traceback format_excrs r_ParallelCopyExceptionHandlerr4sJ**3q6!**C'')+rcB[URURUR5n[ XS9nU Sn[ R "URRURRURURS9nURn[R(a[R Ul [#UR$UURURUUUUR&U[(SSSUR*S9n[R(aXtl SSS5 WSn [-UR.UR0U UR&UR2S9 U$![R(aXtl ff=f!,(df  Nk=f) aAFunction argument to Apply for performing parallel composite uploads. Args: cls: Calling Command class. args: PerformParallelUploadFileToObjectArgs tuple describing the target. thread_state: gsutil Cloud API instance to use for the operation. Returns: StorageUrl representing a successfully uploaded component.  thread_stateN)namebucket contentType storageClassFT) gzip_extsallow_splitting is_component gzip_encodedencryption_key_sha256)r$filename file_start file_lengthrJapitools_messagesObjectdst_url object_name bucket_name content_type storage_class prefer_apiglobal_copy_helper_opts canned_aclrXML_UploadFileToObjectsrc_urlrrrr( tracker_filetracker_file_lockr) rargsrfp gsutil_api preconditionsdst_object_metadataorig_prefer_apiret components r"_PerformParallelUploadFileToObjectr<s` t0@0@A""3B* M,22 \\ % %||''%%'' ) !++O0 + +!,   " $ 0 0 $ 3 - * # # =*.05-1-1->-> @c ! + + /G J!f)+   jj 66 8 * ! + + / ,E rs%A"FA6E/F/F  F FCopyHelperOpts perform_mv no_clobber daisy_chainread_args_from_stdin print_ver use_manifest preserve_aclrskip_unsupported_objectstest_callback_filedest_storage_classc 4[UUUUUUUUUU U S9 q[$)z9Creates CopyHelperOpts for passing options to CopyHelper.r)rrrs rCreateCopyHelperOptsrs8+/7++ - ! rc[$)z.Returns namedtuple holding CopyHelper options.)rrrrGetCopyHelperOptsrs  ! rcSnUR5(ahUR[R:XaSn[R"UR5R n[R "U5(aSnU(a[RR$[RR$![a NGf=f)zGet download strategy based on the destination object. Args: dst_url: Destination StorageUrl. Returns: gsutil Cloud API DownloadStrategy. FT) IsFileUrlrosdevnullstatst_modeS_ISCHROSErrorrDownloadStrategyONE_SHOT RESUMABLE)rdst_is_specialmodes r_SelectDownloadStrategyrs. bjj(n WWW(( ) 1 1d d    $ $ - --  $ $ . ..    sAB;; CCcUR5(aUR5(a1UR5(a,UR5(aU(d[ SU-5eggg)amEnsures the destination URL names a container. Acceptable containers include directory, bucket, bucket subdir, and non-existent bucket subdir. Args: exp_dst_url: Wildcard-expanded destination StorageUrl. have_existing_dst_container: bool indicator of whether exp_dst_url names a container (directory, bucket, or existing bucket subdir). command_name: Name of command making call. May not be the same as the calling class's self.command_name in the case of commands implemented atop other commands (like mv command). Raises: CommandException: if the URL being checked does not name a container. zuDestination URL must name a directory, bucket, or bucket subdirectory for the multiple source form of the %s command.N)r IsDirectory IsCloudUrlIsBucketr!) exp_dst_urlhave_existing_dst_container command_names rInsistDstUrlNamesContainerrsk${'>'>'@'@K$8$8$:$: & <>JK LL '%;rcfU(agUR5(aU=(d U=(a U$g)aChecks whether dst_url should be treated as a bucket "sub-directory". The decision about whether something constitutes a bucket "sub-directory" depends on whether there are multiple sources in this request and whether there is an existing bucket subdirectory. For example, when running the command: gsutil cp file gs://bucket/abc if there's no existing gs://bucket/abc bucket subdirectory we should copy file to the object gs://bucket/abc. In contrast, if there's an existing gs://bucket/abc bucket subdirectory we should copy file to gs://bucket/abc/file. And regardless of whether gs://bucket/abc exists, when running the command: gsutil cp file1 file2 gs://bucket/abc we should copy file1 to gs://bucket/abc/file1 (and similarly for file2). Finally, for recursive copies, if the source is a container then we should copy to a container as the target. For example, when running the command: gsutil cp -r dir1 gs://bucket/dir2 we should copy the subtree of dir1 to gs://bucket/dir2. Note that we don't disallow naming a bucket "sub-directory" where there's already an object at that URL. For example it's legitimate (albeit confusing) to have an object called gs://bucket/dir and then run the command gsutil cp file1 file2 gs://bucket/dir Doing so will end up with objects gs://bucket/dir, gs://bucket/dir/file1, and gs://bucket/dir/file2. Args: have_multiple_srcs: Bool indicator of whether this is a multi-source operation. dst_url: StorageUrl to check. have_existing_dest_subdir: bool indicator whether dest is an existing subdirectory. src_url_names_container: bool indicator of whether the source URL is a container. recursion_requested: True if a recursive operation has been requested. Returns: bool indicator. TN)r)have_multiple_srcsrhave_existing_dest_subdirsrc_url_names_containerrecursion_requesteds r _ShouldTreatDstUrlAsBucketSubDirrs5X    > $ <)<?rcU(aU(agUR5(aUR5(+$U(+=(a U(+=(a UR5$)a\Checks that dst_url names a single file/object after wildcard expansion. It is possible that an object path might name a bucket sub-directory. Args: src_url_names_container: Bool indicator of whether the source for the operation is a container (bucket, bucket subdir, or directory). have_multiple_srcs: Bool indicator of whether this is a multi-source operation. have_existing_dest_subdir: bool indicator whether dest is an existing subdirectory. dst_url: StorageUrl to check. recursion_requested: True if a recursive operation has been requested. Returns: bool indicator. F)rrIsObject)rrrrrs r_ShouldTreatDstUrlAsSingletonrsU(4  ""$ $$" " +D'D      rcZUR5(dgURRUR5nUR UR5u p#U[ R ;=(a4 U[ R Vs/sHnURS-U-PM sn;$s snf)a?Returns True if not FileUrl ending in relative path symbols. A URL is invalid if it is a FileUrl and the parent directory of the file is a relative path symbol. Unix will not allow a file itself to be named with a relative path symbol, but one can be the parent. Notably, "../obj" can lead to unexpected behavior at the copy destination. We examine the pre-recursion "url", which might point to "..", to see if the parent is valid. If the user does a recursive copy from a URL, it may not end up the final parent of the copied object. For example, see: "dir/nested_dir/obj". If you ran "cp -r dir gs://bucket" from the parent of "dir", then the "url" arg would be "dir", but "nested_dir" would be the parent of "obj". This actually doesn't matter since recursion won't add relative path symbols to the path. However, we still return if "url" is valid because there are cases where we need to copy every parent directory up to "dir" to prevent file name conflicts. Args: url: StorageUrl before recursion. Returns: Boolean indicating if the "url" is valid as a parent directory. Tz://)rversionless_url_stringrstripdelim rpartitionr-RELATIVE_PATH_SYMBOLSscheme)url#url_string_minus_trailing_delimiter_after_last_delimitersymbols r_IsUrlValidParentDirr3s2  (+(B(B(I(I ii)% C N N ii!!Q [%F%F F #99#9f **u v %9# #sB(c UR5(a'UR5(aU(a [S5e[X#XeU5(aU$UR5(aZUR5(dUR 5(a0U(a'UR5(aSOSn [SU -5eU$U(dpU(diUR R UR5Sn [URRUR5<UR<U <35$U(d1U(d*UR5(adUR5(aOURRUR5(d%[UR<UR<35n[U5n U (d!U(a[SRU55eSUR ;=(a U =(a U=(d Un U (d1U(aUR#5(dUR5(ab[%X5n UR ['U 5SR)UR5nU (dUR+UR5SnO(UR R UR5SnUR5(d[-X5UX'5(aUR (acUR RUR5(a9UR RUR5<UR<U<3nO=UR (a UROS nUR =(d S <U<U<3nUR/5nUR1URUR5UlU$) aConstructs the destination URL for a given exp_src_url/exp_dst_url pair. Uses context-dependent naming rules that mimic Linux cp and mv behavior. Args: src_url: Source StorageUrl to be copied. exp_src_url: Single StorageUrl from wildcard expansion of src_url. src_url_names_container: True if src_url names a container (including the case of a wildcard-named bucket subdir (like gs://bucket/abc, where gs://bucket/abc/* matched some objects). have_multiple_srcs: True if this is a multi-source request. This can be true if src_url wildcard-expanded to multiple URLs or if there were multiple source URLs in the request. has_multiple_top_level_srcs: Same as have_multiple_srcs but measured before recursion. exp_dst_url: the expanded StorageUrl requested for the cp destination. Final written path is constructed from this plus a context-dependent variant of src_url. have_existing_dest_subdir: bool indicator whether dest is an existing subdirectory. recursion_requested: True if a recursive operation has been requested. preserve_posix: True if preservation of posix attributes has been requested. Returns: StorageUrl to use for copy. Raises: CommandException if destination object name not specified for source and source is a stream. z/Cannot preserve POSIX attributes with a stream.stream named pipez2Destination object name needed when source is a %szmPresence of multiple top-level sources and invalid expanded URL make file name conflicts possible for URL: {}z**N)rIsStreamr!rIsFiforrrr1 url_stringrrendswithrr#formatrrGetPathBeforeFinalDirlenlstrip partitionrClonereplace)r exp_src_urlrrhas_multiple_top_level_srcsrrrpreserve_posix type_textsrc_final_compsrc_url_is_valid_parentpreserve_src_top_level_dirssrc_url_path_sans_final_dir dst_key_namernew_exp_dst_urls rConstructDstUrlr![sfN+"6"6"8"8^ L MM"#:#<#688 +"6"6"8"8"-"4"4"6"6 )2244(,i .09: ;;  %7!,,77 FrJN K,B,B,I,I-'--~!? @@4K$;$;$=$=  ) )+*;*; < <&((+*;*;<>K19 %@  99? JJR"&W-K-K!K"=!8"="=#<";!%<&1&<&<&>&>&1&=&=&?&? #8"M55 '()++16'--+@ '!++GMM:2>l**55gmmDRHL!A'@"4"4;#:#:#C#C$$!,!8!8!?!?   ")//?l$/#:#:ke!,!8!8"$!#"$%*L:l %%'/ , 4 4W]]5@5F5F!H/ rc0nU(aOUHIn[R"XR55RS5R S5X'MK U$)N ascii)base64 b64encodedigestrdecode) digestersdigestsalgs r_CreateDigestsFromDigestersr,sO '%% .   !##)6%=l .rc0nSU;a [5US'SU;a"[RRS5US'[ US5n[ UU[ UR[UUSS9R5S9 SSS5 0n[R"U5H up[U R55Xx'M" U$!,(df  NL=f) avCreates a base64 CRC32C and/or MD5 digest from file_name. Args: status_queue: Queue for posting progress messages for UI/Analytics. algs: List of algorithms to compute. file_name: File to digest. src_url: StorageUrl for local object. Used to track progress. src_obj_metadata: Metadata of source object. Returns: Dict of algorithm name : base 64 encoded digest md5rzzcrc-32crbHashing)roperation_name)callback_processorN)rZcrcmod predefinedCrcopenrUr+rr*callsix iteritemsrS hexdigest) status_queuealgs file_namersrc_obj_metadata hash_dictrr*alg_namer's r_CreateDigestsFromLocalFilerA s) d]xIe  ++// :Ih It )3N$4$9$9$?(4077@%BCG$ 4HI '-- 2h()9)9);dst_obj_metadatacheck_hashes_config checked_onedownload_hashes upload_hashes src_md5hashsrc_crc32c_hash dst_md5hashdst_crc32c_hashr+upload_b64_digestdownload_b64_digests r_CheckCloudHashesrW.s& 8^#?A,, +/-##$4$<$<=K(E''(8(?(?@O /H##$4$<$<=K&%''(8(?(?@O-( # m <c !). ww:1377/66w? LLN2G/ ! #%6 A BBK!=  KK FGN rc |Un0nUR(a4[R"UR5nURS5US'UR(a4[R"UR5n U RS5US'Sn UHrn X;aM [R"Xk5n X{n UR SU X@+ !  "25C)5*6?7Y+N P QQ K  kk 89@B  kk 89BD rc|[U[5=(d& [U[5=(a SUR;$)aBChecks to see if the server attempted to clobber a file. In this case we specified via a precondition that we didn't want the file clobbered. Args: e: The Exception that was generated by a failed copy operation Returns: bool indicator - True indicates that the server did attempt to clobber an existing file. 412)rIrrmessage)rs rIsNoClobberServerExceptionrfs4 a. / K a1 2 Iu 7ILrcUR5(agURn[RR U5n[RR U5(a[ SUR<SU<S35e[RRU5(a[ SUR<SU<S35eg)aChecks whether copying exp_src_url into dst_url is not possible. This happens if a directory exists in local file system where a file needs to go or vice versa. In that case we print an error message and exits. Example: if the file "./x" exists and you try to do: gsutil cp gs://mybucket/x/y . the request can't succeed because it requires a directory where the file x exists. Note that we don't enforce any corresponding restrictions for buckets, because the flat namespace semantics for buckets doesn't prohibit such cases the way hierarchical file systems do. For example, if a bucket contains an object called gs://bucket/dir and then you run the command: gsutil cp file1 file2 gs://bucket/dir you'll end up with objects gs://bucket/dir, gs://bucket/dir/file1, and gs://bucket/dir/file2. Args: exp_src_url: Expanded source StorageUrl. dst_url: Destination StorageUrl. Raises: CommandException: if errors encountered. NzCannot retrieve z> because a file exists where a directory needs to be created (z).z because a directory exists (z%) where the file needs to be created.) rrrpathdirnameisfiler!risdir)rrdst_path final_dirs rCheckForDirFileConflictrns2   (ggooh')WW^^I '22I? @@WW]]8 '22H> ??rc F[[R"SS[55n [ U[ U 5up0n/n[ U 5Hn[R"[UR-5n[5nURU5 UR5nU[-U-S-[U5-nUR!5nUUlUU S- :aUnO X=S- U-- nUU-n[%URUUUUUXXX5 nUR'U5 UUU'M U$)aPartitions a file into FilePart objects to be uploaded and later composed. These objects, when composed, will match the original file. This entails splitting the file into parts, naming and forming a destination URL for each part, and also providing the PerformParallelUploadFileToObjectArgs corresponding to each part. Args: canned_acl: The user-provided canned_acl, if applicable. content_type: content type for the component and final objects. dst_bucket_url: CloudUrl for the destination bucket. file_size: The size of fp, in bytes. fp: The file object to be partitioned. random_prefix: The randomly-generated prefix used to prevent collisions among the temporary component names. src_url: Source FileUrl from the original command. storage_class: storage class for the component and final objects. tracker_file: The path to the parallel composite upload tracker file. tracker_file_lock: The lock protecting access to the tracker file. encryption_key_sha256: Encryption key SHA256 for use in this upload, if any. gzip_encoded: Whether to use gzip transport encoding for the upload. Returns: dst_args: The destination URIs for the temporary component objects. rC(parallel_composite_upload_component_sizerr)rsr rFr_GetPartitionInforr r8rGPARALLEL_UPLOAD_STATIC_SALTrrZupdater:PARALLEL_UPLOAD_TEMP_NAMESPACErrrrappend)rrdst_bucket_url file_sizer random_prefixrrrrrrrpnum_componentscomponent_sizedst_args file_namesi encoded_name content_md5r'temp_file_name tmp_dst_urlfile_part_lengthoffset func_argss r_PartitionFilersKJ.B jjEAC.D*&7"$L&N">(*  a $$%@277%JKL(K|$  " " $F#&DDvMF#N &&(K,KNQ '$(:n'LM  F5 )7K\-In%(H^7 !: /rcj[URURRS5S-S5$)zGets component number from component CloudUrl. Used during parallel composite upload. Args: component: CloudUrl representing component. Returns: component number rrN)intrrfind)rs r_GetComponentNumberr7s3 Y " "9#8#8#>#>s#Ca#G#H I JJrc  [R"5n [UR5n URURS9n[ [ 5nU(a UROSn[U[RX5n[UU 5unnn[UUUUUXU [[5 unnUbURS5OSnUbUO[!US9n[#UUUUS9 [$R&"5n0n[)UUR*U UUUUUR,UUUU S9 n[/UUX5unnnUHn2UR44UUR6R8'[;UR<[1UR>UR6[R"5UR4S[AUR65[0R2S95 M UHnUSRBn[0RDUS 4UU'[;UR<[1UUS[R"5SUS [AUS5[0RDS 95 M UHEnR<[1UU"5S[0RFS 95 MG URI[JUU S [LRNRPURRRTS S9n/nUHnURWUS5 M UUV s/sHn U SPM sn -n![YU!5[YU5:XGa[[U![@S9n!/n"U!Hjn#[\R^RaU#RbS9n$U#Re5(a[gU#Rh5U$l4U"RWU$5 Ml URkU"UUUR/SQUS9n%U!U-n&URI[U&[[LRNRPURRRTS9 U!H[nURBn[;UR<[1UU"5S [AU5UUS UUSS95 M] UHEnR<[1UU"5S [0RFS 95 MG U [uU5 SSS5 O [wS5e[R"5U - n'U'U%4$s sn f! [;UR<[1UU"5S S95 GM&=f![la4 U RoSSRqURs55-5 Nf=f!,(df  N=f!U [uU5 SSS5 f!,(df  f=f=f)aUploads a local file to a cloud object using parallel composite upload. The file is partitioned into parts, and then the parts are uploaded in parallel, composed to form the original destination object, and deleted. Args: fp: The file object to be uploaded. src_url: FileUrl representing the local file. dst_url: CloudUrl representing the destination file. dst_obj_metadata: apitools Object describing the destination object. canned_acl: The canned acl to apply to the object, if any. file_size: The size of the source file in bytes. preconditions: Cloud API Preconditions for the final object. gsutil_api: gsutil Cloud API instance to use. command_obj: Command object (for calling Apply). copy_exception_handler: Copy exception handler (for use in Apply). logger: logging.Logger for outputting log messages. gzip_encoded: Whether to use gzip transport encoding for the upload. Returns: Elapsed upload time, uploaded Object with generation, crc32c, and size fields populated. providerNr$r)rrFrfinished component_num message_typerr)rrrr)rr)rtotal_bytes_transferredT arg_checkerparallel_operations_overrideshould_return_resultsrkey)r)rzr}r)rrfieldsencryption_tuple)rr)rrrr)rz:Failed to delete some of the following temporary objects:  zSSome temporary components were not uploaded successfully. Please retry this upload.)<timer1bucket_url_stringGetApiSelectorrrRr crypto_key_sha256r9r>PARALLEL_UPLOADr&r'_DeleteTempComponentObjectFnrr(r%r)rA CreateLockrrrFilterExistingComponentsr3COMPONENT_TO_UPLOADrrrr`r;rrrEXISTING_COMPONENTEXISTING_OBJECT_TO_DELETEApplyrgslibcommandDummyArgCheckerParallelOverrideReasonSLICErursortedrComposeRequestSourceObjectsValueListEntryr HasGenerationlongr} ComposeObject ExceptionrKjoinkeysr6r!)(rrrrLrrwrr command_objcopy_exception_handlerrr start_timerv api_selectorencryption_keywrapperrtracker_file_nameexisting_enc_key_sha256existing_prefixexisting_componentsrxrcomponents_infor{components_to_uploadexisting_objects_to_deleter component_str cp_resultsuploaded_components cp_resultr} componentsrequest_components component_urlr>composed_objectobjects_to_delete elapsed_times( r_DoParallelCompositeUploadrEsFyy{*'(A(AB.**GNN*C,1&931BB9= ))8)H)H)5@ 9&"O ,P0/0. *,?,A(?' 4?177@EI&5&A?0-BD !!2!.!47LN 1;;=/ J,88*%)#,99--2G)5 7(#;$n#B,(i'')>)>5@OI%%001I%%%%IIK"..""5i6G6G"H!,!@!@  BC('iaL77M&1&D&D&/l&4OM"GaLIIK""1"5il"C!,!?!?  AB '.iGIIK"!,!F!F  HI.  (M--//#.#E#E#K#K !"*iy|,"4G%H4Gqad4G%HH*_H % (; NOTE: You are uploading one or more large file(s), which would run significantly faster if you enable parallel composite uploads. This feature can be enabled by editing the "parallel_composite_upload_threshold" value in your .boto configuration file. However, note that if you do this large files will be uploaded as `composite objects `_,which means that any user who downloads such objects will need to have a compiled crcmod installed (see "gsutil help crcmod"). This is because without a compiled crcmod, computing checksums on composite objects is so slow that gsutil disables downloads of composite objects.TN)rsr rFrr r r'PARALLEL_COMPOSITE_SUGGESTION_THRESHOLDsuggested_sliced_transfers_locksuggested_sliced_transfersinfortextwrapwrap) rrrrrwrrrall_factors_but_sizes r _ShouldDoParallelCompositeUploadr(s 2)= jj@<>)?%  nn  ..D . BaG:: ( ' + +K 8 8 DII MM % & '*. . /37";/! )$  ;#F#J ;  :<% ) (s !AD D+c VU=(d [R"5n[U5(aa[[ UUUUS95n[ U5S:wa[ SU-5eUSn[UR5UR5(+4$[U5nUR"5(aXwR"54$UR"5(aUS4$[U5(aUS4$URRS5nUR!UR"USUR$SS/S 9n U Hn U R&[(R*R,:XaU R.US-:XaUS4s $U R&[(R*R0:Xa-U R.R2URS -:XaUS4s $U R&[(R*R0:XdMU R.R2UR:XdMUS 4s $ Xs4$) a Expands wildcard if present in url_str. Args: url_str: String representation of requested url. gsutil_api: gsutil Cloud API instance to use. project_id: project ID to use (for iterators). treat_nonexistent_object_as_subdir: indicates if should treat a non-existent object as a subdir. logger: logging.Logger instance to use for output. If None, the root Logger will be used. Returns: (exp_url, have_existing_dst_container) where exp_url is a StorageUrl and have_existing_dst_container is a bool indicating whether exp_url names an existing directory, bucket, or bucket subdirectory. In the case where we match a subdirectory AND an object, the object is returned. Raises: CommandException: if url_str matched more than 1 URL. ) project_idrrz)Destination (%s) must match exactly 1 URLrT/prefixesz items/name)prefix delimiterrrz _$folder$F)loggingLoggerr.listrwrr!r1rrrrrr0rr ListObjectsrrdatatyperCsObjectOrPrefixTypePREFIXdataOBJECTr) url_strrr"treat_nonexistent_object_as_subdirr blr_expansionblrr-r list_iterator obj_or_prefixs rExpandUrlToSingleBlrrhs6  %W^^%&gw)*4&, ./M  =Q H$% &&  C ! 0clln2D EE$W-+ 002 33  k**    " " ) )# .&(()@)@06362=2D2D1;\0J )L- %m ("?"?"F"FFfsl*4   H$A$A$H$H H    ! ![%<%<{%J J4   H$A$A$H$H H    ! ![%<%< <5 !!5%<  ::rc[R"SSS5(dgUR5(dg[UR5(agUS:XagUR UR S/URS9 g![a4n[U[RR5(aeSnAgSnAff=f)aDMakes a request to the destination API provider to trigger reauth. Addresses https://github.com/GoogleCloudPlatform/gsutil/issues/1639. If an API call occurs in a child process, the library that handles reauth will fail. We need to make at least one API call in the main process to allow a user to reauthorize. For cloud source URLs this already happens because the plurality of the source name expansion iterator is checked in the main thread. For cloud destination URLs, only some situations result in a similar API call. In these situations, this function exits without performing an API call. In others, this function performs an API call to trigger reauth. Args: destination_url (StorageUrl): The destination of the transfer. gsutil_api (CloudApiDelegator): API to use for the GetBucket call. worker_count (int): If greater than 1, assume that parallel execution is used. Technically, reauth challenges can be answered in the main process, but they may be triggered multiple times if multithreading is used. Returns: None, but performs an API call if necessary. rC0trigger_reauth_challenge_for_parallel_operationsFNrlocation)rr) r getboolrr.r GetBucketrrrrIpyu2ferrors PluginError)destination_urlr worker_countrs r.TriggerReauthForDestinationProviderIfNecessaryrs: BE K K   # # % % o0011 Q  ##| '')  !U\\--.. / s&A?? B= *B88B=cUR5(aRURS:XaBUR5(a-[R"SSUR 5n[ U5nU$)aTranslates Windows pathnames to cloud pathnames. Rewrites the destination URL built by ConstructDstUrl(). Args: src_url: Source StorageUrl to be copied. dst_url: The destination StorageUrl built by ConstructDstUrl(). Returns: StorageUrl to use for copy. \z\\r)rrrresubrr1)rr trans_url_strs rFixWindowsNamingr sSgmmt38J8J8L8LFF5#w'9'9:M"=1G .rctUR5(akUR5(aV[RRUR5n[RRUR5nX#:H$UR UR :H=(a UR UR :H$)zChecks if src_url and dst_url represent the same object or file. We don't handle anything about hard or symbolic links. Args: src_url: Source StorageUrl. dst_url: Destination StorageUrl. Returns: Bool indicator. )rrrhnormpathrrr})rr new_src_path new_dst_paths r SrcDstSamers W..0077##G$7$78L77##G$7$78L  ''   '"4"4 4 5   '"4"4 46rcUR5(a(U(a!UR(aSUR-nOSnUR5(aVUR5(dUR 5(a,UR5(aSOSnUR SXT5 gUR SUR U5 g)aLogs copy operation, including Content-Type if appropriate. Args: logger: logger instance to use for output. src_url: Source StorageUrl. dst_url: Destination StorageUrl. dst_obj_metadata: Object-specific metadata that should be overidden during the copy. z [Content-Type=%s]r zr zCopying from %s%s...zCopying %s%s...N)rrrr r rr)rrrrLcontent_type_msgsrc_texts r_LogCopyOperationr5s/""+.>.J.JJ g..00GNN4D4D#,,..yLH KK&C KK!7#5#57GHrc 6[U5n[[5n[R"5n [ UR UUSS9R n [R(aQ[[RS5n [R"U R55R n SSS5 URUUUR[RUU UR ["UUS9 n [R"5n UR%5n['UU R5Ul[)UR [+UUU [*R,UR.SS95 X- UR.UU R04$!,(df  N=f)aPerforms copy-in-the cloud from specified src to dest object. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata for source object; must include etag and size. dst_url: Destination CloudUrl. dst_obj_metadata: Object-specific metadata that should be overidden during the copy. preconditions: Preconditions to use for the copy. gsutil_api: gsutil Cloud API instance to use for the copy. decryption_key: Base64-encoded decryption key for the source object, if any. Returns: (elapsed_time, bytes_transferred, dst_url with generation, md5 hash of destination) excluding overhead like initial GET. Raises: CommandException: if errors encountered. Copyingrrr1r/N)src_generationrrprogress_callbackrrdecryption_tuplerTrrr)rPrRr rr*r;r7rrr6pickleloadsread CopyObjectr}rrUPLOAD_RETURN_FIELDSrr/r`r3FILE_CLOUD_COPYrr~)rr>rrLrrdecryption_keydecryption_keywrapperrrr test_fpdst_objend_time result_urls r_CopyObjToObjInTheCloudrLsg42.A1&9yy{*1*2I2I:A:AAJLMQD// %88$ ?7 ,,w||~6;; @  ! !"2"2181C1C-D-O-O0=4E+2>>)=3H3H " J'YY[(}}*4Z5<5G5GI*'*::',, !"  !1!6!6 // 9 @ ?s ?/F  FcURGcUR5(GaUR5(GdUR5(GdURnSnUS:wGa\[ R RU5n[R"SSS5(a[(d[R"SSSU/[R[RS 9nUR5upgURR!5 UR"R!5 UR$S :wdU(a['S XER$U4-5eUR)5n[*R,"U5nO@UR1S5u pU [2;a [2U nO[4R6"U5up9U(d[8nX1lggggg![.an['S U<S U<35eSnAff=f)zDetects and sets Content-Type if src_url names a local file. Args: src_url: Source StorageUrl. dst_obj_metadata: Object-specific metadata that should be overidden during the copy. N-rC use_magicfileFfilez-bz--mime)stdoutstderrrzAEncountered error running "file -b --mime %s" (returncode=%d). %sz,Encountered OSError running "file -b --mime z" .)rrr r rrrhrealpathr rrk subprocessPopenPIPE communicater closer! returncoder!rr8 ensure_strrrCOMMON_EXTENSION_RULES mimetypes guess_typern) rrLrrreal_file_pathpoutputrrr extensions r_SetContentTypeFromFiler2s""*w/@/@/B/B     W^^%5%5%%KLcww'' 4n /5 9 9** #hG&0oo&0oo7!--/-& ((..  ((..  \\Q %"'*8,,)NOP P , 5, )33C81 . ./ :,%00@/, )l#/ K&6 0C*4 # q"# # #s.CG G<$G77G<c [URUUSS9Rn[R(aQ[ [RS5n [ R"U R55RnSSS5 [R"5n [[5n UR5(dUR5(a4URUU[RUUU UR ["US9 n O4UR%UU[RUUUU UR ["US9 n [R"5n X- nX4$!,(df  N=f)aUploads the file using a non-resumable strategy. This function does not support component transfers. Args: src_url: Source StorageUrl to upload. src_obj_filestream: File pointer to uploadable bytes. src_obj_size (int or None): Size of the source object. dst_url: Destination StorageUrl for the upload. dst_obj_metadata: Metadata for the target object. preconditions: Preconditions for the upload, if any. gsutil_api: gsutil Cloud API instance to use for the upload. gzip_encoded: Whether to use gzip transport encoding for the upload. Returns: Elapsed upload time, uploaded Object with generation, md5, and size fields populated. Uploadingr r/N)object_metadatarrr rrrr) r5rrrr rrrr)r*r;r7rrr6rrrrrRr r r UploadObjectStreamingrrr UploadObject)rsrc_obj_filestream src_obj_sizerrLrrrr rrruploaded_objectrrs r_UploadFileToObjectNonResumabler;s<42 "#'$  // %88$ ?7 ,,w||~6;; @yy{*1&9 7>>++ 66(*55#+.#!7 #O!--(*55 #+.#!. #OYY[(&,  &&C @ ?s /E E'c ^^[U[RURURS95m[ [ 5n U (a,U R(aU RRS5OSmUU4Sjn [TUTS9n U (aURSUR5 Sn U(a [U5OSn[URUUUSS 9Rn[ R"(aQ[%[ R"S 5n[&R("UR+55RnSSS5 [,R,"5nS nU (aTUR/UU[ R0UURUU U [2U UU S 9 nS n U (d [AT5 U (aMT[,R,"5nUU- nUW4$!,(df  N=f![4GanURSUR-5 US- nU[75:n U (deURSUR8-5 UR;UR8URS9 O![<a O[>a ef=f[AT5 URST<SUR<S35 O4![AT5 URST<SUR<S35 f=fURSURUU4-5 Sn URCS 5 U(a [U5OSn[URUUUSS 9Rn[EUR[GU[,R,"5US95 [,RH"[K[LRL"5SU--[O555 SnAGN1SnAf[Pa S n ef=f!U (d [AT5 ff=f)aUploads the file using a resumable strategy. Args: src_url: Source FileUrl to upload. Must not be a stream. src_obj_filestream: File pointer to uploadable bytes. src_obj_size (int or None): Size of the source object. dst_url: Destination StorageUrl for the upload. dst_obj_metadata: Metadata for the target object. preconditions: Preconditions for the upload, if any. gsutil_api: gsutil Cloud API instance to use for the upload. logger: for outputting log messages. is_component: indicates whether this is a single component or whole file. gzip_encoded: Whether to use gzip transport encoding for the upload. Returns: Elapsed upload time, uploaded Object with generation, md5, and size fields populated. rr$NcL>[T[[U50n[TU5 g)aCreates a new tracker file for starting an upload from scratch. This function is called by the gsutil Cloud API implementation and the the serialization data is implementation-specific. Args: serialization_data: Serialization data used in resuming the upload. N)r7r=rr@)serialization_datarrrs r_UploadTrackerCallback<_UploadFileToObjectResumable.._UploadTrackerCallback/s* ()>*C0B,C D0$7rrzResuming upload for %sTr4)rrrr1r/r) r5rrrrr>rrtracker_callbackr rFz:Caught ResumableUploadStartOverException for upload of %s.rz8Checking that bucket %s exists before retrying upload...zDeleted tracker file z for resumable upload of z before retrying.zRestarting upload of %s from scratch (retry #%d) after exception indicating we need to start over with a new resumable upload ID: %s) num_retriesr))r9r>UPLOADrrrRr rr(r:rrrr*r;r7rrr6rrrrUploadObjectResumablerrrrGrrr rr6seekr`r4sleepminrandomrFr)rr8r9rrLrrrrrrr? tracker_data retryablerr rrnum_startover_attemptsr:rrrrrs @@r_UploadFileToObjectResumablerL s8) %%8:2&9 #8#J#J--44W=  8"&7LN, KK('*<*<=)2>%g.D-1! " #'$ // %88$ ?7 ,,w||~6;; @yy{* H-"88 *,77%>>)0%1-#9 %oir+,S  VYY[(J&,  ((u @ ?8 -3# kkN$$%& !)MO;i   K N$++, --44w~~N "    +, #4g6H6HJ K +, #4g6H6HJ K kk P   !7 ; <=la 6B)'2m5  ! !%$ & '+d   ! !  $ ,B DE  jj fmmo$:!:;  "## ) i  +,sn/G*8G, G), O$78O0AI32J=3 J =J=? J  J= 0O=1K..CO O'O$$O''O<cSnSnURS5nU(a U(aSnXE4$U[:XdU(a![U5S:aUSU;a U(+nUnXE4$)aSelects how an upload should be compressed. This is a helper function for _UploadFileToObject. Args: object_name: The object name of the source FileUrl. is_component: indicates whether this is a single component or whole file. gzip_exts: List of file extensions to gzip prior to upload, if any. If gzip_exts is GZIP_ALL_FILES, gzip all files. gzip_encoded: Whether to use gzip transport encoding for the upload. Used in conjunction with gzip_exts for selecting which files will be encoded. Streaming files compressed is only supported on the JSON GCS API. Returns: A tuple: (If the file should be gzipped locally, if the file should be gzip transport encoded). Fr"Trr )splitrr)rrrr zipped_filegzip_encoded_file fname_partss r _SelectUploadCompressionStrategyrRsr,+!!#&+ l  '' ^# s;'!+ B90L""K$  ''rcUbU[:aURSU5 [R"5upESnUR 5(dUR 5(a [ S5eUb)[U5S[U5-:a[ SU-5e[R"SS[5n[R"USUS 9nUR[5nU(a/UR!U5 UR[5nU(aM/U(aUR#5 [$R""U5 UR#5 [$R&R)U5n [US 5n [+U5X4$!U(aUR#5 [$R""U5 UR#5 f=f) aCompresses a to-be-uploaded local file to save bandwidth. This is a helper function for _UploadFileToObject. Args: src_url: Source FileUrl. src_obj_filestream: Read stream of the source file - will be consumed and closed. src_obj_size (int or None): Size of the source file. logger: for outputting log messages. Returns: StorageUrl path to compressed file, read stream of the compressed file, compressed file size. NzCompressing %s (to tmp)...zgzip compression is not currently supported on streaming uploads. Remove the compression flag or save the streamed output temporarily to a file before uploading.rz|Inadequate temp space available to compress %s. See the CHANGING TEMP DIRECTORIES section of "gsutil help cp" for more info.rCgzip_compression_levelwb) compresslevelr/)rMrtempfilemkstempr r r!rhrr getintrgzipr6rGZIP_CHUNK_SIZEwriter(rrhgetsizer1) rr8r9rgzip_fh gzip_pathgzip_fpcompression_levelr gzip_sizecompressed_filestreams r_ApplyZippedUploadCompressionrds$,2J"J KK,g6!))+7 'W^^--  4 55^I%>'(3|+<'<&= BDKL MM h0H&DFii 47HIG  " "? 3D  mmD  $ $_ 5d $ mmoHHWggooi()y$/ i (*? JJ  mmoHHWs CF ?G cdSnSnU(a4U(d-[RR U"5upxSSS5 O U"5upxUR5 U(a$[R "UR 5 Xx4$Xx4$!,(df  NM=f![a! URSUR 5 Xx4$f=f!UR5 U(aN[R "UR 5 f![a URSUR 5 ff=ff=f)aHandles setup and tear down logic for uploads. This is a helper function for _UploadFileToObject. Args: upload_delegate: Function that handles uploading the file. upload_url: StorageURL path to the file. upload_stream: Read stream of the file being uploaded. This will be closed after the upload. zipped_file: Flag for if the file is locally compressed prior to calling this function. If true, the local temporary file is deleted after the upload. gzip_encoded_file: Flag for if the file will be uploaded with the gzip transport encoding. If true, a lock is used to limit resource usage. parallel_composite_upload: Set to true if this upload represents a top-level parallel composite upload (not an upload of a component). If true, resource locking is skipped. logger: For outputting log messages. Returns: The elapsed upload time, the uploaded object. NzZCould not delete %s. This can occur in Windows because the temporary file is still locked.) rr!concurrent_compressed_upload_lockr(runlinkrrwarning) upload_delegate upload_url upload_streamrOrPparallel_composite_uploadrrr:s r_DelegateUploadFileToObjectrm s+2,/G!: == : :(7(9%  ; :'6&7#lG *(()  &&  &&) ; : G ./9/E/E G  && G G *(()G ./9/E/E GG sW(C B C" B BC&CCD/! DD/&D+(D/*D++D/c v^^^^^^^ ^ ^^^^ ^!^"T(aTR(d%[R"SS5nU(aUTlUm!Um Um[URT X5unmT(aT (dTR SU5 OkU(ad[ XUT5um!m mSTlTR(dSTlO3STRR5;aT=RS- slT (dC[TR[T!T[R"5[RTSS95 S nS n[5m[!U4S jT=(d 055n[#TU T!TUT[$R&S 9nTcS OT[)5:=(d' UR+5=(d UR-5nUR+5(dUR-5(aATR/TR0S 9[2R4:XaT n[7U[955m U(d [;T5(a[=T UTT!T5m"OT m"UU UUUUUUUU U!4 SjnUUUUUUU!U"4SjnUUUUU UUUU!U"4 SjnU(aUnO U(aUnOUn[?UT!T UTUT5unnU(d$[AU5n[CTTUURUSS9 TR]5nURZUl-[_UURZ5Ul-T (dC[TR[T!T[R"5[RTSS95 UUR`UURb4$![Da [F(ag[HRJ"TRLTRNURPS9nTR[R-Tl&TRUUTTR0S 9 TRWTRXTRURZTR0S9 ef=f)aHUploads a local file to an object. Args: src_url: Source FileUrl. src_obj_filestream: Read stream of the source file to be read and closed. src_obj_size (int or None): Size of the source file. dst_url: Destination CloudUrl. dst_obj_metadata: Metadata to be applied to the destination object. preconditions: Preconditions to use for the copy. gsutil_api: gsutil Cloud API to use for the copy. logger: for outputting log messages. command_obj: command object for use in Apply in parallel composite uploads. copy_exception_handler: For handling copy exceptions during Apply. gzip_exts: List of file extensions to gzip prior to upload, if any. If gzip_exts is GZIP_ALL_FILES, gzip all files. allow_splitting: Whether to allow the file to be split into component pieces for an parallel composite upload. is_component: indicates whether this is a single component or whole file. gzip_encoded: Whether to use gzip transport encoding for the upload. Used in conjunction with gzip_exts for selecting which files will be encoded. Streaming files compressed is only supported on the JSON GCS API. Returns: (elapsed_time, bytes_transferred, dst_url with generation, md5 hash of destination) excluding overhead like initial GET. Raises: CommandException: if errors encountered. rCcontent_languagez+Using compressed transport encoding for %s.rZz no-transformz ,no-transformFrNc3:># UHoTU"54v M g7fNr.0r+ hash_algss r &_UploadFileToObject..Fos3)*o)rrrcH> [T T TT[RTTTTTTTS9 $N)r)rrr) rrrLrrrPrr upload_sizerkrjs rCallParallelCompositeUpload8_UploadFileToObject..CallParallelCompositeUploads; %m&0&-&6&=&H&H&1&3&0&1&<&,3D F Frc $>[TTTTTTTTS9$rz)r;)rLrrrPrr{rjwrapped_filestreamsrCallNonResumableUpload3_UploadFileToObject..CallNonResumableUploads) *:+=+6+2+;+8+58I KKrc (> [TT TTTTTTTTS9 $)N)rr)rL) rLrrrPrrrr{rjrs rCallResumableUpload0_UploadFileToObject..CallResumableUploads/ ' (:(3(/(8(5(2(.5A5F H HrT)r[rrr|r}r)2contentLanguager get_valuerRrrrdcontentEncoding cacheControllowerr`r;r3r FILE_UPLOADr[dictrrrrHr r rrrJSONr,rErr\rmr,rbr"_RENAME_ON_HASH_MISMATCHrrrrr|_RENAME_ON_HASH_MISMATCH_SUFFIXr DeleteObjectrr}rr/rr~)#rr8r9rrLrrrrrrrrrrorOrr:r)rlnon_resumable_upload orig_streamr|rrdelegater*corrupted_obj_metadatarrPrtr{rkrjrs# ``````` ` @@@@@@rrrEsX !1!A!A''2DE)9&*$-+#C <$B + | LL>H-J\6.;*J {'-$  ( (&4# /<<BBD D##6# JIIK!,!8!8$"  $%,/!)Fio2oFF)>  (335!, 31.01+4;4D4D4F+!..*W^^--8Krdir_namer need_to_unzipdownload_file_namers r_GetDownloadFiler s&WW__W00 1( bggnnX..kk( -)**M,??H KK68JK,< NOTE: You are downloading one or more large file(s), which would run significantly faster if you enabled sliced object downloads. This feature is enabled by default but requires that compiled crcmod be installed (see "gsutil help crcmod").TN)rsr rFrrYrrVrzrIrWrrrrrrrrrrr) download_strategyr>rrrmax_componentsrMparallel_hashing hashing_okay use_slices r_ShouldDoSlicedDownloadrR sz&: jj=9;&<"==!H!NP.  8^#?A%,,G1E1G!L%8# UHoTU"54v M g7frqrrrs rru5_PerformSlicedDownloadObjectToFile.. rwrx)r start_byteend_byterNrzr)rJr decode_messagerrsrc_obj_metadata_jsonrYrrzr_DownloadObjectToFileResumablerrrrrrrcrcValuer) rrrrr>r)bytes_transferredserver_encoding crc32c_valrts @r"_PerformSlicedDownloadObjectToFiler s $#3B*--.?.F.F.2.H.HJ!#**2B2I2IK)Fio2oFF)*H ll ll  jj&&}}(( +*&* 8$--J *4+=+=z+<+/==4??+JQ+N+: <SLICED_DOWNLOADr6rijsonloadr}r(rKrIOError ValueErrorrIrENOENTrr5r\dumpsr;strerror) r>rrrrryrrrexisting_file_sizetracker_file_datars r#_MaintainSlicedDownloadTrackerFilesr s6,    ,/11 ()8)H)H)57 "  $ 'B$R222+S1l))L1 F #'7'<'< < L )-=-H-H H , - ? hhj  H''( ) hhjW3 G  %"''(33* $567 & %% : /!Z  AGGu||$; kk$c!f./ / hhj & % G'(9::FFGsi A7F25.F2 I*)AI)I*2H"AHH%H""H%%1I I'#I*'I** J4JJcf\rSrSrSrSr\S5rSr\ R4Sjr Sr Sr S rS rg ) SlicedDownloadFileWrapperi aWraps a file object to be used in GetObjectMedia for sliced downloads. In order to allow resumability, the file object used by each thread in a sliced object download should be wrapped using SlicedDownloadFileWrapper. Passing a SlicedDownloadFileWrapper object to GetObjectMedia will allow the download component tracker file for this component to be updated periodically, while the downloaded bytes are normally written to file. cNXlX lX0lSUlX@lXPlg)aInitializes the SlicedDownloadFileWrapper. Args: fp: The already-open file object to be used for writing in GetObjectMedia. Data will be written to file starting at the current seek position. tracker_file_name: The name of the tracker file for this component. src_obj_metadata: Metadata from the source object. Must include etag and generation. start_byte: The first byte to be downloaded for this parallel component. end_byte: The last byte to be downloaded for this parallel component. N)_orig_fp_tracker_file_name_src_obj_metadata_last_tracker_file_byte _start_byte _end_byte)selfrrr>rrs r__init__"SlicedDownloadFileWrapper.__init__ s(M/-#'D !Nrc0[URSS5$)z >@ ?$--.}}))+,I $$,777)CNNQ..'(?(?(,(>(>(8:&6" /rc:U[R:Xa+URRXR-S-5 OURRX5 UR URR 5s=::aURS-::de egr)rSEEK_ENDrrErrr)rrwhences rrESlicedDownloadFileWrapper.seek= ss  mm..0145 mm(   t}}113 It~~7I II II Irc6URR5$rq)rrrs rrSlicedDownloadFileWrapper.tellD s ==   rc8URR5 grq)rflushrs rrSlicedDownloadFileWrapper.flushG sMMrc\UR(aURR5 ggrq)rr(rs rr(SlicedDownloadFileWrapper.closeJ s }} mmr)rrrrrrN)rrrrrrpropertyrr\rSEEK_SETrErrr(rrrrrr sC* 0 06"!# J rrc [[R"SS[55n[R"SS[ 5n[ URXe5upx/n /n [U5Hsn X-n [U S-U-S- URS- 5n U RX- S-5 [R"U5nU R[XUX#XU55 Mu X4$)aPartitions an object into components to be downloaded. Each component is a byte range of the object. The byte ranges of the returned components are mutually exclusive and collectively exhaustive. The byte ranges are inclusive at both end points. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata from the source object with non-pickleable fields removed. dst_url: Destination FileUrl. download_file_name: Temporary file name to be used for the download. decryption_key: Base64-encoded decryption key for the source object, if any. Returns: components_to_download: A list of PerformSlicedDownloadObjectToFileArgs to be used in Apply for the sliced download. rC%sliced_object_download_component_sizerr)rsr rFrrYrrqrr rGrur encode_messager)rr>rrrsliced_download_component_sizerryrzcomponents_to_downloadcomponent_lengthsr}rrrs r_PartitionObjectrO s.$8 jjB>@$A ==!H!NP.$5^$M .  a#JAEn-13C3H3H13LMHX2Q67&445EF!!-a:O.5.8.< >? ! 22rc >SUl[XX#U5up[U 5n [XX5UU 5 [ US5n U R UR 5 SSS5 [U 5HupURUR- S-n[XUURUU5nUUR- n[U [UU[R"5USU[RUS95 M UR[ U U["R$R&UR(R*SS9n[U5U :a[-SUR.-5e[1U[3S 5S 9nUS R4nUb/[7SU 5Hn[9UUUR4X5nM! S n[;U5nUHn[U [UU[R"5UR<SUR>[RS 95 UUR@- nURB=(a) URBRE5RGS 5nU(dMU(aM[-SUR.-5e UU4$!,(df  GN<=f)aDownloads a cloud object to a local file using sliced download. Byte ranges are decided for each thread/process, and then the parts are downloaded in parallel. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata from the source object. dst_url: Destination FileUrl. download_file_name: Temporary file name to be used for download. command_obj: command object for use in Apply in parallel composite uploads. logger: for outputting log messages. copy_exception_handler: For handling copy exceptions during Apply. api_selector: The Cloud API implementation used. decryption_key: Base64-encoded decryption key for the source object, if any. status_queue: Queue for posting file messages for UI/Analytics. Returns: (bytes_transferred, crc32c) bytes_transferred: Number of bytes transferred from server this call. crc32c: a crc32c hash value (integer) for the downloaded bytes, or None if crc32c hashing wasn't performed. Nr+brF)rrrrbytes_already_downloadedTrzSSome components of %s were not downloaded successfully. Please retry this download.rrrrrZzTDownload of %s failed because the server sent back data with an unexpected encoding.)$r{rrrr6truncater enumeraterrr8r`r3rCOMPONENT_TO_DOWNLOADrrrrrrrr!rrrrzr rXr]component_total_sizerrrrr)rr>rrrrrrrr;rrryrr}rrdownload_start_byterrrzr expect_gzipr server_gzips r_DoSlicedDownloadr sH)-%.> n/N+-..%&6&8,&46  &"KK %%&'""89nq    4 4 4q 8D./?/;/8/C/CT1N 3Y5I5IIGIIK""#!,!B!B-E G H:$  (--//#.#E#E#K#K !"* _~%  &(/(;(; < == jj&AB* a=  &  1n %FJqM$8$8:K:NOf&#$45+iGIIK"77!"+"9"9!,!B!B  DE444,,F,,224==fE{;;  !#*#6#6 7 88#* F ""C'&s J  Jc U cURS- n X- S-n USLn URURS9n SnURnU (aUSU-- nSn[ US5nUR U5 URURS9n [ U5n[UUUU UUU5unnUU:d UU S-:a[U5 [SU-5eUX-:HnUU:g=(a U(+nU(aURSU5 OU(aURSU5 [UUURS 9nU(dU(aS nUU- n[U[URUUUS S 9R 5nUU:a|[#[$UU- 5nUR'U5nUU- nUH+nUUR)[*R,"U55 M- UR/[1U55 UU:aM|OU (dUR3S 5 S n[URUU UUUS S9R n[4R6(aQ[ [4R6S5n[8R:"UR'55R nSSS5 U (a&UR[=5:a[?UUUX5n[AU5nU(dUR U5 URCURDURUUU UURFUR[HRJRLURUUU[OU 5S9nU(aURW5 U U- S-n!U!U4$!,(df  N=f![Pa#n URSSU RTU5 eSn A ff=f!U(aURW5 ff=f)a5Downloads an object to a local file using the resumable strategy. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata from the source object. dst_url: Destination FileUrl. download_file_name: Temporary file name to be used for download. gsutil_api: gsutil Cloud API instance to use for the download. logger: for outputting log messages. digesters: Digesters corresponding to the hash algorithms that will be used for validation. component_num: Which component of a sliced download this call is for, or None if this is not a sliced download. start_byte: The first byte of a byte range for a sliced download. end_byte: The last byte of a byte range for a sliced download. decryption_key: Base64-encoded decryption key for the source object, if any. Returns: (bytes_transferred, server_encoding) bytes_transferred: Number of bytes transferred from server this call. server_encoding: Content-encoding string if it was detected that the server sent encoded bytes during transfer, None otherwise. Nrrz component %drzResumable download start point for %s is not in the correct byte range. Deleting tracker file, so if you re-try this download it will start from scratchzResuming download for %szRDownload already complete for %s, skipping download but will run integrity checks.)progress user_projectrr0)rrrr1 Downloading)roverride_total_sizerrrr1r/) rrcompressed_encodingr} object_sizerrr>r)r r z:Caught ResumableDownloadException (%s) for download of %s.),rrrrr6rErir<r6r!rrKrr+r*r;r7rGrLrrsr8rGProgressrrrrrrrHrr]GetObjectMediarr}rrrrPrrhreasonr()"rr>rrrrr)rrrr download_size is_slicedrr download_namerrrrdownload_completeresumingr>bytes_digestedtotal_bytes_to_digest hash_callback bytes_to_readrr@r rrrrs" rrr sD$$q(H'!+-4')**GNN*C,/%%-_}44M "s  % (BGGJ,,gnn,EL$R-L.**Z'+>A+M)*  $&3 4 55 - 0JJ#z1L;L7LH kk,m<  kk '(576$,,. $n1J>1  %j&=&=4A.5.55>  @AE Fm 2 241NBD ww}%-'!H H  $ $S%6%6t%< ="s4y) 2 2kk!n3)#$&'+d11 '::D AW"LL8== B%**.@.BB $R):? gg!""11       (1''&++$55??>>/-2>B2Do* hhj!44q8 O ++U B AB $  NNO88],   hhj sEFN8A-N%/M>CN> N N N<N77N<<N??OcSn[US5n[USURS9n[URUUSS9R n [ R(aQ[[ RS5n [R"U R55R n SSS5 URURURUURUR[ R"R$UR&UUU [)U5S9 n U(aUR+5 URU 4$!,(df  N=f!U(aUR+5 ff=f) a>Downloads an object to a local file using the non-resumable strategy. This function does not support component transfers. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata from the source object. dst_url: Destination FileUrl. download_file_name: Temporary file name to be used for download. gsutil_api: gsutil Cloud API instance to use for the download. digesters: Digesters corresponding to the hash algorithms that will be used for validation. decryption_key: Base64-encoded decryption key for the source object, if any. Returns: (bytes_transferred, server_encoding) bytes_transferred: Number of bytes transferred from server this call. server_encoding: Content-encoding string if it was detected that the server sent encoded bytes during transfer, None otherwise. Nrr)rrr r/)r}rrrr>r)r r )r6rKrr*r;r7rrrrrrrrr}rrrrrrPr() rr>rrrr)rrr>r rrs r!_DownloadObjectToFileNonResumabler s@4 "   # &B6!**A*AC4$ &'+d  11 '::D AW"LL8== B!// %%$))"33<<-+0@0 BO hhj    //' B A  hhj s%A0E 4/D<#A4E < E E E'c ^URRUR5(aYURSR [ R "SUR-555 [SUR-5eURURS9n [U5n [XXt5n[UX5unn[ [RUS5(a[ eS[U'SSS5 UR"=(a U(+n[%UUUR&S9m[)U4S jT=(d 055nSnUR*S :HnS n[,R,"5nU(dU(a0[/UUUUUUUU UUR0S 9 unnS U;a UUS lOtU [4R6R8La[;UUUUUUUS 9unnODU [4R6R<La[?UUUUUUUUS 9unnO[ASU <SU<35e[,R,"5nU=(a URC5RS5n[EUUUUUUUTUU UUU U U S9n[ [RGU5 SSS5 [IUR0[KUUU[JRLUR*SS95 UU- UUU4$!,(df  GN=f!,(df  Nc=f)aDownloads an object to a local file. Args: src_url: Source CloudUrl. src_obj_metadata: Metadata from the source object. dst_url: Destination FileUrl. gsutil_api: gsutil Cloud API instance to use for the download. logger: for outputting log messages. command_obj: command object for use in Apply in sliced downloads. copy_exception_handler: For handling copy exceptions during Apply. allow_splitting: Whether or not to allow sliced download. decryption_key: Base64-encoded decryption key for the source object, if any. is_rsync: Whether or not the caller is the rsync command. preserve_posix: Whether or not to preserve POSIX attributes. use_stet: Decrypt downloaded file with STET binary if available on system. Returns: (elapsed_time, bytes_transferred, dst_url, md5), where time elapsed excludes initial GET. Raises: FileConcurrencySkipError: if this download is already in progress. CommandException: if other errors encountered. rzSkipping attempt to download to filename ending with slash (%s). This typically happens when using gsutil to download from a subdirectory created by the Cloud Console (https://cloud.google.com/console)zInvalid destination path: %srFTN) consider_md5rc3:># UHoTU"54v M g7frqrrrs rru(_DownloadObjectToFile..# rwrxr)rr;rzrzInvalid download strategy z chosen forfile rZ)is_rsyncruse_stet) message_timerrr)'rrrrKrrrr#rrrrropen_files_lockopen_files_maprFrr~rYrzrrrrr;rrrrrrrr!r_ValidateAndCompleteDownloaddeleter`r3 FILE_DOWNLOAD)rr>rrrrrrrrrrrrsliced_downloadrrrr)rr rrrzrr local_md5rts @r_DownloadObjectToFiler! s J !!'--00 KK   N     !"# 87;N;NN OO**GNN*C,-g6+,=,;E/'7w7G'Q#m,e44 $$)-N%& "))A/.A,!&/;2B2I2IK)Fio2oFF)/',,1yy{* #G$4$+$6$/$*$:$03A1;1H1H J  Y '- ($ h77@@ @+L      ',( h77AA A+I       ' ,( /1CE FFYY[(LO$9$9$;$D$DV$L+*6+2+;+2+8+6+4+4+=+7+<+54<:H4<>) ,-''*88',, !" Z !2GY GGsZs +K>K1 K.1 K?cURnSnUHnUU(aMSn O U(a [U5nO[U RXxX5nSnSn[ XX/U5 [ X95 U(d U(dUnO[R "U5nU [":aUR%SU5 Sn[&R("US5n[)US5nUR+[,5nU(a/UR/U5 UR+[,5nU(aM/SSS5 U(aUR55 [R"U5 U(d1[U RUUX5n[ XX/U5 [ X95 U(a[6R8"XUU5 [R"UU5 [;UUU U S 9 S U;aUS $g![ anU(aURS5 SnSnAGNU [R:XaURS5 UnSnSnAGN[ X95 [(a[R"UU[-5 e[R"U5 eSnAff=f!,(df  GNo=f![0a#nS [3U5;a U(aUeSnAGNSnAff=f!U(aUR55 ff=f![ aM [ X95 [(a[R"UU[-5 e[R"U5 ef=f) aValidates and performs necessary operations on a downloaded file. Validates the integrity of the downloaded file using hash_algs. If the file was compressed (temporarily), the file will be decompressed. Then, if the integrity of the file was successfully validated, the file will be moved from its temporary download location to its permanent location on disk. Args: logger: For outputting log messages. src_url: StorageUrl for the source object. src_obj_metadata: Metadata for the source object, potentially containing hash values. dst_url: StorageUrl describing the destination file. need_to_unzip: If true, a temporary zip file was used and must be uncompressed as part of validation. server_gzip: If true, the server gzipped the bytes (regardless of whether the object metadata claimed it was gzipped). digesters: dict of {string, hash digester} that contains up-to-date digests computed during the download. If a digester for a particular algorithm is None, an up-to-date digest is not available and the hash must be recomputed from the local file. hash_algs: dict of {string, hash algorithm} that can be used if digesters don't have up-to-date digests. temporary_file_name: Temporary file name that was used for download. api_selector: The Cloud API implementation used (used tracker file naming). bytes_transferred: Number of bytes downloaded (used for logging). gsutil_api: Cloud API to use for service and status. is_rsync: Whether or not the caller is the rsync function. Used to determine if timeCreated should be used. preserve_posix: Whether or not to preserve the posix attributes. use_stet: If True, attempt to decrypt downloaded files with the STET binary if it's present on the system. Returns: An MD5 of the local file, if one was calculated as part of the integrity check. TFNzDHash did not match but server gzipped the content, will recalculate.zMHash did not match but server may have gzipped the content, will recalculate.z/Uncompressing temporarily gzipped file to %s...r/rUzNot a gzipped filerrr.)rr,rAr;rbr5r"rrrrrrenamerrgrCrrvrrZr6rr[r\rrr(rBdecrypt_downloadrf)rrr>rrrr)rttemporary_file_namerrrrrrfinal_file_namedigesters_succeededr+r\digest_verifiedhash_invalid_exceptionrunzipped_temporary_file_namer`f_outrs rrru sh''/ c S>>! .y9L.z/F/F/8/6JL/ "2w52 ;#6 $7#F#Fw#O 7" kkC!#G -t4g ,d 3u||O, ++d o.$d 4  II!"  01H1H1:1M18Ll6$4! 7 w1M%')) (/:_-&.,:<  l  u   ll"#o  ( ll  !o 7 ! ! %!$CC E  %& - R 4 3 % Q ',B%$ %    !  7 ! ! ( (+J J L   ./  sG3#JA J "J0K) J'J)J6A JJ JJK J K &K>K K  K K&)AMc [U5n[RRUR5nU(a[R "U5 [URS5n[R"5n[R"XG5 SSS5 UR5(dUR5 [R"5n [U[!UUU [ R"U(a UR$OSSS95 U W- [RR'UR5US4$![ a*nUR[R:waeSnAGNSnAff=f!,(df  N=f)akCopies a local file to a local file. Args: src_url: Source FileUrl. dst_url: Destination FileUrl. status_queue: Queue for posting file messages for UI/Analytics. src_obj_metadata: An apitools Object that may contain file size, or None. Returns: (elapsed_time, bytes_transferred, dst_url, md5=None). Raises: CommandException: if errors encountered. NrUTr)rjrrhrirrrrrr6rshutil copyfileobjr r(r`r3FILE_LOCAL_COPYrr]) rrr;r>src_fprrdst_fprrs r_CopyFileToFiler3 s'  (& WW__W00 1( kk(  G  &&J v&'      LLN YY[('*::0@',,d !" Z 1D1D!Ew  '  ELL   !'&s#E),E7 E4 E//E47 Fcgrqr)rs r_DummyTrackerCallbackr5K srcf[R(a%URUR:wa [S5e[R(d/UlSn[R (aQ[ [R S5n [R"U R55RnSSS5 [U5n [[5n [R"5n [UUR UU UUS9n SnUR S:Xa=UR#U U[R$UUR[&UR U S9nO`UR)U U[R$UUR[&UR [+UR,UUSS9R[.U S 9 n[R"5n[1X`X!U5 URO5n[QUURL5Ul&[SUR,[UUUU[TRVUR S S95 X- UR UURX4$!,(df  GN=f![2a [4(at[6R8"UR:UR<UR>S 9nUR@[B-Ul[EU5nURGUUURUU S 9 URIURJUR@URLURS 9 ef=f)aCopies from src_url to dst_url in "daisy chain" mode. See -D OPTION documentation about what daisy chain mode is. Args: src_url: Source CloudUrl src_obj_metadata: Metadata from source object dst_url: Destination CloudUrl dst_obj_metadata: Object-specific metadata that should be overidden during the copy. preconditions: Preconditions to use for the copy. gsutil_api: gsutil Cloud API to use for the copy. logger: For outputting log messages. decryption_key: Base64-encoded decryption key for the source object, if any. Returns: (elapsed_time, bytes_transferred, dst_url with generation, md5 hash of destination) excluding overhead like initial GET. Raises: CommandException: if errors encountered. z"Cross-provider cp -p not supportedNr/)rr rr)r5rrrrrrr4r ) r5rrrrrr rArr)rr rrTr)-rrrNotImplementedErroraclrr6rrrr7r]rRr rr rr7rrrDr*r;r5rWr"rrrrrr|rrrPrrrr}rr/r`r3FILE_DAISY_COPYr~)rr>rrLrrrrr rrrr upload_fpr:rrrrs r_CopyObjToObjDaisyChainModer;P s@** nn& B CC - -// %88$ ?7 ,,w||~6;; @,,<=1&9yy{* 0 5 5 *4G2C/= ?) /a --(*55##  " "..0O!66(*55##  " "5  # #& ()-..70OYY[( fw%'.}}*4Z5D5O5OQ*'*::',, !"  !1!6!6  ! ! ##i @ ?d  077$$!((## % '22> ?5nE2,%,^^-B-B D G//#//'6'A'A%,^^5 ' s/I I+ I(+CL0c[5nU(aFUR/SQ5 U(aURS/5 U(dURS/5 OvUR/SQ5 U(aURS[-S/5 U(a;S[-S[-S[-S[ -S[ -/nURU5 U(aURSS/5 U(aURS/5 [U5$)aDetermines the metadata fields needed for a copy operation. This function returns the fields we will need to successfully copy any cloud objects that might be iterated. By determining this prior to iteration, the cp command can request this metadata directly from the iterator's get/list calls, avoiding the need for a separate get metadata HTTP call for each iterated result. As a trade-off, filtering objects at the leaf nodes of the iteration (based on a remaining wildcard) is more expensive. This is because more metadata will be requested when object name is all that is required for filtering. The rsync command favors fast listing and comparison, and makes the opposite trade-off, optimizing for the low-delta case by making per-object get metadata HTTP call so that listing can return minimal metadata. It uses this function to determine what is needed for get metadata HTTP calls. Args: dst_is_cloud: if true, destination is a Cloud URL. skip_unsupported_objects: if true, get metadata for skipping unsupported object types. preserve_acl: if true, get object ACL. is_rsync: if true, the calling function is rsync. Determines if metadata is needed to verify download. preserve_posix: if true, retrieves POSIX attributes into user metadata. delete_source: if true, source object will be deleted after the copy (mv command). file_size_will_change: if true, do not try to record file size. Returns: List of necessary field metadata field names. )rcomponentCountcontentDispositionrrrrzr{r|r}r~ mediaLinkmetadatametagenerationr timeCreatedr8r) rzrrr{r|r?r~rr}z metadata/%srBr)setrsrerarcrdrgr) dst_is_cloudrrrr delete_sourcefile_size_will_changesrc_obj_fields_set posix_fieldss rGetSourceFieldsNeededForCopyrI sNu&( )  !;] KL * $ * $ ( " ) # ( " l - ~./  !!rZim)nearlinecoldlinearchivecURS:XaU(aUR(aUR(aURR5n[R US5nU(aWU[ UR5-n[R"5U:a%URSX0RU[-5 ggggggg)aWarns when deleting a gs:// object could incur an early deletion charge. This function inspects metadata for Google Cloud Storage objects that are subject to early deletion charges (such as Nearline), and warns when performing operations like mv that would delete them. Args: src_url: CloudUrl for the source object. src_obj_metadata: source object metadata with necessary fields (per GetSourceFieldsNeededForCopy). logger: logging.Logger for outputting warning. rNzWarning: moving %s object %s may incur an early deletion charge, because the original object is less than %s days old according to the local system time.) rrBrrEARLY_DELETION_MINIMUM_LIFETIMErFrbrrKrru)rr>robject_storage_classearly_deletion_cutoff_secondsminimum_delete_ages r"WarnIfMvEarlyDeletionChargeAppliesrTIsnn!1""'7'D'D+88>>@$C$G$Gd%$!$ ' !1!=!= > ? ) )  ; !"4"4 )_ <  > * % (E""2rcURS:Xa1[R(aURS:Xa [ 5eggg)a Skips unsupported object types if requested. Args: src_url: CloudUrl for the source object. src_obj_metadata: source object metadata with storageClass field (per GetSourceFieldsNeededForCopy). Raises: SkipGlacierError: if skipping a s3 Glacier object. s3GLACIERN)rrrrSkipGlacierError)rr>s rMaybeSkipUnsupportedObjectrYhs?nn66##y0  17rcUR(aT[URR[5nU(d'[ SURR<SU<35eU$g)aEnsures a matching decryption key is available for the source object. Args: src_url: CloudUrl for the source object. src_obj_metadata: source object metadata with optional customerEncryption field. Raises: EncryptionException if the object is encrypted and no matching key is found. Returns: Base64-encoded decryption key string if the object is encrypted and a matching key is found, or None if object is not encrypted. z(Missing decryption key with SHA256 hash z#. No decryption key matches object N)r{rQ keySha256r r)rr>rs rGetDecryptionCSEKr\ys`((1++55v?N    . . 8 8' C DD )rcU(aUR5nO0n[U5nUR5(aURS:Xa [ U5nO [ 5nSnSnSnUR5(Ga3UR5(a1URUR:Xa[ R(dSn[ R(a [XU5 [X5 [X5nURnURUl [ R(aUR5(anURS:XaUR(d[!SU-5eURUlURS:Xa[#U5nU(a [%UU5 OU(a[&R("XU5nOUn[+U5nUR75(dUR95(aSnOUU(a%UR(aU(d URnO)[:R<R?UR@5n[ RB(aU REURFS U5 URS:Xa4US:wa.Ub+U[H:a![!S U<S [K[H5<S35e[L(afURO5(aQUR75(a<[PRR"[+U5RU5[:RV5 [ RX(aURZ(a []S5eSUl-URO5(a8[:R<R_UR@5(a [a5eUR5(aAURcURdUR@URS9nU(a [a5eUR5(aUR@Ul4URdUl5UR5(a#UR@Ul4URdUl5O [mUU5 [o[p5nU(a?URr[tRv:Xa!URS:XaURxUl=U(a [}UUSS9 [ R~(a[ R~Ul@[pR"SS5[:XaSUlC[XUU5 UR5(GaURO5(aW[UR[UU[R"5[RUSS95 [UUUUUUUUUU U US9 $U(aR[UR[UU[R"5[RUSS95 [UUUUUUUS9$[UR[UU[R"5[RUSS95 [UUUUUUUUS9$UR5(a>[UUUUUUUUUUU UU S9 nU(a [:R"URh5 U$[UR[UU[R"5[RUSS95 [UUURUS9nUR75(d+UR75(d[UR@UU U S9 U$![,a`nSU<S[/U5<S 3nUR0(a+U=R2S - slUR5U5 SnAg[!U5eSnAff=f![fa SnGNf=f)afPerforms copy from src_url to dst_url, handling various special cases. Args: logger: for outputting log messages. src_url: Source StorageUrl. dst_url: Destination StorageUrl. gsutil_api: gsutil Cloud API instance to use for the copy. command_obj: command object for use in Apply in parallel composite uploads and sliced object downloads. copy_exception_handler: for handling copy exceptions during Apply. src_obj_metadata: If source URL is a cloud object, source object metadata with all necessary fields (per GetSourceFieldsNeededForCopy). Required for cloud source URLs. If source URL is a file, an apitools Object that may contain file size, or None. allow_splitting: Whether to allow the file to be split into component pieces for an parallel composite upload or download. headers: optional headers to use for the copy operation. manifest: optional manifest for tracking copy operations. gzip_exts: List of file extensions to gzip, if any. If gzip_exts is GZIP_ALL_FILES, gzip all files. is_rsync: Whether or not the caller is the rsync command. preserve_posix: Whether or not to preserve posix attributes. gzip_encoded: Whether to use gzip transport encoding for the upload. Used in conjunction with gzip_exts. Streaming files compressed is only supported on the JSON GCS API. use_stet: If True, will perform STET encryption or decryption using the binary specified in the boto config or PATH. Returns: (elapsed_time, bytes_transferred, version-specific dst_url) excluding overhead like initial GET. Raises: ItemExistsError: if no clobber flag is specified and the destination object already exists. SkipUnsupportedObjectError: if skip_unsupported_objects flag is specified and the source is an unsupported type. CommandException: if other errors encountered. rNFTzZNo OWNER permission found for object %s. OWNER permission is required for preserving ACLs.rVzError opening file "z": r"rr"zV" exceeds the maximum gsutil-supported size for an S3 upload. S3 objects greater than zB in size require multipart uploads, which gsutil does not support.zASpecifying x-goog-if-generation-match is not supported with cp -nrr)overriderCrDr)rrrrrr)rrr)r;r>r#)TcopyrorrrprrrrrTrYr\rrrr8r!rqrlrBencrypt_uploadrjrrcontinue_on_errorrrr r rrhr]rrSetrS3_MAX_UPLOAD_SIZErtrkrmsvcrtsetmodefilenoO_BINARYr gen_matchr rItemExistsErrorGetObjectMetadatarrrrr2rRr crypto_typerOCMEK crypto_key kmsKeyNamermrrrFrWr~rr`r;r3rrr!rrr9r;rrgr0r3rf)rrrrrrr>rheadersmanifestrrrrrdst_obj_headersrLrr8rcopy_in_the_cloudr9acl_textsource_stream_urlrre dst_objectruploaded_metadataresults r PerformCopyrysn llnOO /? gnn4,_=M!OM. 7>>!A # / /))(FKw9&wAN#((L#3#?#? ++0B0B0D0D 4 (8(<(< ,.5 67 7.11 4 01AB  ()98 D#227VL! (/0ABW^^--l .33H&**lWW__%6%B%BCl)) LL##V\: nnD''  #$67 9 ::ZG%%''G,<,<,>,> NN'07792;;G'' 9 ::!"mrww~~g.A.ABB       11'2E2E292E2E;B>>2K  #//%11&11 ' 3 3g'783F;))]-?-??$$9$D$D!')9EJ//$;$N$N! ZZ.)-== $FW.>?   ! ! giik#.#<#<'$  &'#7#3#*#-#)#.#93B2@,42@,4 6 6   ! ! giik#.#>#>'$  &'%W%5%,%5%2%/4B DD  ! ! giik#.#>#>'$  &'))9)0)9)6)3)/8FHH.g.@.:.5.>.;.8.4.9.D8A>M;G I  $))*   ! ! giik#.#>#>'$  &'w&,6,C,C0@Bf     (8(8(:(:"7#6#6#3,42@ BmI (293q6Bg  & &$$)$ Ww''(z s1 ]<!/_)< _&A _! _!!_&) _98_9cR\rSrSrSrSrSrSrSrSr Sr SS jr S r S r S rg )Manifestiz2Stores the manifest items for the CpCommand class.c0Ul0Ul[R"5Ul[ R RU5UlUR5 UR5 grq) itemsmanifest_filterrArlockrrh expanduser manifest_path_ParseManifest_CreateManifestFile)rrhs rrManifest.__init__sPDJD*557DI++D1Drc6[RRUR5(a[ URS5nSn[ R "U5nUHOnU(a#URS5nURS5nSnUWnUWnUS;dMAXRU'MQ SSS5 gg![a [SUR-5ef=f!,(df  g=f![a [S UR-5ef=f) zpLoad and parse a manifest file. This information will be used to skip any files that have a skip or OK status. rTSourceResultz$Missing headers in manifest file: %sF)OKskipNzCould not parse %s) rrhrrr6csvreaderindexrr!r~r) rf first_rowrrow source_index result_indexsourcerxs rrManifest._ParseManifests H ** + +$$$c *a)::a=&c;"yy2 "yy2 I&F&F'.4""6*+ * , ;&'M'+'9'9(:;;;+ *& H 1D4F4FF GGHsHAC5%C$,"B>C$"C$4C5>#C!!C$$ C2.C52C55#DcXR;$)z?Returns whether the specified src url was marked as successful.)r~)rsrcs r WasSuccessfulManifest.WasSuccessfuls && &&rc[RRUR5(a.[R"UR5R S:Xa[ R(aH[URSSS9n[R"U5nUR/SQ5 SSS5 g[URSS5n[R"U5nUR/SQ5 SSS5 gg!,(df  g=f!,(df  g=f![a [S 5ef=f) z;Opens the manifest file and assigns it to the file pointer.rrr newline) r DestinationStartEndMd5UploadIdz Source SizezBytes Transferredr DescriptionNrUrzCould not create manifest file.)rrhrrrst_sizer8rHr6rwriterwriterowrr!)rrrs rrManifest._CreateManifestFiles#@ww~~d0011 774%% & . .! 3 77D&&R8AZZ]F OO  98D&&a0AZZ]F OO  10% 49810 @ > ??@sNBD(*D2D(;D(*D<D( DD(D( D%!D(%D((D>cnUcgXR;aX0RUU'gX#0URU'grqr})rrrvalues rrc Manifest.Sets5 }  jj"jjoc djjorcURUSU5 URUSU5 URUS[RR[RRS9R SS95 g)N source_uridestination_urirtztzinfo)rcdatetimenowtimezoneutcr)r source_urlrs r InitializeManifest.Initialize)sbHHZz2HHZ*O<HHZx'8'8'<'<@Q@Q@U@U'<'V'^'^fj'^'klrc`URUSU5 URUSU5 URUSU5 URUS[RR[RRS9R SS95 UR U5 URU5 g)Nbytesrx descriptionrrr)rcrrrrr_WriteRowToManifestFile_RemoveItemFromManifest)rrrrxrs r SetResultManifest.SetResult/sHHZ"34HHZ6*HHZ 4HHZX%6%6%:%:h>O>O>S>S%:%T%\%\dh%\%ij  ,  ,rc URUnUSUSSUSR5-SUSR5-SU;aUSOSSU;aUSOSS U;a[US 5OSS U;a[US 5OSUS US / nUVs/sHn[R"U5PM nnUR  [ (a,[R(a[URS SSS9nO[URS S5n[R"U5nURU5 UR5 SSS5 gs snf!,(df  g=f)zBWrites a manifest entry to the manifest file for the url argument.rrz%sZrrr.r  upload_idrrrxrarrN)r} isoformatrr8r*rrkrHr6rrrrr()rrrow_itemrrrrs rr Manifest._WriteRowToManifestFile7sCzz#H"# &0022 $..00 H,"!,!8b!'8!3HV ")X"5HW 2 D04 4teCNN5 !tD 4   ##S!R 8 ##S! ,zz!}f oodggi  5 s EBE E"cURU grqr)rrs rr Manifest._RemoveItemFromManifestTs 3r)r}rr~rN)r )rrrrrrrrrrcrrrrrrrrr{r{s7:H@'%@N %m -:rr{c\rSrSrSrSrg)rji[zHException class for objects that are skipped because they already exist.rNrrrrrjrj[sPrrjc,^\rSrSrSrU4SjrSrU=r$)SkipUnsupportedObjectErrori`CException for objects skipped because they are an unsupported type.c8>[[U] 5 SUlg)NUnknown)superrrunsupported_typer __class__s rr#SkipUnsupportedObjectError.__init__cs $d46%Drrrrrrrrr __classcell__rs@rrr`K&&rrc,^\rSrSrSrU4SjrSrU=r$)rXihrc8>[[U] 5 SUlg)NrW)rrXrrrs rrSkipGlacierError.__init__ks D*,%Drrrrs@rrXrXhrrrXcURnUR5(aTUR[S5SnUR U5S:XagSUR U5R U5S-$UR5(aSUR-$URR U5R U5Sn[XA5$)a)Returns the path section before the final directory component of the URL. This handles cases for file system directories, bucket, and bucket subdirectories. Example: for gs://bucket/dir/ we'll return 'gs://bucket', and for file://dir we'll return file:// Args: url: StorageUrl representing a filesystem directory, cloud bucket or bucket subdir. exp_src_url: StorageUrl representing the fully expanded object to-be-copied; used for resolving cloud wildcards. Returns: String name of above-described path, sans final path separator. zfile://Nr z file://%srz%s://) rrrrfindrrrr$ResolveWildcardsInPathBeforeFinalDir)rrsep past_schemepath_sans_final_dirs rrrps #]]__..Y1K"  ;--c2==cB1E EE\\^^ SZZ --c2==cB1E -.A OOrct[U5(dU$[U5RRS5nURRS5n[ [ U55Hn[X$5(dMX4X$'M SR U5nUR5nXVlUR$)aBReturns the path section for a bucket subdir with wildcards resolved. This handles cases for bucket subdirectories where the initial source URL contains a wildcard. In this case, src_url must be wildcard-expanded before calculating the final directory. Example: A bucket containing: gs://bucket/dir1/subdir/foo gs://bucket/dir2/subdir/foo and source URL gs://bucket/*/subdir and src_url_path_sans_final dir gs://bucket/* should yield final path gs://bucket/dir1 or gs://bucket/dir2 according to the expanded source URL. Args: src_url_path_sans_final_dir: URL string with wildcards representing a bucket subdir as computed from GetPathBeforeFinalDir. exp_src_url: CloudUrl representing the fully expanded object to-be-copied. Returns: String name of above-described path, sans final path separator. r) r.r1rrNr rrrr)rrwildcarded_src_obj_pathexpanded_src_obj_pathpath_segment_indexresolved_src_pathfinal_path_urls rrrs6 5 6 6 &&1!##.;uuSz%1177<!#&=">?/CDD 31 @ hh67$$&.0  " ""rc^[X5n[[X15S5n[X5nX44$)a Gets info about a file partition for parallel file/object transfers. Args: file_size: The number of bytes in the file to be partitioned. max_components: The maximum number of components that can be composed. default_component_size: The size of a component, assuming that max_components is infinite. Returns: The number of components in the partitioned file, and the size of each component (except the last, which will have a different size iff file_size != 0 (mod num_components)). r)rrmaxrG)rwrdefault_component_sizeryrzs rrqrqs6!C.s>:A>.!;.  ))rc[X5nURURURURUR S9 g![ a gf=f)zGWrapper func to be used with command.Apply to delete temporary objects.rN)rJrrrr}rr)r url_to_deleterrs rrrs]"35*  M55)55'4'?'?%2%9%9;    s:A AAc /nUVs/sHoURPM nnUHnXv;dM URX5 M /n/n /n UGHn U RUR5;dU RU;aEUR5n U RU lU RU lU RU 5 MwX Rn [ U R U RU R5n[U5nU RnURURURURUR/SQS9nURnUU:waURU 5 URU R5 U R(a?U RR5nU RUlU RU5 GMGMU RR5n U RU lU RXR45 URU R5 GM U (a [ R""S[%U 55 XJU 4$s snf![a SnGN+f=f)a/Determines course of action for component objects. Given the list of all target objects based on partitioning the file and the list of objects that have already been uploaded successfully, this function determines which objects should be uploaded, which existing components are still valid, and which existing components should be deleted. Args: dst_args: The map of file_name -> PerformParallelUploadFileToObjectArgs calculated by partitioning the file. existing_components: A list of ObjectFromTracker objects that have been uploaded in the past. bucket_url: CloudUrl of the bucket in which the components exist. gsutil_api: gsutil Cloud API instance to use for retrieving object metadata. Returns: components_to_upload: List of components that need to be uploaded. uploaded_components: List of components that have already been uploaded and are still valid. Each element of the list contains the dst_url for the uploaded component and its size. existing_objects_to_delete: List of components that have already been uploaded, but are no longer valid and are in a versioned bucket, and therefore should be deleted. )r{r|r~)r}rrNz0Found %d existing temporary components to reuse.)rrurrr}r$rrrrTrrkrrr~rrrr)r{r bucket_urlrrrexisting_component_namescomponent_nameobjects_already_chosenrrtracker_objectrdst_arg file_partrr dst_metadata cloud_md5!invalid_component_with_generations rrrsc:-@-@ -@!n5!!(":;! "+n""(--/9""&<<    c&22co%00cn '',112G))7+=+= ,,.I5Y?K g11      ''>>: 2$>?  " "-4OO,A,A,C)7E7P7P)4"))*KL # OO ! ! #c%00cn  #':':!;<##N$>$>?a,d LLC()+ 5O PPIX isI1AI66 JJrq) FFFFFFFNFNN)F)NF)FN)FF)FFF)NTFF)NN)NrNN)TNFFF)FFFF) NTNNNFFFF( r __future__rrrrr% collectionsrrrrrZrrr,operatorrrrrrHrr.r8rr$rWrrr six.movesr apitools.base.protorpcliter botor r3rgslib.cloud_apir r rrrrrrrrrrgslib.commands.composergslib.commands.configrrrrrrgslib.cs_api_maprgslib.daisy_chain_wrapperr gslib.exceptionr!r"r#gslib.file_partr$gslib.parallel_tracker_filer%r&r'r(r)gslib.progress_callbackr*r+ gslib.resumable_streaming_uploadr,r-gslib.storage_urlr.r/r0r1"gslib.third_party.storage_apitoolsr2rgslib.thread_messager3r4gslib.tracker_filer5r6r7r8r9r:r;r<r=r>r?r@ gslib.utilsrArBrCrDgslib.utils.boto_utilrErFrGrHrIgslib.utils.cloud_api_helperrJrKgslib.utils.constantsrLrMrNgslib.utils.encryption_helperrOrPrQrRgslib.utils.hashing_helperrSrTrUrVrWrXrYrZr[r\gslib.utils.metadata_utilr]&gslib.utils.parallelism_framework_utilr^r_r`gslib.utils.posix_utilrarbrcrdrerfrggslib.utils.system_utilrhrirjrkgslib.utils.translation_helperrlrmrnrorprqgslib.utils.unit_utilrrrsrtrurvgslib.wildcard_iteratorrwrHrrre is_availabletop_level_managerrrrrrrtrrrrrrr[rrrrdrrr+rrrrrrrrrrrrrr!r,rArWrbrfrnrrrrrrrrrrr2r;rLrRrdrmrrrrrobjectrrrrrr!rr3r5r;rIrPrTrYr\ryr{rjrrXrrrqrrrrrrs /&%' "     0 1-$/-1)694=,4RMOOJ@(7,1+$EELQF??P.862W,690>313B>A.@92!+!;2/46<E::&7AFA7IBC73:-8?9=WH-9+,-=+2/8.I=?DCH/631):777 $46CCC( 9 9IMP -779!",M4)3+)% )3+))*% %/'%!" '*;', (46CCC( 9 9IMP#="G"G"I   IyI +9 x, /  %*$)%*.3#(&+&+$(27,0,0!@! /6L40?f :% `$)ePD=J! @DFL"%?d*. %JZ K2-2`'R15 =>"D"& $$"nb RvRji &&&1&P<,#^*0  bQr