~cSrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK r SSK r SSK J r SSK J r SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SrSrSr"SS\R25r\R6"SSS/5r\R6"S/SQ5rSrSrSr Sr!Sr"Sr#S/Sjr$Sr%S r&S!r'S"r(S/S#jr)S$r*S%r+S0S&jr,S'r-S/S(jr.S)r/S*r0S+r1S,r2S1S-jr3S.r4g)2zUtilities for tracker files.)absolute_import)division)unicode_literalsN)encryption_util)errors)log) properties)files)hashing) platforms)scaled_integerdz[/\\]c,\rSrSrSrSrSrSrSrSr Sr g ) TrackerFileType-uploaddownloaddownload_componentparallel_uploadsliced_downloadrewriteN) __name__ __module__ __qualname____firstlineno__UPLOADDOWNLOADDOWNLOAD_COMPONENTPARALLEL_UPLOADSLICED_DOWNLOADREWRITE__static_attributes__r;lib/googlecloudsdk/command_lib/storage/tracker_file_util.pyrr-s! & (+%/%/ 'r%rCompositeUploadTrackerDataencryption_key_sha256 random_prefixResumableUploadTrackerDatacompleter(serialization_datacl[US5nU(dSn[U5"SRX55$)zEdits error to use custom unwritable message. Args: error (Exception): Python error to modify message of. tracker_file_path (str): Tracker file path there were issues writing to. Returns: Exception argument with altered error message. strerrorz [No strerror]zCould not write tracker file ({}): {}. This can happen if gcloud storage is configured to save tracker files to an unwritable directory.)getattrtypeformat)errortracker_file_pathoriginal_error_texts r&"_get_unwritable_tracker_file_errorr6@s< z2 ) eQ 6 88r%c[RRRR 5n[ R "U5 U$)zLooks up or creates the gcloud storage tracker file directory. Resumable transfer tracker files will be kept here. Returns: The path string to the tracker directory. )r VALUESstoragetracker_files_directoryGetr MakeDir)tracker_directorys r&#_create_tracker_directory_if_neededr>Ss8!''//GGKKM--!" r%c[RR5[RR:XaM[R R RR5(a[R"U5$U$)z7Converts colons and characters that make Windows upset.) r OperatingSystemCurrentWINDOWSr r8r9,convert_incompatible_windows_path_charactersGetBoolMakePathWindowsCompatible file_names r&_windows_sanitize_file_namerHas_'')Y-F-F-N-NN    # # P P X X Z Z  . .y 99 r%c~[U5[:a*[R"SR [U55eg)Nz/File name is over max character limit of {}: {})len_MAX_FILE_NAME_LENGTHrErrorr2rFs r&raise_exceeds_max_length_errorrMks8^++ ,,9@@ !9  ,r%c[R"URS55n[SR UR 5US[ -S55$)aApplies a hash function (SHA1) to shorten the passed file name. The spec for the hashed file name is as follows: TRACKER__ 'hash' is a SHA1 hash on the original file name, and 'trailing' is the last chars of the original file name. Max file name lengths vary by operating system, so the goal of this function is to ensure the hashed version takes fewer than _MAX_FILE_NAME_LENGTH characters. Args: file_name (str): File name to be hashed. May be unicode or bytes. Returns: String of shorter, hashed file_name. zutf-8z{}.{}N)hashlibsha1encoderHr2 hexdigest*_TRAILING_FILE_NAME_CHARACTERS_FOR_DISPLAY)rGname_hash_objects r&get_hashed_file_namerVtsT \\)"2"27";< $ nn  $ $ & BCCE F r%c[U5nSRURR5U5nUcUnOUSRU5-n[ U5 [ R RX&5nU$)aHashes and returns a tracker file path. Args: tracker_file_name (str): The tracker file name prior to it being hashed. tracker_file_type (TrackerFileType): The TrackerFileType of res_tracker_file_name. resumable_tracker_directory (str): Path to directory of tracker files. component_number (int|None): The number of the component is being tracked for a sliced download or composite upload. Returns: Final (hashed) tracker file path. Raises: Error: Hashed file path is too long. z {}_TRACKER_{}z_{})rVr2valuelowerrMospathjoin)tracker_file_nametracker_file_typeresumable_tracker_directorycomponent_numberhashed_tracker_file_nametracker_file_name_with_typefinal_tracker_file_namer4s r&_get_hashed_tracker_file_pathrds,22CD / 6 6##%'?!99ELL=!!89ggll! r%cU[RLaaUb URRS5un nO URnSR UR X@R R5nGOU[RLaOSR [RRUR5UR R5nGOxU[RLaPSR [RRUR5UR RU5nGOU[RLa=SR UR URUUR R5nOU[RLaNSR [RRUR5UR R5nOdU[RLaQSR UR URUR URUR R5n[!W5n[#5n[%UUUU5$)aRetrieves path string to tracker file. Args: destination_url (storage_url.StorageUrl): Describes the destination file. tracker_file_type (TrackerFileType): Type of tracker file to retrieve. source_url (storage_url.StorageUrl): Describes the source file. component_number (int): The number of the component is being tracked for a sliced download or composite upload. Returns: String file path to tracker file. _z resumable_upload__{}__{}__{}.urlzresumable_download__{}__{}.etagz#resumable_download__{}__{}__{}.etagz#parallel_upload__{}__{}__{}__{}.urlzsliced_download__{}__{}.etagz!rewrite__{}__{}__{}__{}__{}.token)rr resource_name rpartitionr2 bucket_nameschemerXrrZr[realpathr r!r"r#get_delimiterless_file_pathr>rd) destination_urlr^ source_urlr` object_namerfraw_result_tracker_file_nameresult_tracker_file_namer_s r&get_tracker_file_pathrrs /000#*77BB3Gk1a#11k$F#L#L##[2H2H2N2N$P O444#D#K#K 667$$$& O>>>$I#O#O 667$$&6$8 O;;;$I#O#O##_%B%BJ$$$& O;;;#A#H#H 667$$$& O333#F#M#M 8 8##_%B%B$$$& 9"!D E &!  r%c:[R"[SU5$)zReturns a string representation of the given path without any delimiters. Args: file_path (str): Path from which delimiters should be removed. Returns: A copy of file_path without any delimiters. rf)resubRE_DELIMITER_PATTERN) file_paths r&rlrls $c9 55r%c [U[R5nU/nSn[R"U5n[ R "U5SnU(aUR5 [U5H+nUR[U[RUS95 M- U$![Ra UsU(aUR5 $$f=f!U(aUR5 ff=f)a'Gets a list of tracker file paths for each slice of a sliced download. The returned list consists of the parent tracker file path in index 0 followed by component tracker files. Args: destination_url: Destination URL for tracker file. Returns: List of string file paths to tracker files. Ntotal_componentsr`) rrrr"r FileReaderjsonloadMissingFileErrorcloserangeappendr )rmparallel_tracker_file_pathtracker_file_paths tracker_fileryis r&'_get_sliced_download_tracker_file_pathsrs 566 823,##$>?Lyy./AB ! "a   . . ! #     s#/B$$C:CCCC2cU(a<[RRU5(a[R"U5 ggg)z"Deletes tracker file if it exists.N)rZr[existsremover4s r&delete_tracker_filer+s-277>>*;<<II =r%c[U5nUHn[U5 M [[U[R55 g)aDeletes all tracker files for an object download. Deletes files for different strategies in case download was interrupted and resumed with a different strategy. Prevents orphaned tracker files. Args: destination_url (storage_url.StorageUrl): Describes the destination file. N)rrrrrr)rmsliced_download_tracker_filesrs r&delete_download_tracker_filesr1s@#J#3l %4O_-E-EFHr%c URRURRURURRURR4n[ U5(d[ R "S5eUR=(d URnSnU(aUR(a[URR[R5(a}URRnUR[RR La UR"nO3UR[RR$La UR&nU[(R*"[,R.R0R2R555[7USS5[7USS5[7USS5UU4nXH-n SR9U V s/sHn [;U 5PM sn 5R=S5n [>R@"U 5n U RC5$s sn f)aCreates an MD5 hex digest of the parameters for GCS rewrite call. Resuming rewrites requires that the input parameters are identical, so the tracker file needs to represent the input parameters. This is done by hashing the API call parameters. For example, if a user performs a rewrite with a changed ACL, the hashes will not match, and we will restart the rewrite. Args: source_object_resource (ObjectResource): Must include bucket, name, etag, and metadata. destination_object_resource (ObjectResource|UnknownResource): Must include bucket, name, and metadata. destination_metadata (messages.Object|None): Separated from destination_object_resource since UnknownResource does not have metadata. request_config (request_config_factory._RequestConfig|None): Contains a variety of API arguments. Returns: MD5 hex digest (string) of the input parameters. Raises: Error if argument is missing required property. z"Missing required parameter values.Nprecondition_generation_match!precondition_metageneration_matchpredefined_acl_stringUTF8)" storage_urlrirgetagallrrLdecryption_key_hash_sha256kms_key resource_args isinstanceencryption_keyr EncryptionKeyr1KeyTypeCSEKsha256CMEKkeyr ParseIntegerr r8r9copy_chunk_sizer;r0r\strrRr get_md5rS) source_object_resourcedestination_object_resourcedestination_metadatarequest_configmandatory_parameterssource_encryptiondestination_encryptionroptional_parametersall_parameters parameterparameters_bytesparameters_hashs r&,hash_gcs_rewrite_parameters_for_tracker_filerDs61<<HH0<<JJ0555AAMM5AAOO Q ! " " ,,; <<77%$$ 55--<< ..00  & & 5 5C xx?**///"zz _,,11 1"ww!!    # # 3 3 7 7 9; n=tD nA4H n5t< (=.WWnnc)nn &vOO$45/  " " $$s2Ic[R"SRU55 [R"U[R [R -[R-S5n[R"US5nURU5 SSS5 g!,(df  g=f![an[X@5eSnAff=f)z/Creates a tracker file, storing the input data.zWriting tracker file to {}.iwN) rdebugr2rZopenO_WRONLYO_CREATO_TRUNCfdopenwriteOSErrorr6)r4datafile_descriptor write_streames r&_write_tracker_filers)) ) 0 01B CDCgg/ kkBJJ6CULO ?C (L ) ( ( C ,Q BBCs6AB2B!B2! B/+B2/B22 C < CC cF[R"U5n[X5 g)zCreates a tracker file and writes JSON to it. Args: tracker_file_path (str): The path to the tracker file. data (object): JSON-serializable data to write to file. N)r|dumpsr)r4r json_strings r&_write_json_to_tracker_filers 4 +'5r%cJ[UUS9n[XR55 g)aUpdates or creates a tracker file for a composite upload. Args: tracker_file_path (str): The path to the tracker file. random_prefix (str): A prefix used to ensure temporary component names are unique across multiple running instances of the CLI. encryption_key_sha256 (str|None): The encryption key used for the upload. Returns: None, but writes data passed as arguments at tracker_file_path. )r(r)N)r'r_asdict)r4r)r(rs r&#write_composite_upload_tracker_filers% $1! #$/@r%cL[UUUS9n[XR55 g)aUpdates or creates a tracker file for a resumable upload. Args: tracker_file_path (str): The path to the tracker file. complete (bool): True if the upload is complete. encryption_key_sha256 (Optional[str]): The encryption key used for the upload. serialization_data (dict): Data used by API libraries to resume uploads. Returns: None, but writes data passed as arguments at tracker_file_path. r+N)r*rr)r4r,r(r-rs r&#write_resumable_upload_tracker_filers( $1+ -$/@r%cURURS.nUbUb[R"S5eX$S'UbX4S'[ X5 g)aUpdates or creates a tracker file for component or multipart download. Args: tracker_file_path (str): The path to the tracker file. source_object_resource (resource_reference.ObjectResource): Needed for object etag and optionally generation. slice_start_byte (int|None): Where to resume downloading from. Signals this is the tracker file of a component. total_components (int|None): Total number of components in download. Signals this is the parent tracker file of a sliced download. )r generationNzCannot have a tracker file with slice_start_byte and total_components. slice_start_byte signals a component within a larger operation. total_components signals the parent tracker for a multi-component operation.slice_start_bytery)rrrrLr)r4rrrycomponent_datas r&&write_tracker_file_with_component_datarsd %))*55.!# LL *  *:%&!)9%&/@r%c:[USRX55 g)aWrites rewrite operation information to a tracker file. Args: tracker_file_name (str): The path to the tracker file. rewrite_parameters_hash (str): MD5 hex digest of rewrite call parameters. rewrite_token (str): Returned by API, so rewrites can resume where they left off. z{} {}N)rr2)r]rewrite_parameters_hash rewrite_tokens r&write_rewrite_tracker_filers'oo&=MOr%c[RRU5(dg[R"U5n[ R "U5nU"S0UD6sSSS5 $!,(df  g=f)zHReturns an instance of named_tuple_class with data at tracker_file_path.Nr)rZr[rr r{r|r})named_tuple_classr4r tracker_dicts r&_read_namedtuple_from_json_filersO ) * *  )*l99\*L  ,| ,+**s A$$ A2c"[[U5$)zReads a composite upload tracker file. Args: tracker_file_path (str): The path to the tracker file. Returns: A CompositeUploadTrackerData instance with data at tracker_file_path, or None if no file exists at tracker_file_path. )rr'rs r&"read_composite_upload_tracker_filer ) "3 55r%c"[[U5$)zReads a resumable upload tracker file. Args: tracker_file_path (str): The path to the tracker file. Returns: A ResumableUploadTrackerData instance with data at tracker_file_path, or None if no file exists at tracker_file_path. )rr*rs r&"read_resumable_upload_tracker_filerrr%cUR(d[R"S5eU(aUcUb[R"S5eUb-SR UR U5n[ RnO0UR nUb[ RnO[ Rn[XUS9n[R"SR U55 SnSn [R"U5nU[ RLa1UR5R!S5n XR:XaS n O["R$"UR'55n U S UR:XaPU S UR(:Xa=U[ RLa U S U:XaS n OU[ RLa U S U:XaS n U (aA[R"SR U55 US 4U(aUR+5 $$U(aUR+5 U(a [/U5 [R"SR U55 U[ RLa[1XpRS-5 US4$U[ RLa[3UUUS9 US4$U[ RLa [3UUUS9 US4$![R,a Nf=f!U(aUR+5 ff=f)aChecks for a download tracker file and creates one if it does not exist. Args: source_object_resource (resource_reference.ObjectResource): Needed for object etag and generation. destination_url (storage_url.StorageUrl): Destination URL for tracker file. slice_start_byte (int|None): Start byte to use if we cannot find a matching tracker file for a download slice. component_number (int|None): The download component number to find the start point for. Indicates part of a multi-component download. total_components (int|None): The number of components in a sliced download. Indicates this is the parent tracker for a multi-component operation. Returns: tracker_file_path (str): The path to the tracker file (found or created). found_tracker_file (bool): False if tracker file had to be created. Raises: ValueCannotBeDeterminedError: Source object resource does not have necessary metadata to decide on download start byte. z'Source object resource is missing etag.Nztotal_components indicates this is the parent tracker file for a multi-component operation. slice_start_byte and component_number cannot be present since this is not for an individual component.z{} component {}rzz!Searching for tracker file at {}.F TrrryrzFound tracker file for {}.z No matching tracker file for {}.)r)ry)rrValueCannotBeDeterminedErrorrLr2rgrr r"rrrrrr r{readlinerstripr|loadsreadrrr~rrr) rrmrr`rydownload_name_for_loggerr^r4rdoes_tracker_file_match etag_valuers r&$read_or_create_download_tracker_filers4 $ $  - -1 33+7+7 ,, L  !077%%'7 9'::.<<#)99)22+;KM)) / 6 67H IJ,!##$56LO444((*11$7j 22 2"&zz,"3"3"56n  $:$?$? ?  &*@*K*K K !@!@ @ - .2B B$( ! /"D"D D J #3J4$( ! ii,334LMN  $!/2)) . 5 56N OP/222)+F+F+MN E !!O>>>*)+ E !! O;;;*)+ E !!3     s+DK K$ K!K$ K!!K$$K>c<[RRU5(dg[R"U5nUR 5Vs/sHo3R S5PM snupEXA:Xa UsSSS5 $SSS5 gs snf!,(df  g=f)aGAttempts to read a rewrite tracker file. Args: tracker_file_path (str): The path to the tracker file. rewrite_parameters_hash (str): MD5 hex digest of rewrite call parameters constructed by hash_gcs_rewrite_parameters_for_tracker_file. Returns: String token for resuming rewrites if a matching tracker file exists. Nr)rZr[rr r{ readlinesr)r4rrline existing_hashrs r&#get_rewrite_token_from_tracker_filers ) * *  )*l&2&<&<&>$&>d D&>$ M/  +*0 +  $+* sB B) B B  B)NN)N)NNN)5__doc__ __future__rrr collectionsenumrPr|rZrt"googlecloudsdk.command_lib.storagerrgooglecloudsdk.corerr googlecloudsdk.core.utilr r r r rKrTrvEnumr namedtupler'r*r6r>rHrMrVrdrrrlrrrrrrrrrrrrrrrrr%r&rsF#&'  >5#**,.3 -/*dii)33 o.0 )33 ?A 8& 2&V&*+/BJ 6!H! H*GK@D>%B C6?CA*A0=A<@AD O- 5 5 ;?:>:> d"Nr%