`SrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK Jr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SrSrSrSr"SS\5rSr Sr!Sr"S&Sjr#Sr$S'Sjr%Sr&Sr'S(Sjr(Sr)Sr*S'Sjr+S'S jr,S!r-S"r.S#r/S'S$jr0S%r1g))z0Helper functions for tracker file functionality.)absolute_import)print_function)division)unicode_literalsN)config)CommandException)GetGsutilStateDir)ResumableThreshold)UTF8)GetMd5)CreateDirIfNeededdzCouldn't write tracker file (%s): %s. This can happen if gsutil is configured to save tracker files to an unwritable directory)encryption_key_sha256serialization_datac,\rSrSrSrSrSrSrSrSr Sr g ) TrackerFileType4uploaddownloaddownload_componentparallel_uploadsliced_downloadrewriteN) __name__ __module__ __qualname____firstlineno__UPLOADDOWNLOADDOWNLOAD_COMPONENTPARALLEL_UPLOADSLICED_DOWNLOADREWRITE__static_attributes__r%platform/gsutil/gslib/tracker_file.pyrr4s! & (+%/%/ 'r&rc[U[R5(aUR[5nUnOUnUR [5n[ R"U5nSUR5-S-USS-$)a Apply a hash function (SHA1) to shorten the passed file name. The spec for the hashed file name is as follows: TRACKER__ where hash is a SHA1 hash on the original file name and trailing is the last 16 chars from the original file name. Max file name lengths vary by operating system so the goal of this function is to ensure the hashed version takes fewer than 100 characters. Args: filename: file name to be hashed. May be unicode or bytes. Returns: shorter, hashed version of passed file name TRACKER_.iN) isinstancesix text_typeencoder decodehashlibsha1 hexdigest)filenamefilename_bytes filename_strms r' _HashFilenamer7=sk$#--((__T*NLN??4(L ll>"! akkm #c )L,> >>r&c[R"SS[RR [ 5S55n[ U5 U$)zLooks up or creates the gsutil tracker file directory. This is the configured directory where gsutil keeps its resumable transfer tracker files. This function creates it if it doesn't already exist. Returns: The pathname to the tracker directory. GSUtilresumable_tracker_dirz tracker-files)rgetospathjoinr r ) tracker_dirs r'CreateTrackerDirIfNeededr@Ys< 8%<77<<(9(;_MO+K r&c[R"SSSU<SU<SU<SU<SU<S3 5n[U[R5$)aCGets the tracker file name described by the arguments. Args: src_bucket_name: Source bucket (string). src_obj_name: Source object (string). dst_bucket_name: Destination bucket (string). dst_obj_name: Destination object (string) api_selector: API to use for this operation. Returns: File path to tracker file. [/\\]_ rewrite____z.token)resub_HashAndReturnPathrr$)src_bucket_name src_obj_namedst_bucket_name dst_obj_name api_selectorres_tracker_file_names r'GetRewriteTrackerFilePathrOhs? 66 o| 1?3J3J KKr&c U[R:Xa;[R"SSSUR<SUR <SU<S35nGOU[R :XaJ[R"SSS[RRUR 5<SU<S35nGO&U[R:XaE[R"SSS[RRUR 5X$4-5nOU[R:Xa>[R"SSS UR<SUR <SU<SU<S3 5nO{U[R:XaI[R"SSS [RRUR 5<SU<S35nOU[R:Xa [5e[WU5$) aGets the tracker file name described by the arguments. Args: dst_url: Destination URL for tracker file. tracker_file_type: TrackerFileType for this operation. api_selector: API to use for this operation. src_url: Source URL for the source file name for parallel uploads. component_num: Component number if this is a download component, else None. Returns: File path to tracker file. rBrCresumable_upload__rEz.urlresumable_download__z.etagz#resumable_download__%s__%s__%d.etagparallel_upload__sliced_download__)rrrFrG bucket_name object_namer r<r=realpathr!r"r#r$NotImplementedErrorrH)dst_urltracker_file_typerMsrc_url component_numrNs r'GetTrackerFilePathr]s}"/000VV3   g11<ABO444VV3   '-- . >?O>>> VV3=   '-- . LMNO;;; VV3   g117LJKO;;;VV3   '-- . >?O333   13D EEr&c[[U[RU55 [ X5nUHn[U5 M g)zDeletes all tracker files corresponding to an object download. Args: dst_url: StorageUrl describing the destination file. api_selector: The Cloud API implementation used. N)DeleteTrackerFiler]rr !GetSlicedDownloadTrackerFilePaths)rYrM tracker_files tracker_files r'DeleteDownloadTrackerFilesrcs=/":":LIK4GJ-#ll#$r&c [U[RU5nU/nUc@Sn[US5n[R "U5SnU(aUR5 [U5H,nUR[U[RUUS95 M. U$![ [4a UsU(aUR5 $$f=f!U(aUR5 ff=f)aGets a list of sliced download tracker file paths. The list consists of the parent tracker file path in index 0, and then any existing component tracker files in [1:]. Args: dst_url: Destination URL for tracker file. api_selector: API to use for this operation. num_components: The number of component tracker files, if already known. If not known, the number will be retrieved from the parent tracker file on disk. Returns: File path to tracker file. Nrnum_componentsr\) r]rr#openjsonloadIOError ValueErrorcloserangeappendr!)rYrMrfparallel_tracker_file_pathtracker_file_pathsrbis r'r`r`s" 2 ..  >23L4c:lyy./?@n   a7*==')* ,- !  Z       s#%BC 1C C  CC)c[5n[U5n[U5R5<SU<3nU<[R <U<3n[ U5[:deU$)zHashes and returns a tracker file path. Args: res_tracker_file_name: The tracker file name prior to it being hashed. tracker_file_type: The TrackerFileType of res_tracker_file_name. Returns: Final (hashed) tracker file path. rC)r@r7strlowerr<seplenMAX_TRACKER_FILE_NAME_LENGTH)rNrZr:hashed_tracker_file_nametracker_file_nametracker_file_paths r'rHrHse34*+@A!$%6!7!=!=!?!9;"7"35  "> >> > r&cU(a<[RRU5(a[R"U5 gggN)r<r=existsunlink)rzs r'r_r_s-277>>*;<<II =r&c U(acUR(aRUR(aAUR(a0U(a)UR(aUR(aU(dg[5n UUUUUUUU UUU 4 H<n U cMU R [ R "U 5RS55 M> U R5$)a,Creates an MD5 hex digest of the parameters for a rewrite call. Resuming rewrites requires that the input parameters are identical. Thus, the rewrite tracker file needs to represent the input parameters. For easy comparison, hash the input values. If a user does a performs a same-source/same-destination rewrite via a different command (for example, with a changed ACL), the hashes will not match and we will restart the rewrite from the beginning. Args: src_obj_metadata: apitools Object describing source object. Must include bucket, name, and etag. dst_obj_metadata: apitools Object describing destination object. Must include bucket and object name projection: Projection used for the API call. src_generation: Optional source generation. gen_match: Optional generation precondition. meta_gen_match: Optional metageneration precondition. canned_acl: Optional canned ACL string. max_bytes_per_call: Optional maximum bytes rewritten per call. src_dec_key_sha256: Optional SHA256 hash string of decryption key for source object. dst_enc_key_sha256: Optional SHA256 hash string of encryption key for destination object. fields: Optional fields to include in response to call. Returns: MD5 hex digest Hash of the input parameters, or None if required parameters are missing. Nr ) bucketnameetagr updater,r-r.r2) src_obj_metadatadst_obj_metadata projectionsrc_generation gen_matchmeta_gen_match canned_aclmax_bytes_per_callsrc_dec_key_sha256dst_enc_key_sha256fieldsmd5_hash input_params r'HashRewriteParametersrsR "2"9"9   '7'<'< "2"9"9   z X(  koocmmK077?@! "    r&c"SnU(dg[US5nUR5RS5nX1:Xa7UR5RS5U(aUR5 $$U(aUR5 gg![aZnUR [ R :wa2[RRSU<SUR<S35 SnANxSnAff=f!U(aUR5 ff=f)anAttempts to read a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. Returns: String rewrite_token for resuming rewrite requests if a matching tracker file exists, None otherwise (which will result in starting a new rewrite). Nre z!Couldn't read Copy tracker file (): z. Restarting copy from scratch.) rhreadlinerstriprmrkerrnoENOENTsysstderrwritestrerror)rzrewrite_params_hashrb existing_hashes r'ReadRewriteTrackerFilerKs,  )3/L ))+2248M+  " " $ + +D 1, ?ww%,, jj/ =??s1AB 3C4 C1AC,'C4,C11C44Dc,[UU<SU<S35 g)aWrites a rewrite tracker file. Args: tracker_file_name: Tracker file path string. rewrite_params_hash: MD5 hex digest of rewrite call parameters constructed by HashRewriteParameters. rewrite_token: Rewrite token string returned by the service. rN)_WriteTrackerFile)rzr rewrite_tokens r'WriteRewriteTrackerFilerls%"5}EGr&c UR(deSnUR[5:aXt4$URnUc[R n O[R n USU-- n[UU UUS9nSn [US5n U [R LaJU R5RS5n XR:XaXu4U (aU R5 $$O{U [R Lah[R"U R55n U SUR:Xa1U SUR:XaX|S4U (aU R5 $$UR!S U-5 U (aU R5 U [R La[/US UR-5 Xt4$U [R La [1XpU5 Xt4$!["[$4a`n ['U [$5(dU R([(R*:wa#UR!S U<S [-U 5<S 35 Sn A NSn A ff=f!U (aU R5 ff=f)aChecks for a download tracker file and creates one if it does not exist. The methodology for determining the download start point differs between normal and sliced downloads. For normal downloads, the existing bytes in the file are presumed to be correct and have been previously downloaded from the server (if a tracker file exists). In this case, the existing file size is used to determine the download start point. For sliced downloads, the number of bytes previously retrieved from the server cannot be determined from the existing file size, and so the number of bytes known to have been previously downloaded is retrieved from the tracker file. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. logger: For outputting log messages. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: tracker_file_name: The name of the tracker file, if one was used. download_start_byte: The first byte that still needs to be downloaded. Nz component %drgrerr generationdownload_start_bytezPTracker file doesn't match for download of %s. Restarting download from scratch.z%Couldn't read download tracker file (rz#. Restarting download from scratch.z%s )rsizer rVrr r!r]rhrrrmriloadsreadrwarnrkrlr+rrrtr!WriteDownloadComponentTrackerFile)rrYloggerrM start_byteexisting_file_sizer\rz download_namerZrb etag_valuecomponent_datars r'ReadOrCreateDownloadTrackerFilerzsRB    /11  ((%%-'00'::_}44M():)57DF,)3/LO444((*11$7j ,, , 4"% - o@@ @zz,"3"3"56n  $4$9$9 9  &*:*E*E E 1F"GG KK)+89:/222'2B2G2G)GH  &&O>>>%&7&02  &&# : J!Z  AGGu||$; kk.?QIJ Js98AG A#GGI(AI>I II I%c"UR(deSnUR[5:aU$Uc[RnO[R n[ UUUUS9nSn[US5nU[RLaIUR5RS5n XR:XaUU(aUR5 $$OzU[R Lag[R"UR55n U SUR:Xa0U SUR:XaU SU(aUR5 $$U(aUR5 U$![[ 4a N+f=f!U(aUR5 ff=f)aFReturns the download starting point. The methodology of this function is the same as in ReadOrCreateDownloadTrackerFile, with the difference that we are not interested here in possibly creating a tracker file. In case there is no tracker file, this means the download starting point is start_byte. Args: src_obj_metadata: Metadata for the source object. Must include etag and generation. dst_url: Destination URL for tracker file. api_selector: API to use for this operation. start_byte: The start byte of the byte range for this download. existing_file_size: Size of existing file for this download on disk. component_num: The component number, if this is a component of a parallel download, else None. Returns: download_start_byte: The first byte that still needs to be downloaded. Nrgrerrrr)rrr rr r!r]rhrrrmrirrrrkrl) rrYrMrrr\rzrZrbrrs r'GetDownloadStartByters4    /11 '00'::():)57DF,)3/LO444((*11$7j ,, ,! - o@@ @zz,"3"3"56n  $4$9$9 9  &*:*E*E E34  :   s2#AE E4 A!EE1.E40E11E44FcxURURUS.n[U[R"U55 g)zUpdates or creates a download component tracker file on disk. Args: tracker_file_name: The name of the tracker file. src_obj_metadata: Metadata for the source object. Must include etag. current_file_pos: The current position in the file. )rrrN)rrrridumps)rzrcurrent_file_posrs r'rrs6##$//-. %tzz.'ABr&ct[R"U[R[R-[R-S5n[R "US5nUR U5 SSS5 g!,(df  g=f![[4an[XR5eSnAff=f)z/Creates a tracker file, storing the input data.iwNF) r<rhO_WRONLYO_CREATO_TRUNCfdopenrrkOSError#RaiseUnwritableTrackerFileExceptionr)rzdatafdtfrs r'rr+sM "BKK"**$rs7&%'  ,34&-5 #C' #:%9"f?8 L6 $%) 0Ff$&6:(V(!*.$()-%)-1-1-1!%@FB G(37 V'|(, CLC$ M 18+v9r&