b4SrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK r SSK J r SSK Jr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJr \ R>(a\ r!"SS\"5r#g)apBoto translation layer for resumable uploads. See https://cloud.google.com/storage/docs/resumable-uploads-xml for details. Resumable uploads will retry interrupted uploads, resuming at the byte count completed by the last upload attempt. If too many retries happen with no progress (per configurable num_retries param), the upload will be aborted in the current process. Unlike the boto implementation of resumable upload handler, this class does not directly interact with tracker files. Originally Google wrote and contributed this code to the boto project, then copied that code back into gsutil on the release of gsutil 4.0 which supports both boto and non-boto codepaths for resumable uploads. Any bug fixes made to this file should also be integrated to resumable_upload_handler.py in boto, where applicable. TODO: gsutil-beta: Add a similar comment to the boto code. )absolute_import)print_function)division)unicode_literalsN)urllib) http_client)config) UserAgent)AWSAuthConnection)ResumableTransferDisposition)ResumableUploadException)InvalidUrlError)GetMaxRetryDelay) GetNumRetries)XML_PROGRESS_CALLBACKS)UTF8c\rSrSrSrSr\R\\ R\ R4r Sr SSjrSrSSjrS rS rSS jrS rS rSrSSjrSS\4SjrSrg)BotoResumableUploadJz3Upload helper class for resumable uploads via boto.i )rNcU(aURU5 OSUlX@lSUlSUlXlX lg)a=Constructor. Instantiate once for each uploaded file. Args: tracker_callback: Callback function that takes a string argument. Used by caller to track this upload across upload interruption. logger: logging.logger instance to use for debug messages. resume_url: If present, attempt to resume the upload at this URL. num_retries: Number of times to retry the upload making no progress. This count resets every time we make progress, so the upload can span many more than this number of retries. Nr) _SetUploadUrl upload_url num_retriesservice_has_bytesupload_start_pointtracker_callbacklogger)selfrr resume_urlrs .platform/gsutil/gslib/boto_resumable_upload.py__init__BotoResumableUpload.__init__Us?" $do"D#D,Kcz[RRU5nURR 5S;dUR (d[ SU-5eXl[R"SSS5=(d UR Ul UR<SUR<3Ul SUlg)zSaves URL and resets upload state. Called when we start a new resumable upload or get a new tracker URL for the upload. Args: url: URL string for the upload. Raises InvalidUrlError if URL is syntactically invalid. )httphttpszInvalid upload URL (%s) Credentialsgs_hostN?r)rparseurlparseschemelowernetlocrrr getupload_url_hostpathqueryupload_url_pathr)rurl parse_results r!r!BotoResumableUpload._SetUploadUrlrs<<((-L!!#+<<    5; <<O"JJ}iF0(// &2&7&79K9KLDDr$cSU<SU<3$)Nzbytes /)r range_spec length_specs r!_BuildContentRangeHeader,BotoResumableUpload._BuildContentRangeHeaders & 44r$c URSU5SS.n[R"USURURUURS9$)aQueries service to find out state of given upload. Note that this method really just makes special case use of the fact that the upload service always returns the current start/end state whenever a PUT doesn't complete. Args: conn: HTTPConnection to use for the query. file_length: Total length of the file. Returns: HTTP response from sending request. Raises: ResumableUploadException if problem querying service. *0) Content-RangeContent-LengthPUTr2 auth_pathheadershost)r=r make_requestr4r1)rconn file_length put_headerss r!_QueryServiceState&BotoResumableUpload._QueryServiceStatesX(77[IK  ) )$*//3/C/C484H4H2=/3/C/C  EEr$cURX5nURS:XaSUS- 4$URS:wa'[SUR-[R5eSnUR S5nU(aU[ R"SU5nU(a6[URS55n[URS 55nS nO UR$U(d4[S [UR55-[R5eURS:aURRS WW5 WW4$) alQueries service to find out what bytes it currently has. Args: conn: HTTPConnection to use for the query. file_length: Total length of the file. Returns: (service_start, service_end), where the values are inclusive. For example, (0, 2) would mean that the service has bytes 0, 1, *and* 2. Raises: ResumableUploadException if problem querying service. ri4z2Got non-308 response (%s) from service state queryFrangezbytes=(\d+)-(\d+)Tz7Couldn't parse upload service state query response (%s)zService has: Range: %d - %d.)rMstatusr r START_OVER getheaderresearchlonggroupSERVICE_HAS_NOTHINGstr getheadersdebugr) rrJrKrespgot_valid_responser;m service_start service_ends r!_QueryServicePos$BotoResumableUpload._QueryServicePoss+  " "4 5D {{cq !! {{c % > L & 1 1 33(J ))(* 5a QWWQZ( 1771:& ! % %%  $ D doo  !">"I"I KK zzQ kk6 #% ; ''r$cpURRnURS:aURRS5 SUl0nUH7nUR 5S:Xa[ S[R5eX%XE'M9 SXCRR'URSURRURU5nUR5nURS;a'[ S UR-[R5eURS :wa7URS :wa'[ S UR-[R5eUR!S 5nU(d[ SU-[R5eUR#U5 UR%U5 g)zStarts a new resumable upload. Args: key: Boto Key representing the object to upload. headers: Headers to use in the upload requests. Raises: ResumableUploadException if any errors occur. rQzStarting new resumable upload.rzcontent-lengthz5Attempt to specify Content-Length header (disallowed)startPOST)zEGot status %d from attempt to start resumable upload. Will wait/retryrPz>Got status %d from attempt to start resumable upload. AbortingLocationzHNo resumable upload URL found in resumable initiation POST response (%s)N)bucket connectionr^rrr.r r ABORTproviderresumable_upload_headerrInamereadrTWAIT_BEFORE_RETRYrVrr) rkeyrGrJ post_headerskr_bodyrs r!_StartNewResumableUpload,BotoResumableUpload._StartNewResumableUploads :: D zzQ kk89DL  & && C ( . .0 0 lo  ;BL667   VSZZ__chh MD 99;D {{o% $ "kk * & 8 8 ::   s 2 $ {{ #$@$F$F HH  +J  $ !% & & 8 8 :: z"*%r$c >URUR5n U(a0US:aX@R- US- - n O US:aSn OSn Sn U"XT5 U(aUR5O0n U(a5XT:XaURSU5n OURSXTS- 4-U5n XS'[ XE- 5U S'[ R "US URS U URS 9nURS UR5 U HnURXU5 M UR5 URS5 U (a[R(a UR!U 5 U[#U 5- nOi[%U [&5(a UR!U 5 U[#U 5- nO4U R)[*5nUR!U5 U[#U5- nU(aW S- n U W :XdU S:Xa U"XT5 Sn URUR5n U (aMURUR,5 U(aU"XT5 XT:wa[/S XT4-[0R25eUR55nUR6S :Xa2UR9S5UR9S5UR9S54$UR6S;a[0R:nO[0R2n[/SUR6UR<4-U5e)a Attempts to upload file bytes. Makes a single attempt using an existing resumable upload connection. Args: conn: HTTPConnection from the boto Key. http_conn: Separate HTTPConnection for the transfer. fp: File pointer containing bytes to upload. file_length: Total length of the file. total_bytes_uploaded: The total number of bytes uploaded. cb: Progress callback function that takes (progress, total_size). num_cb: Granularity of the callback (maximum number of times the callback will be called during the file transfer). If negative, perform callback with each buffer read. headers: Headers to be used in the upload requests. Returns: (etag, generation, metageneration) from service upon success. Raises: ResumableUploadException if any problems occur. rSrrr@z%d-%drQrBrCrDNrEz 6<< < !  123)) = 6AA A ! M)) % r$cURU:aSUlOU=RS- slURUR:a[S[R 5e[ [R"5SUR--[55nUS:a'URRSURU5 [R"U5 g)aTracks the number of iterations without progress. Performs randomized exponential backoff. Args: service_had_bytes_before_attempt: Number of bytes the service had prior to this upload attempt. debug: debug level 0..3 rrQzaToo many resumable upload attempts failed without progress. You might try this upload again laterrSzZGot retryable failure (%d progress-less in a row). Sleeping %3.1f seconds before re-tryingN) rprogress_less_iterationsrr r rminrandomrrr^timesleep)r service_had_bytes_before_attemptr^sleep_time_secss r!TrackProgressLessIterations/BotoResumableUpload.TrackProgressLessIterationss  @@&'d# ##q(# $$t'7'77 $ < & 8 8 :: &--/Q0M0M-MN*,.O z kk 4  ' ': JJr$cU(d0nSnX;aXHcXH U(aXTURR'[US'Un URRR n UR c[5UlSUlURn URXXXg5uol Ul URUl U S:aURR S5 g!URan U S:a*URR SU R55 [!U ["5(aLU R$[$R&:Xa.URRRR)5 Sn A O+Sn A f[*an UR-X5 Sn A OSn A ff=fUR/U U S9 GMD) a>Upload a file to a key into a bucket on GS, resumable upload protocol. Args: key: `boto.s3.key.Key` or subclass representing the upload destination. fp: File pointer to upload size: Size of the file to upload. headers: The headers to pass along with the PUT request canned_acl: Optional canned ACL to apply to object. cb: Callback function that will be called to report progress on the upload. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted to GS, and the second representing the total number of bytes that need to be transmitted. num_cb: (optional) If a callback is specified with the cb parameter, this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. Providing a negative integer will cause your callback to be called with each buffer read. Raises: ResumableUploadException if a problem occurs during the transfer. z Content-TypeNz User-AgentrrQzResumable upload complete.zCaught exception (%s))r^)rq acl_headerr rnror^rrrrr generationmetagenerationrRETRYABLE_EXCEPTIONS__repr__rIOErrorerrnoEPIPErr rr)rrvrsizerG canned_aclrr content_typerKr^r_rs r!SendFileBotoResumableUpload.SendFiles> g"L7#8#@  )3cll%%&%GLK JJ ! ! ' 'E &d$%D! )-)?)?&6372N2N [237/?D/ A: ++  8 9  & & 3 A: ++  3QZZ\ B a ! !agg&< **   * * 0 0 2 %6 ++A556 &&'G-2'45 s&AC&&F46BF  F4F//F4) rrrrrrrrrr1r4)NN)r@r@)N)r)__name__ __module__ __qualname____firstlineno____doc__r~r HTTPExceptionrrrgaierrorrr[r"rr=rMrdrzrrrrrr__static_attributes__r:r$r!rrJs;+%33Wfll //+    :*5E<8(t:&xuAnM^%,)*! P,Q4r$r)$r __future__rrrrrrrWrrr six.movesrrbotor r boto.connectionr boto.exceptionr r gslib.exceptionrgslib.utils.boto_utilrrgslib.utils.constantsrrPY3intrYobjectrr:r$r!rsh,,'%' !-73+2/8&77 $b4&b4r$