BpSrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK J r SSK J r SSK Jr SSK J r SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKr"SS\R6"\R8\55r"SS\5r"SS\5r "SS\ 5r!"SS\ 5r"g)z&Upload workflow using gRPC API client.)absolute_import)division)unicode_literalsN) retry_util) grpc_util) metadata_util) hash_util)resource_reference) copy_util)log) properties)scaled_integercr\rSrSrSrS SjrSrSrSrSr SS jr S r \ RS 5rS rg)_Upload'z1Base class shared by different upload strategies.NctXlX lX0lX@lX`lX`lSUlSUlXPlg)atInitializes _Upload. Args: client (StorageClient): The GAPIC client. source_stream (io.IOBase): Yields bytes to upload. destination_resource (resource_reference.ObjectResource|UnknownResource): Metadata for the destination object. request_config (gcs_api.GcsRequestConfig): Tracks additional request preferences. source_resource (FileObjectResource|ObjectResource|None): Contains the source StorageUrl and source object metadata for daisy chain transfers. Can be None if source is pure stream. start_offset (int): The offset from the beginning of the object at which the data should be written. FN) _client_source_stream_destination_resource_request_config _start_offset_uploaded_so_far_source_stream_finished _chunk_size_source_resource)selfclient source_streamdestination_resourcerequest_configsource_resource start_offsets 5lib/googlecloudsdk/api_lib/storage/gcs_grpc/upload.py__init___Upload.__init__*s=0L'!5)%(#(D D+cURRbUURRRb4[R"URRR5$g)zReturns MD5 hash bytes sequence from resource args if given. Returns: bytes|None: MD5 hash bytes sequence if MD5 string was given, otherwise None. N)r resource_argsmd5_hashr get_bytes_from_base64_stringrs r#_get_md5_hash_if_given_Upload._get_md5_hash_if_givenNsY **6  . . 7 7 C  3 3    , , 5 577 r&cURUlURRUR[R 5 SUlgNF)rrrseekosSEEK_SETrr+s r#_initialize_generator_Upload._initialize_generator[s:!..DT//=#(D r&c ## Sn[XRRR5(aUnSnOSnUnUR 5 UR R URRRRR5nU(dSnOSnSnU(aSnSnO3URRRUR5S9nSnURRRUUURURRRUS9UUS9v U=R[U5- sl U(aSUlgUR"cGM-[U5URRRRR:HnURUR$- n U(aXR":agGM7f)aYields the WriteObjectRequest for each chunk of the source stream. The amount_of_data_sent_so_far is equal to the number of bytes read from the source stream. If _chunk_size is not None, this function will yield the WriteObjectRequest object until the amount_of_data_sent_so_far is equal to or greater than the value of the new _chunk_size and the length of data sent in the last WriteObjectRequest is equal to MAX_WRITE_CHUNK_BYTES, or if there are no data in the source stream. MAX_WRITE_CHUNK_BYTES is a multiple 256 KiB. Clients must only send data that is a multiple of 256 KiB per message, unless the object is being finished with``finish_write`` set to ``true``. This means that if amount_of_data_sent_so_far >= _chunk_size, it must also be ensured before stopping yielding requests(WriteObjectRequest) that all requests have sent data multiple of 256 KiB, in other words length of data % 256 KiB is 0. The source stream data is read in chunks of MAX_WRITE_CHUNK_BYTES, that means that each request yielded will send data of size MAX_WRITE_CHUNK_BYTES, except if there is a last request before the final request(``finish_write`` set to ``true``) where the data length is less than MAX_WRITE_CHUNK_BYTES, this means if the length of data in the last request yielded is equal to MAX_WRITE_CHUNK_BYTES, all requests sent before have sent data of size MAX_WRITE_CHUNK_BYTES, therefore all requests have sent data that is multiple of 256 KiB, thus satisfying the condition stated before. If the the length of data in the last request yielded is not equal to MAX_WRITE_CHUNK_BYTES, then stop when there are no data in the source stream(the final request is sent). Otherwise if _chunk_size is None, it will yield all WriteObjectRequest objects until there are no data in the source stream. Args: first_message (WriteObjectSpec|str): WriteObjectSpec for Simple uploads, str that is the upload id for Resumable and Streaming uploads. Yields: (googlecloudsdk.generated_clients.gapic_clients.storage_v2.types.WriteObjectRequest) WriteObjectRequest instance. FNT)r))content)write_object_spec upload_id write_offsetchecksummed_dataobject_checksums finish_write) isinstancertypesWriteObjectSpecr3rreadServiceConstantsValuesMAX_WRITE_CHUNK_BYTESObjectChecksumsr,WriteObjectRequestrChecksummedDatalenrrr) r first_messagefirst_request_doner7r8datar;r<0is_length_of_data_equal_to_max_write_chunk_bytesamount_of_data_sent_so_fars r#&_upload_write_object_request_generator._Upload._upload_write_object_request_generatorcsZ-!3!3!C!CDD'ii   % % ,,   - - 4 4 J Jd !    <<--==002>  LL   1 1-,,<<--==d=K+# 2  s4y( '+$   ! d) \\   0 0 7 7 M M N7$(#8#84;M;M#M :(,<,<< c sG*G,c[UR[R5(dgURR(dg[ R "URR5Ulg)aCopies metadata from _source_resource to object_metadata. It is copied if _source_resource is an instance of ObjectResource, this is in case a daisy chain copy is performed. Args: object_metadata (gapic_clients.storage_v2.types.storage.Object): Existing object metadata. N)r=rr ObjectResource custom_fieldscopydeepcopymetadata)robject_metadatas r#*_set_metadata_if_source_is_object_resource2_Upload._set_metadata_if_source_is_object_resourcesS d++-?-N-N O O  . . #}} ++ -Or&cURRRURRR [ R"URRR5US9nURU5 [R"X RUR5 URRRU[R "UR5URR"US9$)zReturns the WriteObjectSpec instance. Args: size (int|None): Expected object size in bytes. Returns: (gapic_clients.storage_v2.types.storage.WriteObjectSpec) The WriteObjectSpec instance. )namebucketsize)resourceif_generation_matchif_metageneration_match object_size)rr>Objectr storage_url resource_namerget_full_bucket_name bucket_namerVr*update_object_metadata_from_request_configrrr?r get_generation_match_value!precondition_metageneration_match)rr[destination_objects r#_get_write_object_spec_Upload._get_write_object_specs++22  ' ' 3 3 A A--  & & 2 2 > >@  3  33<<00$2G2G <<   - -#%@@     B B . r&cURRRURUS9[R "[ R"URRR55S9$)zCalls write object api method with routing header. Args: first_message (WriteObjectSpec|str): WriteObjectSpec for Simple uploads. Returns: (gapic_clients.storage_v2.types.WriteObjectResponse) Request response. )rH)requestsrT) rstorage write_objectrMrget_bucket_name_routing_headerrrcrrard)rrHs r#_call_write_object_Upload._call_write_objectso <<   , ,<<'= ==  * ***66BB  -  r&c[e)z=Performs an upload and returns and returns an Object message.)NotImplementedErrorr+s r#run _Upload.runs  r&) rrrrrrrrr)NrN)__name__ __module__ __qualname____firstlineno____doc__r$r,r3rMrVrirpabcabstractmethodrt__static_attributes__r&r#rr'sN9",H )iV-*!F&r&rc<\rSrSrSr\R S5rSrg) SimpleUploadiz(Uploads an object with a single request.cURURRR5nUR U5$)zUploads the object in non-resumable mode. Returns: (gapic_clients.storage_v2.types.WriteObjectResponse) A WriteObjectResponse instance. )rirr(r[rp)rr7s r#rtSimpleUpload.run"sB33 **//  " " r&rN) rwrxryrzr{rgrpc_default_retryerrtr~rr&r#rrs0 "" # r&rc>\rSrSrSrSrSrS SjrSrSr S r g) RecoverableUploadi2z7Common logic for strategies allowing retries in-flight.cUR5nURRRUS9nURRR US9R nSUlU$)zSets up the upload session and returns the upload id. This method sets the start offset to 0. Returns: (str) Session URI for resumable upload operation. )r7requestr)rirr>StartResumableWriteRequestrmstart_resumable_writer8r)rr7rr8s r#_initialize_upload$RecoverableUpload._initialize_upload5so335ll  ;;+<G $$::;"D r&cURRRUS9nURRR US9R $)a Returns the amount of data persisted on the server. Args: upload_id (str): Session URI for resumable upload operation. Returns: (int) The total number of bytes that have been persisted for an object on the server. This value can be used as the write_offset. )r8r)rr>QueryWriteStatusRequestrmquery_write_statuspersisted_size)rr8rs r#_get_write_offset#RecoverableUpload._get_write_offsetJsRll  889G <<   2 2 3 nr&Nc[R"X#XE5(dgURU5nX`R:nU(aX`lg)NFT)r is_retriablerr)rr8exc_type exc_value exc_tracebackstater"is_progress_made_since_last_uplaods r# _should_retryRecoverableUpload._should_retry[sC  " "8 M M ++I6N)7:L:L)L&)) r&c$URU5$rv)rp)rr8s r#_perform_upload!RecoverableUpload._perform_uploadgs  " "9 --r&cUR5n[R"URU5n[R "UR UU/S9$)N)targetshould_retry_if target_args)r functoolspartialrstorage_retry_utilretryerr)rr8new_should_retrys r#rtRecoverableUpload.runjsM'')I ((););YG  % %##(K r&)r)NNNN) rwrxryrzr{rrrrrtr~rr&r#rr2s'?*"?C.2 .r&rcB^\rSrSrSrSU4SjjrU4SjrSrU=r$)ResumableUploadiuzDUploads objects with support for resuming between runs of a command.cL>[[U] UUUUU5 XPlXplgrv)superrr$_serialization_data_tracker_callback) rrrrr serialization_datar!tracker_callback __class__s r#r$ResumableUpload.__init__xs1 /4)  2-r&c>URb>URSnURU5nX l[R"SU5 O[ [ U]5nURbURSU05 U$)zSets up the upload session and returns the upload id. Additionally, it does the following tasks: 1. Grabs the persisted size on the backend. 2. Sets the appropiate write offset. 3. Calls the tracker callback. Returns: The upload session ID. r8zWrite offset after resuming: %s) rrrr debugrrrr)rr8r9rs r#r"ResumableUpload._initialize_uploadsz +**;7i++I6l' ii1<@ACi ) k956 r&)rrr)NNN) rwrxryrzr{r$rr~ __classcell__rs@r#rrus L.(r&rcD^\rSrSrSrSU4SjjrSrSrSrSr U=r $) StreamingUploadiaUploads objects from a stream with support for error recovery in-flight. Stream is split into chunks of size set by property upload_chunk_size, rounded down to be a multiple of MAX_WRITE_CHUNK_BYTES. For example if upload_chunk_size is 7 MiB and MAX_WRITE_CHUNK_BYTES is 2 MiB, it will be rounded down to 6 MiB. If upload_chunk_size is less than MAX_WRITE_CHUNK_BYTES, it will be equal to MAX_WRITE_CHUNK_BYTES. cl>[[U] UUUUU5 SUlUR 5Ulgr/)rrr$_log_chunk_warning_get_chunk_sizer)rrrrr r!rs r#r$StreamingUpload.__init__s> /4) $D++-Dr&c[R"[RRR R 55nUURRRRR:aHUURRRRR-nUS:aSUl X- $SUl URRRRR$)aReturns the chunk size corrected to be multiple of MAX_WRITE_CHUNK_BYTES. It also sets the attribute _should_log_message if it is needed to log the warning message. Look at the docstring on StreamingUpload class. Returns: (int) The chunksize value corrected. rT) r ParseIntegerr VALUESrmupload_chunk_sizeGetrr>rArBrCr)rinitial_chunk_sizeadjust_chunk_sizes r#rStreamingUpload._get_chunk_sizes(44!!33779;  <<   . . 5 5 K K L  LL   / / 6 6 L L M Q "&  33"D <<   . . 5 5 K KKr&cdUR(dg[R"[RR R R55n[R"SR[R"UR5[R"U555 g)NzmData will be sent in chunks of {} instead of {}, as configured in the storage/upload_chunk_size config value.) rrrr rrmrrr warningformatFormatBinaryNumberr)rrs r# _log_messageStreamingUpload._log_messages~  " " '44!!33779;KK L &2243C3CD223EF H r&cUR5 SnURU5nUR(aU$URUlM8rv)rrprrr)rr8responses r#rStreamingUpload._perform_uploadsLH ((3h % % O$22d r&)rrrrv) rwrxryrzr{r$rrrr~rrs@r#rrs'  .$L8 r&r)#r{ __future__rrrr|rRrr1googlecloudsdk.api_lib.storagerr'googlecloudsdk.api_lib.storage.gcs_grpcrr"googlecloudsdk.command_lib.storager ,googlecloudsdk.command_lib.storage.resourcesr +googlecloudsdk.command_lib.storage.tasks.cpr googlecloudsdk.corer r googlecloudsdk.core.utilrsixwith_metaclassABCMetaobjectrrrrrrr&r#rs-&'  K=A>8KA#*3 uc  f5up7&@@F/'/dP'Pr&