"TSrSSKJr SSKJr SSKJr SSKrSSKJr SSKJ r SSK J r SS K J r SS K Jr SS K Jr SSKrS r"S S\R$"\R&55r"SS\5r"SS\5r"SS\5r"SS\5r\S4Sjr\S4Sjr\S4Sjrg)aUtilities for parallelizing Cloud Storage file operations. Example usage (for simplicity, use absolute *nix-style paths; in practice you'll want to use os.path.join and friends): >>> upload_tasks = [ ... FileUploadTask('/tmp/file1.txt', 'gs://my-bucket', ... 'path/to/remote1.txt'), ... FileUploadTask('/tmp/file2.txt', 'gs://my-bucket', '/remote2.txt') ... ] >>> UploadFiles(upload_tasks, num_threads=16) This will block until all files are uploaded, using 16 threads (but just the current process). Afterwards, there will be objects at 'gs://my-bucket/path/to/remote1.txt' and 'gs://my-bucket/remote2.txt'. >>> delete_tasks = [ ... ObjectDeleteTask('gs://my-bucket', 'path/to/remote1.txt'), ... ObjectDeleteTask('gs://my-bucket', '/remote2.txt') ... ] >>> DeleteObjects(delete_tasks, num_threads=16) This removes the objects uploaded in the last code snippet. )absolute_import)division)unicode_literalsN) storage_api)log) console_io)parallel)retry)textc@\rSrSrSr\R SSj5rSrg)Task<z8Base clase for a storage tasks that can be parallelized.NcgN)selfcallbacks :lib/googlecloudsdk/command_lib/storage/storage_parallel.pyExecute Task.Execute?srr) __name__ __module__ __qualname____firstlineno____doc__abcabstractmethodr__static_attributes__rrrrr<s@  rrc:\rSrSrSrSrSrSrSrS Sjr S r g) FileUploadTaskDaSelf-contained representation of a file to upload and its destination. Attributes: source_local_path: str, The local filesystem path of the source file to upload. dest_obj_ref: storage_util.ObjectReference, The object the file will be copied to. cXlX lgr)source_local_path dest_obj_ref)rr%r&s r__init__FileUploadTask.__init__Ns.$rcjSRURURR55$)NzUpload: {} --> {}formatr%r&ToUrlrs r__str__FileUploadTask.__str__Rs/  % %  1 1 7 7 9 ;;rcfSRURURR5S9$)Nz@FileUploadTask(source_path={source_path}, dest_path={dest_path}) source_path dest_pathr*r-s r__repr__FileUploadTask.__repr__Vs3J D22++113  56rcD[URUR45$r)hashr%r&r-s r__hash__FileUploadTask.__hash__\s ''):):; < {}r+rMr,rNr-s rr.FileDownloadTask.__str__us/ ' ' !!#T%9%9 ;;rcfSRURR5URS9$)NzBFileDownloadTask(source_path={source_path}, dest_path={dest_path})r1rQr-s rr4FileDownloadTask.__repr__ys3L D//557..  01rcD[URUR45$r)r7rMrNr-s rr8FileDownloadTask.__hash__s $$d&:&:; < {}r+rMr,r&r-s rr.FileRemoteCopyTask.__str__s:  # # !!#T%6%6%<%<%> @@rcSRURR5URR5S9$)NzDFileRemoteCopyTask(source_path={source_path}, dest_path={dest_path})r1r`r-s rr4FileRemoteCopyTask.__repr__s<N D//557++113  56rcD[URUR45$r)r7rMr&r-s rr8FileRemoteCopyTask.__hash__s $$d&7&78 99rNc[R"5n[R"SS9R UR UR UR4S9 U(aU"5 ggr;)rrAr rBrCCopyrMr&rEs rrFileRemoteCopyTask.ExecutesX ..0N MMq!22!!4#4#4 537jr)r&rMrrHrrrr[r[s!%@6 :rr[c:\rSrSrSrSrSrSrSrS Sjr S r g) ObjectDeleteTaskzSelf-contained representation of an object to delete. Attributes: obj_ref: storage_util.ObjectReference, The object to delete. cXlgrobj_ref)rrns rr'ObjectDeleteTask.__init__sLrcTSRURR55$)Nz Delete: {}r+rnr,r-s rr.ObjectDeleteTask.__str__s   t||113 44rcPSRURR5S9$)NzObjectDeleteTask(object={obj})objrqr-s rr4ObjectDeleteTask.__repr__s# * 1 1dll6H6H6J 1 KKrc,[UR5$r)r7rnr-s rr8ObjectDeleteTask.__hash__s  rNc[R"5n[R"SS9R UR UR 4S9 U(aU"5 gg)z8Complete one ObjectDeleteTask (safe to run in parallel).r<r=r?N)rrAr rBrC DeleteObjectrnrEs rrObjectDeleteTask.ExecutesJ ..0N MMq!22##4<URT5$r)r)taskrs rExecuteTasks..sDLL2r) rdebugr GetPoolrTickableProgressBarlenTickNoOpProgressBarrMap)tasks num_threadsprogress_bar_labelpool progress_bartrs @r ExecuteTasksrs)) )) +.   + &$11 E &(L  H--/LHA ! (  t hh2E: ts0C)>D C:D ) C7: D D  DFc[U5nU(a(SRU[R"US55nOSn[ XU5 g)a\Upload the given files to the given Cloud Storage URLs. Uses the appropriate parallelism (multi-process, multi-thread, both, or synchronous). Args: files_to_upload: list of FileUploadTask num_threads: int (optional), the number of threads to use. show_progress_bar: bool. If true, show a progress bar to the users when uploading files. z'Uploading {} {} to Google Cloud StoragefileNrr+r Pluralizer)files_to_uploadrshow_progress_bar num_fileslabels r UploadFilesrsB/") 5 < <4>>)V4 6E EU3rc[U5nU(a(SRU[R"US55nOSn[ XU5 g)aODelete the given Cloud Storage objects. Uses the appropriate parallelism (multi-process, multi-thread, both, or synchronous). Args: objects_to_delete: list of ObjectDeleteTask num_threads: int (optional), the number of threads to use. show_progress_bar: bool. If true, show a progress bar to the users when deleting files. z(Deleting {} {} from Google Cloud StorageobjectNr)objects_to_deleterr num_objectsrs r DeleteObjectsrsD%&+ 6 = =T^^K: rs2'' 6#2-*)  3  ckk * !T!H t F!!Ht8%8$(;D.A"'4,2E$)6r