*SrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKJ r SSKJ r SSK J r SSKJr SS KJr SS KJr SSKJ r SS KJr SS KJr SS KJr SSKrSrSr\R:R<R>R@RC\5 \R:R<R>RDRC\5 SSjr#SSjr$Sr%SSjr&SSjr'SSjr(Sr)"SS\*5r+"SS\*5r,g)zHelpers for accessing GCS. Bulk object uploads and downloads use methods that shell out to gsutil. Lightweight metadata / streaming operations use the StorageClient class. )absolute_import)division)unicode_literalsN) exceptions)transfer) storage_api) storage_util)apis)log) properties) resourcesgs<c[RRRR 5nU(a [ X5 g[ XUS9 g)N)storage_client)r VALUESstorage use_gsutilGetBool _UploadGsutil_UploadStorageClient)files destinationrrs 6lib/googlecloudsdk/api_lib/dataproc/storage_helpers.pyUploadr8s9  ((33;;=*%%NKc U=(d [R"5nUHrn[RR U5n[RR X5n[ RRU5nURXG5 Mt g![Ra:n[R"SRSR U5X55eSnAff=f)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. storage_client: Storage api client used to copy files to gcs. z)Failed to upload files ['{}'] to '{}': {}', 'N)r StorageClientospathbasenamejoinr ObjectReferenceFromUrl CopyFileToGCSrBadFileException dp_exceptionsFileUploadErrorformat) rrrclientfile_to_upload file_namedest_url dest_objecterrs rrrBs  8[668&n  0Iww||K3H..66x@K5 >7   & &5  ) ) 5 < <kk% + 4 555sBC&,5C!!C&cUnX!/- n[R"SU5nUS:wa5[R"SR SR U5U55eg)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. cprz5Failed to upload files ['{0}'] to '{1}' using gsutil.rN)r RunGsutilCommandr(r)r*r#)rrargs exit_codes rrrWs` $-$++D$7)!^  ' '?FF KK   - ..rcU=(d [R"5nURU5$![Ra gf=f)zGets a bucket if it exists. Args: bucket: The bucket name. storage_client: Storage client instance. Returns: A bucket message, or None if it doesn't exist. N)rr GetBucketBucketNotFoundError)bucketrr+s rr7r7gsE  8[668&   F ##  ( ( s1AAc`U=(d [R"5nURXUS9 g)a[Creates a bucket. Creates a bucket in the specified region. If the region is None, the bucket will be created in global region. Args: bucket: Name of bucket to create. region: Region to create bucket in. storage_client: Storage client instance. project: The project to create the bucket in. If None, current Cloud SDK project is used. )locationprojectN)rrCreateBucketIfNotExists)r9regionrr<r+s rr=r=ys*  8[668&  ' JrcdU=(d [R"5n[RR U5nUR U5n[ R"USS9nUR5$![Ra& [R"SRU55ef=f)a9Reads an object's content from GCS. Args: object_url: The URL of the object to be read. Must have "gs://" prefix. storage_client: Storage api client used to read files from gcs. Raises: ObjectReadError: If the read of GCS object is not successful. Returns: A str for the content of the GCS object. zutf-8)encodingzFailed to read file '{0}'.)rrr r$r% ReadObjectio TextIOWrapperreadrr'r(ObjectReadErrorr*) object_urlrr+ object_refbytes_iowrappers rrArAs  8[668&++33J?*9  ,Hx':G <<>  $ $9  ' '$++J7 999s 5A55:B/c[RRU5nURURUR S9$)z.Build an Object proto message from a GCS path.)r9name)r REGISTRYParseStorageURLObjectr9object)r!messagesresources r GetObjectRefrRs3    / / 5( hoo FFrc4\rSrSrSrSrS SjrSrSrSr g) rzMicro-client for accessing GCS.ct[R"SS5Ul[R"SS5Ulg)Nrv1) core_apisGetClientInstancer+GetMessagesModulerPselfs r__init__StorageClient.__init__s*--i>DK// 4@DMrNcURRURURS9nURR R X2S9$![Ra gf=f)N)r9rO)requestdownload) rPStorageObjectsGetRequestr9rKr+objectsGetapitools_exceptionsHttpNotFoundError)r[rGr`r_s r _GetObjectStorageClient._GetObjectsemm44  5:G [[ $ $W $ HH  0 0 s"AA*)A*c$URU5$)a Get the object metadata of a GCS object. Args: object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Raises: HttpError: If the responses status is not 2xx or 404. Returns: The object if it exists otherwise None. )rf)r[rGs r GetObjectStorageClient.GetObjects ??: &&rct[RRXRSS9nUR X#S9 U$)aBuild an apitools Download from a stream and a GCS object reference. Note: This will always succeed, but HttpErrors with downloading will be raised when the download's methods are called. Args: stream: An Stream-like object that implements write() to write into. object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Returns: The download. F) total_size auto_transfer)r`)rDownload FromStreamsizerf)r[streamrGr`s rBuildObjectStreamStorageClient.BuildObjectStreams<  ++??%,AHOOJO2 Or)r+rPN) __name__ __module__ __qualname____firstlineno____doc__r\rfrirr__static_attributes__rrrrs'A' rrcd\rSrSrSrS Sjr\S5rSrSr Sr \ R4S jr S rg) StorageObjectSeriesStreamzFI/O Stream-like class for communicating via a sequence of GCS objects.NcjXlU=(d [5UlSUlSUlSUlg)aConstruct a StorageObjectSeriesStream for a specific gcs path. Args: path: A GCS object prefix which will be the base of the objects used to communicate across the channel. storage_client: a StorageClient for accessing GCS. Returns: The constructed stream. TrN) _base_pathr_gcs_open_current_object_index_current_object_pos)r[r!rs rr\"StorageObjectSeriesStream.__init__s2O1-/DIDJ"#D !DrcUR$)zWhether the stream is open.rrZs ropenStorageObjectSeriesStream.opens ::rcSUlg)zClose the stream.FNrrZs rCloseStorageObjectSeriesStream.Closes DJrc<UR(d [S5eg)NzI/O operation on closed stream.)r ValueErrorrZs r _AssertOpen%StorageObjectSeriesStream._AssertOpens 99 8 99 rcSRURU5nURR[ X RR 55$)z!Get the ith object in the series.z {0}.{1:09d})r*rrrirRrP)r[ir!s rrf$StorageObjectSeriesStream._GetObjects<    3D 99  |D))2D2DE FFrcUR5 SnSnUnX5:GazURURS-5nU(aU(a&URUR5nU(dU$URUR- nUS:a$[SRUR55eURS:XaUR5 U$XS- n [X5n U S:a`URRX5n U R!URURU -S- 5 U=RU - slX:- nU=(a URUR:Hn U (a!UnU=RS- slSUlGM}U$U$![Ra"n[ R "SU5 SnAU$SnAff=f)aORead from this stream into a writable. Reads at most n bytes, or until it sees there is not a next object in the series. This will block for the duration of each object's download, and possibly indefinitely if new objects are being added to the channel frequently enough. Args: writable: The stream-like object that implements write() to write into. n: A maximum number of bytes to read. Defaults to sys.maxsize (usually ~4 GB). Raises: ValueError: If the stream is closed or objects in the series are detected to shrink. Returns: The number of bytes read. rNzFailed to fetch GCS output: %szObject [{0}] shrunk.)rrfrrd HttpErrorr warningrprrr*rKrminrrrGetRange) r[writablen bytes_read object_infomax_bytes_to_readnext_object_infoerrornew_bytes_availablebytes_left_to_readnew_bytes_to_readr`object_finisheds rReadIntoWritable*StorageObjectSeriesStream.ReadIntoWritable s* JK  ()C)Ca)GH, (B(BC+ L I(,,t/G/GG q /66{7G7GHII   Q   8 5-90F Q 99..xE  $ $  $ $'8 81 < >   $55 '  Kt77;;K;KK &  ""a'"#$   :W#,,  ++7 ? R W s FG-G  G)rrrrrrt)rurvrwrxryr\propertyrrrrfsysmaxsizerrzr{rrr}r}s=N!*  :G *-Lrr}rt)NN)-ry __future__rrrrBr rapitools.base.pyrrdrgooglecloudsdk.api_lib.dataprocr(googlecloudsdk.api_lib.storagerr googlecloudsdk.api_lib.utilr rWgooglecloudsdk.calliopegooglecloudsdk.corer r r six.moves.urllib.parsesixSTORAGE_SCHEME HTTP_TIMEOUTmovesurllibparse uses_relativeappend uses_netlocrrrr7r=rArRrOrr}r{rrrs  '' >%G679.#*)  $$++N; "")).9L5* . $K"92G 4F4nvvr