G6SrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK J r SSK J r SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJ r SSK!r!Sr"Sr#S#Sjr$\RJRL4Sjr'\RJRL4Sjr(Sr)Sr*\RJRL4Sjr+Sr,Sr-Sr.\RJRL4Sjr/Sr0Sr1S$S jr2S!r3\RJRL4S"jr4g)%zCCommon utility functions for Composer environment storage commands.)absolute_import)division)unicode_literalsN) exceptions)transfer)environments_util) storage_api) storage_util)base)util) gcloudignore)execution_utils)log) properties)filesz8Could not retrieve Cloud Storage bucket for environment.cU(aOSU;d"SU;d[R"SU5(a&[R"SR U55 ggg)z9Logs deprecation warning if gsutil wildcards are in args.*?z\[.*\]zUse of gsutil wildcards is no longer supported in {0}. Set the storage/use_gsutil property to get the old behavior back temporarily. However, this property will eventually be removed.N)researchrwarningformat)path flag_names 7lib/googlecloudsdk/command_lib/composer/storage_util.pyWarnIfWildcardIsPresentr-sF sd{cTkRYYy$-G-GKK"6),..HTczU(a[R"X5$[RRX5$)z@Joins paths using the appropriate separator for local or gsutil.) posixpathjoinosr)path1path2 gsutil_paths r _JoinPathsr%8s' >>% '' 77<< %%rcb[XS9n[R"5nURX1S-S9$)a Lists all resources in one folder of bucket. Args: env_ref: googlecloudsdk.core.resources.Resource, Resource representing the Environment whose corresponding bucket to list. gcs_subdir: str, subdir of the Cloud Storage bucket which to list release_track: base.ReleaseTrack, the release track of command. Will dictate which Composer client library will be used. Returns: list of Objects inside subdirectory of Cloud Storage bucket for environment Raises: command_util.Error: if the storage bucket could not be retrieved  release_track/)prefix)_GetStorageBucketr StorageClient ListBucket)env_ref gcs_subdirr( bucket_refstorage_clients rListr2@s5 !F*,,..  " ":36F " GGrc[XS9n[RRRR 5nU(a [ XAU5 g[XAU5 g)a|Imports files and directories into a bucket. Args: env_ref: googlecloudsdk.core.resources.Resource, Resource representing the Environment whose bucket into which to import. source: str, a path from which to import files into the environment's bucket. Directory sources are imported recursively; the directory itself will be present in the destination bucket. Must contain at least one non-empty value. destination: str, subdir of the Cloud Storage bucket into which to import `sources`. Must have a single trailing slash but no leading slash. For example, 'data/foo/bar/'. release_track: base.ReleaseTrack, the release track of command. Will dictate which Composer client library will be used. Returns: None Raises: command_util.Error: if the storage bucket could not be retrieved command_util.GsutilError: the gsutil command failed r'N)r+rVALUESstorage use_gsutilGetBool _ImportGsutil_ImportStorageApir.source destinationr( gcs_bucketr6s rImportr>UsE.!F*  ((33;;=**k2j+6rc [R"5nUnURS5nXA:gnU(dUR[R5nUR S5(+nU(a:[ RRU5(d[R"S5e[[ RRU5SU(+S9nU(Ga[ RRU5(a[R"U5nUR!U5Hrn [X5n [ RRU 5(aM4[#XzUS5n [$R&R)UU 5n UR+X5 Mt g[#XqUS5n [$R&R)UU 5n UR+X5 g[$R&R-U5n [/XUS9nUHNn[$R&R)U[#XR15US55nUR3UU5 MP g) ,Imports files and directories into a bucket.rgs://z!Source for import does not exist.r$Fobject_is_subdirclientN)r r,rstriprsep startswithr!rexists command_utilErrorr%dirnameisdirr GetFileChooserForDirGetIncludedFiles _GetDestPathr ObjectReference FromBucketRef CopyFileToGCSFromUrl_GetObjectOrSubdirObjectsToUrlCopy)r=r;r<rF old_sourcerEsource_is_localsource_dirname file_chooserrel_path file_path dest_pathobj_ref source_ref to_importobj dest_objects rr9r9us  $ $ &&* == &  ) ]]9== )F))'22/RWW^^F33   @ AAggoofr?/BD. ww}}V!66v>l"33F;(v0 77== # #  KO ..<>  ~yy{K GIk kk#{# rc[RRX5n[R"SSXR 5/S[ R R[ RRS9nU(a[R"S5eg![R[R4a/n[R"[R"U55eSnAff=f)r@cp-rT command_argsrun_concurrentout_funcerr_funcN%gsutil returned non-zero status code.)r rRrSRunGsutilCommandrWroutwriteerrrPermissionErrorInvalidCommandErrorrK GsutilErrorsix text_type)r=r;r<destination_refretvales rr8r8s 00>>/ 5  * * V%:%:%<=  F  " "#J KK   ) )  - - /5  " "3==#3 445sAB$C*;*C%%C*c[XS9n[RRRR 5nU(a [ XAU5 g[XAU5 g)aExports files and directories from an environment's Cloud Storage bucket. Args: env_ref: googlecloudsdk.core.resources.Resource, Resource representing the Environment whose bucket from which to export. source: str, a bucket-relative path from which to export files. Directory sources are imported recursively; the directory itself will be present in the destination bucket. Can also include wildcards. destination: str, existing local directory or path to a Cloud Storage bucket or directory object to which to export. Must have a single trailing slash but no leading slash. For example, 'dir/foo/bar/'. release_track: base.ReleaseTrack, the release track of command. Will dictate which Composer client library will be used. Returns: None Raises: command_util.Error: if the storage bucket could not be retrieved or a non-Cloud Storage destination that is not a local directory was provided. command_util.GsutilError: the gsutil command failed r'N)r+rr4r5r6r7 _ExportGsutil_ExportStorageApir:s rExportr}sE0!F*  ((33;;=**k2j+6rc jUnURS5nX1:gn[R"5n[RR X5nSnUR S5(a+[UR[R5SSS9nSnO:[RRU5(d[R"S5e[[RR!U5SSS9n[#XdUS9n U(adU H]n [%XR&US5n [(R*"[RR!U 55 UR-XSS 9 M_ g U HHn [RR/[%XR&US55n UR1X5 MJ g ) ?Exports files and directories from an environment's GCS bucket.rTrArBrCF+Destination for export must be a directory.rD) overwriteN)rGr r,r rRrSrIr%striprrHr!rrNrKrLrMrVrQnamerMakeDirCopyFileFromGCSrUrX) r=r;r<rYrErFra dest_is_localr[ to_exportrcr_rds rr|r|s\* == &  )  $ $ &&++99*M*-G$$)--("$@KM 77== % %   J KKbggoof5rtL.'FD)~xxdKi mmBGGOOI./ St<  0088 ~xxe DFk kk##rc[RRX5nURS5(a)[ UR [ R5SSS9nO:[RRU5(d[R"S5e[R"SSUR5U/S[R R"[R$R"S9nU(a[R,"S 5eg ![&R([&R*4a/n[R,"[.R0"U55eS nAff=f) rrArBTrCrrfrgrhNrm)r rRrSrIr%rrrHr!rrNrKrLrnrWrrorprqrrrrsrtrurv)r=r;r<rarxrys rr{r{s++99*M*G$$)--("$@K 77== % %   J KK 5  * * J,,. <  F  " "#J KK   ) )  - - /5  " "3==#3 445sAD$E$5*EE$cUnURU5(aU[U5SnU(d8UR[RR [ R 5n[X$U(+S9$)z?Get dest path without the dirname of the source dir if present.NrC)rIlenreplacer!rrHrr%)r[ source_pathr<rdest_path_suffixs rrQrQse N++"3~#6#78 '// Y]]K ]1B DDrc[XS9n[RRRR 5nU(a [ XAU5 O [XAU5 [RRS5 [XB5 g)a}Deletes objects in a folder of an environment's bucket. gsutil deletes directory marker objects even when told to delete just the directory's contents, so we need to check that it exists and create it if it doesn't. A better alternative will be to use the storage API to list objects by prefix and implement deletion ourselves Args: env_ref: googlecloudsdk.core.resources.Resource, Resource representing the Environment in whose corresponding bucket to delete objects. target: str, the path within the gcs_subdir directory in the bucket to delete. If this is equal to '*', then delete everything in gcs_subdir. gcs_subdir: str, subdir of the Cloud Storage bucket in which to delete. Should not contain slashes, for example "dags". release_track: base.ReleaseTrack, the release track of command. Will dictate which Composer client library will be used. r'zDeletion operation succeeded.N) r+rr4r5r6r7 _DeleteGsutil_DeleteStorageApirstatusPrint_EnsureSubdirExists)r.targetr/r(r=r6s rDeleter's`*!F*  ((33;;=**j1j*5**23j-rc [R"5nUS:HnU(aSOUn[RR U[ X!SS95n[ XTUS9nUHnURU5 M g)zHDeletes objects in a folder of an environment's bucket with storage API.rrBTrCrDN)r r,r rRrSr%rV DeleteObject)r=rr/rF delete_all target_ref to_deleter`s rrrHsp  $ $ && }*2&++99*ZTBD*(f>)g  rc [RRU[X!SS95n[R"SSUR 5/S[ RR[ RRS9nU(a[R"S5eg![R[R4a/n[R"[R "U55eSnAff=f)zCDeletes objects in a folder of an environment's bucket with gsutil.TrCrmrgrhNrm)r rRrSr%rnrWrrorprqrrrrsrKrtrurv)r=rr/rrxrys rrr\s++99*ZTBD* 5  * * Z--/0  F  " "#J KK   ) )  - - /5  " "3==#3 445sAB $C3*C..C3cJU=(d [R"5n/nSnU(d#URU5 URU5 U(dU(a[ R"URS5nURURU5nUHinU(aURUR:XaM&UR[RRURUR55 Mk U$![R a SnNf=f)zAGets object_ref or the objects under object_ref is it's a subdir.FTrB)r r, GetObjectappendapitools_exceptionsHttpNotFoundErrorrr rr-r0r rRFromNamebucket) object_refrErFobjectstarget_is_subdir target_pathsubdir_objectsrcs rrVrVns  0[..0& '  z" nnZ ).."5K&&z'<'r9r8r}r|r{rQrrrrVrr+rrrsJ&'  >%V67(D8/#** P.&-1,=,=,@,@H*8<7H7H7K7K7@2$jL$8<7H7H7K7K7B#$LL. D7;6G6G6J6J.B!(L$6#'L.2->->-A-A r