ySrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK J r SSKJr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKrSr\ R@"S5r!\ R@"S5r"\#"SS\ RH-SS\ RH-/5r%Sr&Sr'SSS\RPRRSSSSS\RTRV\RXRVS\RZR\SSSS4Sjr/Sr0"SS\Rb"\Rd55r3"SS\35r4"SS \35r5"S!S"5r6S#r7g)$z1Utilities for expanding wildcarded GCS pathnames.)absolute_import)division)unicode_literalsN)Iterator) api_factory) cloud_api)errors)request_config_factory) folder_util) storage_url)resource_reference)log) debug_outputz!Expected files but got stream: {}z\*{3,}z[*?\[\]].z..cdUR[R5SRS5$)Nr) rpartitionossep startswith)paths ;lib/googlecloudsdk/command_lib/storage/wildcard_iterator.py _is_hiddenr3s%   # . .s 33c>[[RU55$)zuChecks whether url_string contains a wildcard. Args: url_string: URL string to check. Returns: bool indicator. )boolWILDCARD_REGEXsearch) url_strings rcontains_wildcardr 7s n##J/ 00rTFc&[R"U5n[U[R5(a[ UUUUUUUUU U U U UUUS9$[U[R 5(a[ UUUUU US9$[R"SU-5e)aInstantiate a WildcardIterator for the given URL string. Args: url_str (str): URL string which may contain wildcard characters. error_on_missing_key (bool): If true, and the encryption key needed to decrypt an object is missing, the iterator raises an error for that object. exclude_patterns (Patterns|None): Don't return resources whose URLs or local file paths matched these regex patterns. fetch_encrypted_object_hashes (bool): Fall back to GET requests for encrypted cloud objects in order to fetch their hash values. fields_scope (cloud_api.FieldsScope): Determines amount of metadata returned by API. files_only (bool): Skips containers. Raises error for stream types. Still returns symlinks. force_include_hidden_files (bool): Include local hidden files even if not recursive iteration. URL should be for directory or directory followed by wildcards. get_bucket_metadata (bool): If true, perform a bucket GET request when fetching bucket resources. halt_on_empty_response (bool): Stops querying after empty list response. See CloudApi for details. ignore_symlinks (bool): Skip over symlinks instead of following them. managed_folder_setting (folder_util.ManagedFolderSetting): Indicates how to deal with managed folders. folder_setting (folder_util.FolderSetting): Indicates how to deal with folders. next_page_token (str|None): Used to resume LIST calls. object_state (cloud_api.ObjectState): Versions of objects to query. preserve_symlinks (bool): Preserve symlinks instead of following them. raise_managed_folder_precondition_errors (bool): If True, raises precondition errors from managed folder listing. Otherwise, suppresses these errors. This is helpful in commands that list managed folders by default. soft_deleted_buckets (bool): If true, soft deleted buckets will be queried. list_filter (str|None): If provided, objects with matching filters will be returned, The prefixes would still be returned regardless of whether they match the specified filter, See go/gcs-object-context-filtering for more details. Returns: A WildcardIterator object. )error_on_missing_keyexclude_patternsfetch_encrypted_object_hashes fields_scope files_onlyget_bucket_metadatahalt_on_empty_responsemanaged_folder_settingfolder_settingnext_page_token object_state(raise_managed_folder_precondition_errorssoft_deleted_buckets list_filter)r#r&force_include_hidden_filesignore_symlinkspreserve_symlinkszUnknown url type %s.) r storage_url_from_string isinstanceCloudUrlCloudWildcardIteratorFileUrlFileWildcardIteratorcommand_errorsInvalidUrlError)url_strr"r#r$r%r&r0r'r(r1r)r*r+r,r2r-r.r/urls rget_wildcard_iteratorr=Cs~ ++G4#[))**  1)&C!/55%'!1Y1 "#{**++  )#='+    ( ()?#)E FFrc[R"[SUR5nURbUSUR-- n[ R "U5$)zAsterisk counts greater than two treated as single * to mimic globs. Args: url (StorageUrl): Url to compress wildcards in. Returns: StorageUrl built from string with compressed wildcards. *#)resubCOMPRESS_WILDCARDS_REGEXversionless_url_string generationr r3)r<compressed_url_strings r_compress_url_wildcardsrGsQ&&!93!$!;!;=^^S3>>11  , ,-B CCrc,\rSrSrSrSSjrSrSrg)WildcardIteratorzClass for iterating over Google Cloud Storage strings containing wildcards. The base class is abstract; you should instantiate using the wildcard_iterator() static factory method, which chooses the right implementation depending on the base string. Nc<[U5UlX lX0lg)z@Initializes class. See get_wildcard_iterator for Args docstring.N)rG_url_exclude_patterns _files_only)selfr<r#r&s r__init__WildcardIterator.__init__s(,DI-!rc6S[URSS5-$)z2Returns string representation of WildcardIterator.zWildcardIterator(%s)rN)getattrrLrOs r__repr__WildcardIterator.__repr__s !GDII|T$J JJr)rMrNrL)NF)__name__ __module__ __qualname____firstlineno____doc__rPrU__static_attributes__rrrIrIs "KrrIc@^\rSrSrSrSU4SjjrSrSrU=r$)r8z,Class to iterate over files and directories.c>[[U] UUUS9 XPlX`lU(aFUR R S5S[R:wa[R"S5eURR Ul SUR;Ul UR=(d U=(d [UR5Ulg)a9Initialize FileWildcardIterator instance. Args: url (FileUrl): A FileUrl instance representing a file path. exclude_patterns (Patterns|None): See get_wildcard_iterator. files_only (bool): Returns files and symlinks, skips folders, errors on streams. force_include_hidden_files (bool): Include hidden files even if not recursive iteration. URL should be for directory or directory followed by wildcards. ignore_symlinks (bool): Skip over symlinks instead of following them. preserve_symlinks (bool): Preserve symlinks instead of following them. )r&r?z`If force-including hidden files, input URL must be directory or directory followed by wildcards.**N)superr8rP_ignore_symlinks_preserve_symlinks resource_namerstriprrr9r:rL_path_recurser_include_hidden_files)rOr<r#r&r0r1r2 __class__s rrPFileWildcardIterator.__init__s, . / ,/ #    $ $S )" - 7  * * . ((DJDJJ&DM M3Mz$**7M rc#@ ^ # URR(aqUR(a=[R"[ R URR55e[R"UR5v gUR[;a#[R"UR5v g[R"UR5R5nUR (aB[#UR$SS5n[&R(R*"U6nUR,nOSnURnUR/S5(a [&R(R+US5nS[&R0-nURR3U5(aUm OSm U 4Sj[R"U5R5U55nUGHnUR6(a UR6R9U5(d!UR:(d[=U5(aMXUR(a~[&R(R?U5(dZ[@RB"U5(a=[R"[ R URR55eM[&R(REU5nU(anURF(d][&R(RIU5(dURJ(a([LRN"SR U55 GM{URR/S5(acU(aURF(dK[&R(RQU5(a$[&R(RIU5(aGM[@RR"U5n U(d?[&R(RIU5(a[R"U 5v GMZU(a,URF(a[RT"U 5v GM[R"XS9v GM g7f) Nrrbr?c3@># UHnT[U5-v M g7fN)str).0p path_prefixs r 0FileWildcardIterator.__iter__..s!?A c!f?szSkipping symlink {}) is_symlink)+rLis_stdiorNr9r:_FILES_ONLY_ERROR_FORMATformatrfr FileObjectResourcerh_RELATIVE_PATH_SYMBOLSFileDirectoryResourcepathlibPath expanduserrootlistpartsrrjoinanchorendswithrrglobrMmatchrjrisfiler is_named_pipeislinkreisdirrdrwarningexistsr7FileSymlinkPlaceholderResource) rO pathlib_path path_components_relative_to_rootpath_relative_to_rootr current_working_directory_prefix path_iteratorrrxfile_urlrus @r__iter__FileWildcardIterator.__iter__s yy  ,, $ + +DII,C,C D    1 1$)) <<  zz++  4 4TYY ?? << +668L*.l.@.@.D)E& ggll,LM  d d"jj%%d++ ggll+@#F(+RVV|$ zz=>>4kkd#(()>?M   T%;%;%A%A$%G%G((Z-=-=  "''.."6"6  $ $T * *..&--dii.E.EF    77>>$'j ))ww}}T""d&;&; )0067 **  d # #$"9"977>>$''277==+>+>$$T*h BGGMM$// 66x@@ $11 ??II 33   UsRR)rdrjrhreri)NFFFF) rWrXrYrZr[rPrr\ __classcell__rks@rr8r8s)4 !&*XV V rr8c^\rSrSrSrSSS\R RSSS\RR\RRS\RRSSS4U4SjjrSrSrS rSS jrS rSS jrSS jrSrSSjrSSjrSrSrSrS\S\\R>4Sjr S\S\\R>4Sjr!Sr"Sr#U=r$$)r6iRzAClass to iterate over Cloud Storage strings containing wildcards.TNFc>[[U] XUS9 [R"UR R 5UlX lX@l XPl Xpl Xl Xl XlXlXlU UlXlXlU [(R*R,La7UR R.b [(R*R0UlOXlU [(R*R4LUlUR [(R*R0L=(d+ UR6=(a UR R.SLUlg)aInstantiates an iterator that matches the wildcard URL. Args: url (CloudUrl): CloudUrl that may contain wildcard that needs expansion. error_on_missing_key (bool): If true, and the encryption key needed to decrypt an object is missing, the iterator raises an error for that object. exclude_patterns (Patterns|None): See get_wildcard_iterator. fetch_encrypted_object_hashes (bool): Fall back to GET requests for encrypted objects in order to fetch their hash values. fields_scope (cloud_api.FieldsScope): Determines amount of metadata returned by API. files_only (bool): Returns cloud objects, not prefixes or buckets. Also skips directory placeholder objects, although they are technically objects. get_bucket_metadata (bool): If true, perform a bucket GET request when fetching bucket resources. Otherwise, bucket URLs without wildcards may be returned without verifying the buckets exist. halt_on_empty_response (bool): Stops querying after empty list response. See CloudApi for details. managed_folder_setting (folder_util.ManagedFolderSetting): Indicates how to deal with managed folders. folder_setting (folder_util.FolderSetting): Indicates how to deal with folders. next_page_token (str|None): Used to resume LIST calls. object_state (cloud_api.ObjectState): Versions of objects to query. raise_managed_folder_precondition_errors (bool): If True, raises precondition errors from managed folder listing. Otherwise, suppresses these errors. This is helpful in commands that list managed folders by default. soft_deleted_buckets (bool): If true, soft deleted buckets will be queried. list_filter (str|None): If provided, objects with matching contexts will be returned. The prefixes would still be returned regardless of whether they match the specified context, See go/gcs-object-context-filtering for more details. )r#r&N)rcr6rPrget_apirLscheme_client_error_on_missing_key_fetch_encrypted_object_hashes _fields_scope_get_bucket_metadata_halt_on_empty_response_managed_folder_setting_folder_setting_next_page_token _object_state)_raise_managed_folder_precondition_errors_soft_deleted_buckets _list_filterr ObjectStateLIVErELIVE_AND_NONCURRENT_object_state_for_listing SOFT_DELETED _soft_deleted _object_state_requires_expansion)rOr<r"r#r$r%r&r'r(r)r*r+r,r-r.r/rks rrPCloudWildcardIterator.__init__Us+n / :0&&tyy'7'78DL!5*G'% 3#9 #9 )+%0 2"6#  --222 II ,'0'<'<'P'Pd$'3$%)>)>)K)KKD i33GGG A    ?499#7#74#? )rc#,# UR(a?URR5(dURR5(agURR5(a:URR UR URS9HnUv M gUR5GHnURR5(aUv M)URURR5nURURRUS9GH[nUR(a6URRURR5(aMKUR(ai[!U["R$5(aHURR&R)[R*5(aUR,S:XaMUR.[0R2R4La"[!U["R65(dGMUR8[0R:R4La"[!U["R<5(dGMWUv GM^ GM g7f)Nr% soft_deleted) is_hns_bucketr)rNrL is_provider is_bucketr list_bucketsrr_fetch_buckets_is_hns_bucketr bucket_name_fetch_sub_bucket_resourcesrMrrDr4r ObjectResourcerfrCLOUD_URL_DELIMITERsizerr ManagedFolderSettingLIST_WITHOUT_OBJECTSManagedFolderResourcer FolderSettingFolderResource)rObucket_resourcebucket_or_unknown_resourcerresources rrCloudWildcardIterator.__iter__s TYY2244 8K8K8M8M  yy!\\66))117/  )-(;(;(= $ 99   * * --(44@@ -::(44@@); h%%$*@*@*F*F$$;;++x);)J)JKK((66??#77! *,,33HHI"0FF$$,,AAB"8-?-N-NOONI )>sJJc UR(Ga[RRURR ;GaUR [RR:wGai[U[R5(GaIUR(Gd7UR(Gd%UR(aYURRUR UR"UR$R&UR UR(S9$UR*(a[,R."UR0UR*UR2S9n[5UR6SS5(aZURRUR UR"UUR$R&UR UR(S9$U$)NrEr%r)decryption_key_hash_sha256r"decryption_key)rr Capability ENCRYPTIONr capabilitiesr FieldsScopeSHORTr4r r crc32c_hashmd5_hashkms_keyget_object_metadatabucketnamerLrErrr get_request_configr rrS resource_args)rOrrequest_configs r_decrypt_resource_if_necessary4CloudWildcardIterator._decrypt_resource_if_necessarysf +++  + +t||/H/H H   )"7"7"="= = x!3!B!B C C%%%):):):   ||// OO MMyy++++++ 0    , ,/BB  '/'J'J!%!;!;= >//1A4 H H11oomm------ 2   Orc URRUURR[R "UR5URR URURS9nURU5$![Ra g[RanURS:XayS[U5;ajURRR!URR"5(a,UR(a[$R&"S5 SnAgeSnAff=f)z/Matches user input that doesn't need expansion.rizYou must specify a generationzGET failed with "must specify generation" error. This is expected for a soft-deleted object listed with a trailing slash. Falling back to a LIST call.N)rrrLrfr rrErrr api_errors NotFoundError GcsApiError status_coderrrr delimiterrdebug)rOrres r_try_getting_object_directly2CloudWildcardIterator._try_getting_object_directly s 11  )) ! ! 3 3DII >YY))))))2h 0 0 ::  # #  " !  ! ! --3 -Q7ii""++DII,?,?@@    3   s%BB E!E4BD>=D>>EcL[URR5=(dL UR=(d9 URRR URR 5nU(dURU5nU(aU/$URX5$)z:Fetch all objects for the given bucket that match the URL.) r rLrfrrrrr_expand_object_path)rOrrneeds_further_expansiondirect_query_results rr1CloudWildcardIterator._fetch_sub_bucket_resources/s $))112 >  0 0 > 99   ( ()<)< = # ==kJ #$$  # #K ??rc## URSLnUR[RR[RR 4;nU(af[ RRURR;a4U(a-URRXR=(d SS9nO/nUHnUv M g![Ra UR(aegf=f7f)Nrprefix)rrr rLIST_WITH_OBJECTSrrrMANAGED_FOLDERSrrlist_managed_foldersrrPreconditionFailedErrorr)rOrwildcard_partsis_recursive_expansionshould_list_managed_foldersmanaged_folder_iteratorrs r_get_managed_folder_iterator2CloudWildcardIterator._get_managed_folder_iterator>s,55="&">">((::((==C# %""22dll6O6OO$"&,,"C"C#,A,A,IT#D# #%-(.  - - 7 7  8s+AC<A>CloudWildcardIterator._get_resource_iterator..sX11< >  , , A A     / /t||7P7P P o''j \\ , ,  *":":  # # os;CC)(C)c8[U5[RLdtUR[R R [R R1;d2[RRURR;aU$UR[R RLa U(dU$URnURRURUR 5$!["R$a Us$f=f)z E ! $ , , .d *55dII> ?@!!)00=o/(o$6   E " "   -77H H6GBJJw((+ ,6G HH Hs04C'c#J# URU5nUHnURR(a0URRURR:waMNUH4nUR URR 5(dM/Uv M M g7f)a.Filter out resources that do not match the wildcard_pattern. Args: resource_iterator (iterable): An iterable resource_reference.Resource objects. wildcard_pattern (str): The wildcard_pattern to filter the resources. Yields: resource_reference.Resource objects matching the wildcard_pattern N)r=rLrEr rrf)rOr/r9regex_patternsr regex_patterns rr''CloudWildcardIterator._filter_resourcests--.>?N% ))      ) )TYY-A-A A)-   x33AA B B. * &s BB#B#c[URR5(a%URURR5$URR 5(aUR (aoUR (a%URURR5$URRURRURS9/$[R"UR5/$)ztFetch the bucket(s) corresponding to the url. Returns: An iterable of BucketResource or UnknownResource objects. )rr%) r rLr_list_buckets_matching_wildcardrrr-_fetch_all_soft_deleted_generations_of_bucketr get_bucketrr UnknownResourcerTs rr$CloudWildcardIterator._fetch_bucketss ..//  1 1$))2G2G HH     4#<#<  # #AA II ! !  ,, ! !))//-- " !00; <CC#)rrrrrrrrrrrrrrrr)F)%rWrXrYrZr[rrNO_ACLr rr rrrrPrrrrrrrrrrr=r'rrrrr BucketResourcerErDrr\rrs@rr6r6Rs I  $)((//!(==II ..::((--/3 !Un:x"H!F @88=$N8=?B0 ('a/b/c/d', '*e/f/*.txt') _split_on_wildcard('*e/f/*.txt') => ('', '*e/f/*.txt') _split_on_wildcard('a/b/c/d') => ('a/b/c/d', '') Nro)rrstart)rarfirst_wildcard_idxr wildcard_strs rr^r^sQ    '% ] 2:{{} %% &&+,,  r)8r[ __future__rrrabcr"r7rrrrAtypingrgooglecloudsdk.api_lib.storagerrr rr "googlecloudsdk.command_lib.storager9r r ,googlecloudsdk.command_lib.storage.resourcesr googlecloudsdk.corergooglecloudsdk.core.utilrsixrzr6rCr frozensetrr}rr rrXrr rrrr=rGwith_metaclassABCMetarIr8r6r%r^r]rrrzs_ 8&'   64?AG::K#1 ?::i0K("CrvvtTBFF]#KL4 1"'&&--$&;;GG,,88&&++-2%\G~ D Ks))#++6K0E +E PU ,U p1+1+hr