SrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK Jr SSK Jr SS K Jr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJr SSK J!r! SSK"J#r# SSK"J$r$ \ RJ(a\&r'\ RP"S5r)Sr*"SS\+5r,"SS\,5r-Sr."SS\,5r/"S S!\'5r0S#S"jr1g)$z1Wildcard iterator class and supporting functions.)absolute_import)print_function)division)unicode_literalsN)BucketListingBucket)BucketListingObject)BucketListingPrefix)AccessDeniedException)CloudApi)NotFoundException)CommandException)ContainsWildcard)GenerationFromUrlAndString)StorageUrlFromString) StripOneSlash)WILDCARD_REGEX)storage_v1_messages)UTF8)FixWindowsEncodingIfNeeded) PrintableStrz (?P.*?)\*\*(?P.*)a\Invalid Unicode path encountered (%s). gsutil cannot proceed with such files present. Please remove or rename this file and try again. NOTE: the path printed above replaces the problematic characters with a hex-encoded printable representation. For more details (including how to convert to a gsutil-compatible encoding) see `gsutil help encoding`.c\rSrSrSrSrSrg)WildcardIterator?zClass for iterating over Google Cloud Storage strings containing wildcards. The base class is abstract; you should instantiate using the wildcard_iterator() static factory method, which chooses the right implementation depending on the base string. c4SURR-$)z2Returns string representation of WildcardIterator.zWildcardIterator(%s)) wildcard_url url_stringselfs *platform/gsutil/gslib/wildcard_iterator.py__repr__WildcardIterator.__repr__Is !D$5$5$@$@ @@N)__name__ __module__ __qualname____firstlineno____doc__r __static_attributes__r#r"rrr?s Ar"rc\rSrSrSrSSjrSSjrSrSrSr SS jr SS jr SS jr S r SS jrSSjrSSjrSrg)CloudWildcardIteratorNaLWildcardIterator subclass for buckets, bucket subdirs and objects. Iterates over BucketListingRef matching the Url string wildcard. It's much more efficient to first get metadata that's available in the Bucket (for example to get the name and size of each object), because that information is available in the object list results. NczXlX0lX lX@lU=(d [R "5Ulg)aUInstantiates an iterator that matches the wildcard URL. Args: wildcard_url: CloudUrl that contains the wildcard to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project ID to use for bucket listings. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. N)r all_versions gsutil_api project_idlogging getLoggerlogger)rrr/r.r0r3s r__init__CloudWildcardIterator.__init__Ws/&%$ O O/G--/DKr"c # # URR5nSnU(a[5nUHnURU5 M UR US9nUR SS/5 UR S/5 U(dUR (a&UR SS/5 UR SS /5 URS /S 9GHnURnURR5(aU(a[U5nURRURS UR URRUS 9Hqn U R[ R"R$:Xa)UR'UU R(UR S9v MTUR+XyR(5v Ms MUv GM[-URR5(dURR/5(aUR (dURR1URRURR2URR4URRUS9n UR'URR6U UR =(d US9v gU(a5U<URR2<SURR4<3n O.U<[=URR25=(d S <3n U /n U (dGMy[U R?S55nURAUR25upnnURCU5nU(a [S/5OUnURRURU UUR =(d UURRUS9GHin UGH^nU R[ R"R$:XaU R(nUREURF5(apU(a[=URF5U:XaMU(a URIUR45(a&UR'UUUR =(d US9v MMU R(n [-U 5(a[KSU<U <S35e[=U 5nUREU5(dGMU(a&UU:wa U<US -U-<3nU RMU5 OUR+X}5v GMg GMl U (aGM&GM g![8[:4a GNf=f7f)aIterator that gets called when iterating over the cloud wildcard. In the case where no wildcard is present, returns a single matching object, single matching prefix, or one of each if both exist. Args: bucket_listing_fields: Iterable fields to include in bucket listings. Ex. ['name', 'acl']. Iterator is responsible for converting these to list-style format ['items/name', 'items/acl'] as well as adding any fields necessary for listing such as prefixes. API implementation is responsible for adding pagination fields. If this is None, all fields are returned. expand_top_level_buckets: If true, yield no BUCKET references. Instead, expand buckets into top-level objects and prefixes. Yields: BucketListingRef of type BUCKET, OBJECT or PREFIX. N) get_fieldsz items/nameprefixesnamezitems/generationzitems/metageneration generationmetagenerationid bucket_fields/) delimiterr.providerfields) with_version)r:rArB#r)prefixr@r.rArBz Cloud folder z] contains a wildcard; gsutil does not currently support objects with wildcards in their name.)'r HasGenerationsetadd_GetToListFieldsupdater._ExpandBucketWildcardsrIsBucketrr/ ListObjects bucket_nameschemedatatyper CsObjectOrPrefixTypeOBJECT _GetObjectRefdata _GetPrefixRefrIsObjectGetObjectMetadata object_namer:bucket_url_stringr r rpop_BuildBucketFilterStrings_GetRegexPatternsmatchr9_SingleVersionMatchesr append)rbucket_listing_fieldsexpand_top_level_bucketssingle_version_requestr7fieldbucket_listing_refrYurl obj_or_prefix get_objectrurls_needing_expansionrEr@prefix_wildcardsuffix_wildcardregex_patternslisting_fieldspattern gcs_objectrstripped_prefixurl_append_strings r__iter__CloudWildcardIterator.__iter__ps0"..<<> J5j(%u)"33*4,""L*#=>! 4#4#4$$ !7 8 :<)9:; #999O,77   # # % % $$%67##::oo,,((//* ; ,m %%)F)F)M)MM&&'8'4'9'9484E4E'GG&&'8:L:LMM ,# " !!2!2!=!=>>    & & ( (1B1B ::!!--!!--,,77**11! ;#J $$T%6%6%H%H%/373D3D4J3I%L L  "$5$($5$5$A$A$($5$5$@$@B* t/@/@/L/L!M""*#-$$$%;%?%?%BC##==cooN 6o 11/B. /  -4I  $::oo!,,F0F((//# ; %m*''8+H+H+O+OO*// ==11(]:??-K0?.@122:3H3HII ..+$(,(9(9)?(> /AA 2'++#F++ )"3F<==$1#8 ==!122$)9_)L2C2BS2H2A3B)C%+112CD,,->GGS* %%$uPN$%:;   s@H"V%B'U1 A4VG VAV+V1VVVVcLU/nSU;a$URSS5nURU5 OUnX4H/nURS5(dMURUSS5 M1 UVs/sH-n[R"[ R "U55PM/ sn$s snf)aReturns list of regex patterns derived from the wildcard patterns. Args: wildcard_pattern (str): A wilcard_pattern to filter the resources. Returns: List of compiled regex patterns. This translates the wildcard_pattern and also creates some additional patterns so that we can treat ** in a/b/c/**/d.txt as zero or more folders. This means, a/b/c/d.txt will also be returned along with a/b/c/e/f/d.txt. z/**/r?**/N)replacer_ startswithrecompilefnmatch translate)rwildcard_patternwildcard_patternsupdated_patternrmps rr\'CloudWildcardIterator._GetRegexPatternss** !!)00=o/(o$6   E " "   -77H H6GBJJw((+ ,6G HH Hs*4B!c [R"U5nU(d UnSnUnSnOUR5S:a&USUR5nXR5SnOSnUnURS5nUS:waUSUS-nU=(d SU-nUR S5(d [ U5nXR 5SnURS5nUS:XaSnOXhS-SnURS5S:wa SnXV- nSnOSnURRS [U5[U5[U5[U5[U55 X4XV4$) aOBuilds strings needed for querying a bucket and filtering results. This implements wildcard object name matching. Args: wildcard: The wildcard string to match to objects. Returns: (prefix, delimiter, prefix_wildcard, suffix_wildcard) where: prefix is the prefix to be sent in bucket GET request. delimiter is the delimiter to be sent in bucket GET request. prefix_wildcard is the wildcard to be used to filter bucket GET results. suffix_wildcard is wildcard to be appended to filtered bucket GET results for next wildcard expansion iteration. For example, given the wildcard gs://bucket/abc/d*e/f*.txt we would build prefix= abc/d, delimiter=/, prefix_wildcard=d*e, and suffix_wildcard=f*.txt. Using this prefix and delimiter for a bucket listing request will then produce a listing result set that can be filtered using this prefix_wildcard; and we'd use this suffix_wildcard to feed into the next call(s) to _BuildBucketFilterStrings(), for the next iteration of listing/filtering. Raises: AssertionError if wildcard doesn't contain any wildcard chars. r?rNrtz**zMwildcard=%s, prefix=%s, delimiter=%s, prefix_wildcard=%s, suffix_wildcard=%s ) rsearchstartfindendswithrendr3debugr) rwildcardr]rEr@rirj wildcard_partrs rr[/CloudWildcardIterator._BuildBucketFilterStrings9sn:  ! !( +E fi oo  .5;;=) 0    s #c %hsQw/ 26o  % %e , ,(8 .o   %c )'(3   d #r ) *  KK 34@4JVl95_%|O'D F  @@r"c[URU5n[URR5[U5:H$N)rrstrr:)rlisted_generationdecoded_generations rr^+CloudWildcardIterator._SingleVersionMatchess;3D4E4E4EG t  ++ ,4F0G GGr"c ## [URR5nU(aK[U5[S/5:Xa2[ URR 5(d[ U5v gURR5(ar[ URR 5(dN[ UURRURR URRUS9S9v g[R"URR 5n[R"U5nURU5nU(aUR!S5 URR#UUR$URRS9HanUR'UR(5(dM%[URR<SUR(<S35n[ XvS9v Mc g7f) a+Expands bucket and provider wildcards. Builds a list of bucket url strings that can be iterated on. Args: bucket_fields: If present, populate only these metadata fields for buckets. Example value: ['acl', 'defaultObjectAcl'] Yields: BucketListingRefereneces of type BUCKET. r<)rArB root_objectzitems/id)rBr0rAz://r?N)rrrYrGrrNrrLr/ GetBucketrOrzr{rxryrIrH ListBucketsr0r]r<)rr> bucket_urlregexprogrBbucketres rrK,CloudWildcardIterator._ExpandBucketWildcardss&d&7&7&I&IJJ#m,TF ; T..:: ; ; ++    $ $ & &t00<<==  ,0OO,E,E$($5$5$A$A-1->->-E-E+8-F-: ;;  1 1 = =>e ZZ d$$]3f  :OO//__$$++0-& ::fii $&*&7&7&>&> &KL##C< <-s GH  AH cdU(a)[5nUHnURSU-5 M U$g)aPrepends 'items/' to the input fields and converts it to a set. This way field sets requested for GetBucket can be used in ListBucket calls. Note that the input set must contain only bucket or object fields; listing fields such as prefixes or nextPageToken should be added after calling this function. Args: get_fields: Iterable fields usable in GetBucket/GetObject calls. Returns: Set of fields usable in ListBuckets/ListObjects calls. zitems/N)rGrH)rr7 list_fieldsrcs rrI&CloudWildcardIterator._GetToListFieldss4Ek%5()  r"cU(aDURb7[URUR5nU<UR<SU<3nOU<UR<3n[ U5n[ XbS9$)aLCreates a BucketListingRef of type OBJECT from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. gcs_object: gsutil_api root Object for populating the BucketListingRef. with_version: If true, return a reference with a versioned string. Returns: BucketListingRef of type OBJECT. rDr)r:rrr9rr)rrYrnrCgeneration_str object_string object_urls rrS#CloudWildcardIterator._GetObjectRefsf --91$2C2C2<2G2GIn#4joo#13m!2:??Cm%m4J z BBr"c6[U<U<35n[X2S9$)zCreates a BucketListingRef of type PREFIX from the arguments. Args: bucket_url_string: Wildcardless string describing the containing bucket. prefix: gsutil_api Prefix for populating the BucketListingRef Returns: BucketListingRef of type PREFIX. r)rr )rrYrE prefix_urls rrU#CloudWildcardIterator._GetPrefixRefs&0A6&JKJ z >>r"c#@# URUS9HnUv M g7f)a0Iterates over the wildcard, returning refs for each expanded bucket. This ignores the object part of the URL entirely and expands only the the bucket portion. It will yield BucketListingRefs of type BUCKET only. Args: bucket_fields: Iterable fields to include in bucket listings. Ex. ['defaultObjectAcl', 'logging']. This function is responsible for converting these to listing-style format ['items/defaultObjectAcl', 'items/logging'], as well as adding any fields necessary for listing such as 'items/id'. API implemenation is responsible for adding pagination fields. If this is None, all fields are returned. Yields: BucketListingRef of type BUCKET, or empty iterator if no matches. r=N)rK)rr>blrs r IterBuckets!CloudWildcardIterator.IterBucketss%&***G iHc#B# URUUS9HnUv M g7f)aQIterates over the wildcard, yielding bucket, prefix or object refs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. expand_top_level_buckets: If true and the wildcard expands only to Bucket(s), yields the expansion of each bucket into a top-level listing of prefixes and objects in that bucket instead of a BucketListingRef to that bucket. Yields: BucketListingRef, or empty iterator if no matches. r`raNrq)rr`rars rIterAllCloudWildcardIterator.IterAll s,}}3H6NP iPsc#p# URUSS9HnUR5(dMUv M g7f)aIterates over the wildcard, yielding only object BucketListingRefs. Args: bucket_listing_fields: If present, populate only these metadata fields for listed objects. Yields: BucketListingRefs of type OBJECT or empty iterator if no matches. TrN)rqrV)rr`rs r IterObjects!CloudWildcardIterator.IterObjectss5}}3H6:<  $9' W6  ..  " "7 +  ;;;277>>-88 ++   W&@&CD D3))>." >6!mm3 g!9:;<"=> >7 >s+B:G"G&=G$+E GA&F>>GcURcgURup#nU(dg[U5R[UR5SnUR UR R 5(aUSSnURU5(a.UR(aURRSU5 gg)zCheck a directory to see if it should be excluded from os.walk. Args: dir: String representing the directory to check. Returns: True if the directory should be excluded. NFrz!Skipping excluded directory %s...T) rrrlenrwrdelimr]r3r)rdirbase_url exclude_dirsexclude_pattern str_to_checks rr FileWildcardIterator._ExcludeDirs ! 040B0B-X_  '  ZH//013Lt006677!!"%l\**  h0!$$ Br"rc*\rSrSrSrSrSrSrSrg)ri7z+Exception raised for invalid wildcard URLs.c:[RU5 Xlgr) StandardErrorr4reason)rr s rr4WildcardException.__init__:s4 Kr"c SUR-$NzWildcardException: %sr rs rr WildcardException.__repr__> "T[[ 00r"c SUR-$r rrs r__str__WildcardException.__str__Arr"rN) r$r%r&r'r(r4r rr)r#r"rrr7s311r"rc[U5nU=(d [R"5nUR5(a [ UUUUS9$[ UUUUS9$)aInstantiate a WildcardIterator for the given URL string. Args: url_str: URL string naming wildcard object(s) to iterate. gsutil_api: Cloud storage interface. Passed in for thread safety, also settable for testing/mocking. all_versions: If true, the iterator yields all versions of objects matching the wildcard. If false, yields just the live object version. project_id: Project id to use for bucket listings. exclude_tuple: (base_url, exclude_pattern), where base_url is top-level URL to list; exclude_pattern is a regex of paths to ignore during iteration. ignore_symlinks: For FileUrls, ignore symlinks during iteration if true. logger: logging.Logger used for outputting debug messages during iteration. If None, the root logger will be used. Returns: A WildcardIterator that handles the requested iteration. )rrr3)r.r0)rr1r2 IsFileUrlrr+)url_strr/r.r0rrr3res rCreateWildcardIteratorrEs^8 W%#  (W&&(&]]__ .;0?'- // !!+.:,6 88r")FNNFN)2r( __future__rrrrrzrr1rrxrrgslib.bucket_listing_refrrr gslib.cloud_apir r r gslib.exceptionr gslib.storage_urlrrrrr"gslib.third_party.storage_apitoolsrrgslib.utils.constantsrgslib.utils.text_utilrrPY3 Exceptionr ryrrobjectrr+rrrrr#r"rr#s8&%'   8881$-,.82+,W&<.77-**@A> Av A],]@B"uB+uBp 1 1 ).&*)-+0"& '8r"