wSrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK J r SSK r SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKr SSKJr "SS\5r"SS\5r"SS\5rS&Sjr"SS\5r "SS\5r!"SS\5r""SS\5r#\RH"SSS /5r%\RH"S!S"S#/5r&"S$S%\ RN5r(g)'aBName expansion iterator and result classes. Name expansion support for the various ways gsutil lets users refer to collections of data (via explicit wildcarding as well as directory, bucket, and bucket subdir implicit wildcarding). This class encapsulates the various rules for determining how these expansions are done. )absolute_import)print_function)division)unicode_literalsN)encoding)CommandException)NO_URLS_MATCHED_GENERIC)NO_URLS_MATCHED_TARGET)PluralityCheckableIterator)SeekAheadResult)storage_v1_messages)StorageUrlFromStringc$\rSrSrSrSrSrSrg)NameExpansionResult/aHolds one fully expanded result from iterating over NameExpansionIterator. The member data in this class need to be pickleable because NameExpansionResult instances are passed through Multiprocessing.Queue. In particular, don't include any boto state like StorageUri, since that pulls in a big tree of objects, some of which aren't pickleable (and even if they were, pickling/unpickling such a large object tree would result in significant overhead). The state held in this object is needed for handling the various naming cases (e.g., copying from a single source URL to a directory generates different dest URL names than copying multiple URLs to a directory, to be consistent with naming rules used by the Unix cp command). For more details see comments in _NameExpansionIterator. cXlX lX0lX@lXPlU(a[ R "U5UlgSUlg)aInstantiates a result from name expansion. Args: source_storage_url: StorageUrl that was being expanded. is_multi_source_request: bool indicator whether multiple input URLs or src_url_str expanded to more than one BucketListingRef. is_multi_top_level_source_request: same as is_multi_source_request but measured before recursion. names_container: Bool indicator whether src_url names a container. expanded_storage_url: StorageUrl that was expanded. expanded_result: cloud object metadata in MessageToJson form (for pickleability), if any was iterated; None otherwise. Consumers must call JsonToMessage to get an apitools Object. N)source_storage_urlis_multi_source_request!is_multi_top_level_source_requestnames_containerexpanded_storage_urlr MessageToJsonexpanded_result)selfrrrrrrs 'platform/gsutil/gslib/name_expansion.py__init__NameExpansionResult.__init__@sH"1#: -N** 4+$11D15 c SUR-$)Nz%s)rrs r__repr__NameExpansionResult.__repr__Ys $++ ++r)rrrrrrN)__name__ __module__ __qualname____firstlineno____doc__rr!__static_attributes__rrrr/s 62,rrc:\rSrSrSrSSjrSrSrSrg) _NameExpansionIterator]zhClass that iterates over all source URLs passed to the iterator. See details in __iter__ function doc. Nc "XlX lX0lX@lXPlX`lXplURR5URlXl Xl Xl Xl U (d [S/5OU UlSSS.Ulg)a Creates a NameExpansionIterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: PluralityCheckableIterator of URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for bucket retrieval. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Examples of _NameExpansionIterator with recursion_requested=True: - Calling with one of the url_strs being 'gs://bucket' will enumerate all top-level objects, as will 'gs://bucket/' and 'gs://bucket/*'. - 'gs://bucket/**' will enumerate all objects in the bucket. - 'gs://bucket/abc' will enumerate either the single object abc or, if abc is a subdirectory, all objects under abc and any of its subdirectories. - 'gs://bucket/abc/**' will enumerate all objects under abc or any of its subdirectories. - 'file:///tmp' will enumerate all files under /tmp, as will 'file:///tmp/*' - 'file:///tmp/**' will enumerate all files under /tmp or any of its subdirectories. Example if recursion_requested=False: calling with gs://bucket/abc/* lists matching objects or subdirs, but not sub-subdirs or objects beneath subdirs. Note: In step-by-step comments below we give examples assuming there's a gs://bucket with object paths: abcd/o1.txt abcd/o2.txt xyz/o1.txt xyz/o2.txt and a directory file://dir with file paths: dir/a.txt dir/b.txt dir/c/ namez***)TFN) command_namedebuglogger gsutil_apiurl_strsrecursion_requested all_versions HasPlurality has_pluralitycmd_supports_recursion project_idignore_symlinkscontinue_on_errorsetbucket_listing_fields_flatness_wildcard) rr0r1r2r3r4r5r6r9r:r;r<r>s rr_NameExpansionIterator.__init__csB%JK OM2$#'--"<"<">DMM"8 O*.7L#vh-'<  &*#6Drc #"# URGHn[U5nUR5(abUR5(dUR 5(a8URR (a [ S5e[USSSUSS9v MSnUR5(aOUR5(a:UR(d)[URU5RS/S95nO^[URU5RURSS95nUR5(aUR5(aSnUR!5nURR =(d UnUR"URnUR(a[%XUUR5nO ['U5n[U5nUR)5(a6UR*(a[ [,U-5e[ [,U-5e[[3XRUR4UR6UR855n U R!5nURR =(d Un U GH5upU=(d U nU R;5(a&[UU UUU R<U R>S9v ML[U R@5nUR5(aU <[BRD<U<3nOURGUS 9n[URU5RIURS 95nU=(d UR!5nURR =(d Un UH'n [UU USU R<U R>S9v M) GM8 GM g![ a'n U [.R0"5S 4v Sn A GNSn A ff=f7f) a Iterates over all source URLs passed to the iterator. For each src url, expands wildcards, object-less bucket names, subdir bucket names, and directory names, and generates a flat listing of all the matching objects/files. You should instantiate this object using the static factory function NameExpansionIterator, because consumers of this iterator need the PluralityCheckableIterator wrapper built by that function. Yields: gslib.name_expansion.NameExpansionResult. Raises: CommandException: if errors encountered. zPMultiple URL strings are not supported with streaming ("-") URLs or named pipes.FN)rrrrrrid) bucket_fieldsT)r>expand_top_level_bucketswildcard_suffixr>)%r4r IsFileUrlIsStreamIsFifor8rr IsCloudUrlIsBucketr5r WildcardIterator IterBucketsIterAllr>r7r?_ImplicitBucketSubdirIterator_NonContainerTuplifyIteratorIsEmptyr<r sysexc_info_OmitNonRecursiveIteratorr0r9r2IsObject storage_url root_object url_stringossepCreatePrefixUrl IterObjects)rurl_strrXsrc_names_bucketpost_step1_itersrc_url_expands_to_multirsubdir_exp_wildcardpost_step2_iterepost_step3_iterrrblrsrc_names_container expanded_urlurl_to_iteratewc_iters r__iter___NameExpansionIterator.__iter__s"==(1k    ! !    ! ![%7%7%9%9 == & & "MN N![:?DI277B26 8 8    " "{';';'='=&& 5  ! !' * 6 6dV 6 LN 5  ! !' * 2 2&*&@&@)- 3 /0  ! ! # # (<(<(>(>! !0!=!=!?+/==+F+F,D+C(!33D4L4LM ! !7 #6  & &(7G2?Co  " "  ! ! )"#9G#CD D !!7'!AB B3 #O5M5M$($5$5$($?$? NOo "1!=!=!?!%!>#!,&=/1#&??!oo/ /.cnn=,  # # % %),bff6IJN*99 3:5N/##N3??(,(B(B@DE''?'=&-&:&:&< #%)]]%@%@&>%= "c%#.(?5 $%(__ #1 1I%4s!D" )cllnQ'( ( ( )s1HPOGP P %PPP  Pc [RRUURURUR UR URS9$)aHelper to instantiate gslib.WildcardIterator. Args are same as gslib.WildcardIterator interface, but this method fills in most of the values from instance state. Args: url_string: URL string naming wildcard objects to iterate. Returns: Wildcard iterator over URL string. )r6r:r;r2)gslibwildcard_iteratorCreateWildcardIteratorr3r6r:r;r2)rrZs rrN'_NameExpansionIterator.WildcardIteratorQsL  " " 9 9 &&??,,{{ : r) r?r6r>r9r0r<r1r3r;r2r:r5r4FTNFFN) r#r$r%r&r'rrlrNr(r)rrr+r+]s."&*$!&%)T7lV1prr+c2\rSrSrSrSSjrSrSrg)SeekAheadNameExpansionIteratorifzCreates and wraps a _NameExpansionIterator and yields SeekAheadResults. Unlike the NameExpansionIterator, which can make API calls upon __init__ to check for plurality, this iterator does no work until the first iteration occurs. Nc US;=(a U (+UlUR(aS/OSn [UU[R"S5U[ U5UUUUU SU S9 Ulg)z5Initializes a _NameExpansionIterator with the inputs.)cpmvrewritesizeNdummyTr6r9r:r;r<r>)count_data_bytesr+logging getLoggerr name_expansion_iterator) rr0r1r3r4r5r6r9r:r;file_size_will_changer>s rr'SeekAheadNameExpansionIterator.__init__nsu *-DD7!66  )-(=(=VH4#9 '""8,!5'3 $5D rc#0# URHnUR(abUR(aQ[R"[ R UR5nUR=(d Sn[US9v Mv[5v M g7f)Nr) data_bytes) rr}rr JsonToMessageapitools_messagesObjectrzr )rname_expansion_resultiterated_metadata iterated_sizes rrl'SeekAheadNameExpansionIterator.__iter__sp!%!=!=  #8#H#H$22  $ $&;&K&KM)..3! 77">sBB)r}r)FTNFFr#r$r%r&r'rrlr(r)rrrurufs#"&*$%*#5J rruc [U5n[UUUUUUUUUU U U S9 n [U 5n U R5(a[[5eU $)a8Static factory function for instantiating _NameExpansionIterator. This wraps the resulting iterator in a PluralityCheckableIterator and checks that it is non-empty. Also, allows url_strs to be either an array or an iterator. Args: command_name: name of command being run. debug: Debug level to pass to underlying iterators (range 0..3). logger: logging.Logger object. gsutil_api: Cloud storage interface. Settable for testing/mocking. url_strs: Iterable URL strings needing expansion. recursion_requested: True if -r specified on command-line. If so, listings will be flattened so mapped-to results contain objects spanning subdirectories. all_versions: Bool indicating whether to iterate over all object versions. cmd_supports_recursion: Bool indicating whether this command supports a '-r' flag. Useful for printing helpful error messages. project_id: Project id to use for the current command. ignore_symlinks: If True, ignore symlinks during iteration. continue_on_error: If true, yield no-match exceptions encountered during iteration instead of raising them. bucket_listing_fields: Iterable fields to include in expanded results. Ex. ['name', 'acl']. Underyling iterator is responsible for converting these to list-style format ['items/name', 'items/acl']. If this is None, only the object name is included in the result. Raises: CommandException if underlying iterator is empty. Returns: Name expansion iterator instance. For example semantics, see comments in NameExpansionIterator.__init__. r|)r r+rSrr ) r0r1r2r3r4r5r6r9r:r;r<r>rs rNameExpansionIteratorrsm^( 1(2  3%)1 377NO$$&& 2 33  rc$\rSrSrSrSrSrSrg)rRizIterator that produces the tuple (False, blr) for each iterated value. Used for cases where blr_iter iterates over a set of BucketListingRefs known not to name containers. cXlg)zHInstantiates iterator. Args: blr_iter: iterator of BucketListingRef. Nblr_iter)rrs rr%_NonContainerTuplifyIterator.__init__s Mrc#># URH nSU4v M g7f)NFr)rrgs rrl%_NonContainerTuplifyIterator.__iter__s}} CLsrNrr)rrrRrRs rrRc$\rSrSrSrSrSrSrg)rViaIterator wrapper for that omits certain values for non-recursive requests. This iterates over tuples of (names_container, BucketListingReference) and omits directories, prefixes, and buckets from non-recurisve requests so that we can properly calculate whether the source URL expands to multiple URLs. For example, if we have a bucket containing two objects: bucket/foo and bucket/foo/bar and we do a non-recursive iteration, only bucket/foo will be yielded. c@XlX lX0lX@lXPlg)axInstanties the iterator. Args: tuple_iter: Iterator over names_container, BucketListingReference from step 2 in the NameExpansionIterator recursion_requested: If false, omit buckets, dirs, and subdirs command_name: Command name for user messages cmd_supports_recursion: Command recursion support for user messages logger: Log object for user messages N) tuple_iterr5r0r9r2)rrr5r0r9r2s rr"_OmitNonRecursiveIterator.__init__s !O2$"8Krc## URHupUR(dUR5(d[UR5nUR 5(aSnO UR nUR(a3URRSXBRUR5 MURRSXBR5 MX4v M g7f)N directoryz-Omitting %s "%s". (Did you mean to do %s -r?)zOmitting %s "%s".) rr5rWrrZrI type_namer9r2infor0)rrrgridescs rrl"_OmitNonRecursiveIterator.__iter__s"&//  % %cllnn,CNN;  ! ! # #$$  & & ++  J1B1B D ++  .nn E$$#2sC'C))r9r0r2r5rNrr)rrrVrVs $%rrVc$\rSrSrSrSrSrSrg)rQi'aIterator wrapper that performs implicit bucket subdir expansion. Each iteration yields tuple (names_container, expanded BucketListingRefs) where names_container is true if URL names a directory, bucket, or bucket subdir. For example, iterating over [BucketListingRef("gs://abc")] would expand to: [BucketListingRef("gs://abc/o1"), BucketListingRef("gs://abc/o2")] if those subdir objects exist, and [BucketListingRef("gs://abc") otherwise. c4X lXlX0lX@lg)aInstantiates the iterator. Args: name_exp_instance: calling instance of NameExpansion class. blr_iter: iterator over BucketListingRef prefixes and objects. subdir_exp_wildcard: wildcard for expanding subdirectories; expected values are ** if the mapped-to results should contain objects spanning subdirectories, or * if only one level should be listed. bucket_listing_fields: Fields requested in enumerated results. N)rname_exp_instancercr>)rrrrcr>s rr&_ImplicitBucketSubdirIterator.__init__3sM.2!6rc## URHnUR5(a[UR5R UR S9n[ URRU5RURS95nUR5(dUH nSU4v M MSU4v MUR5(aSU4v M[SU-5e g7f)NrFrHTFz7_ImplicitBucketSubdirIterator got a bucket reference %s)rIsPrefixrrZr]rcr rrNrPr>rSrWr)rrg prefix_urlimplicit_subdir_iteratorexp_blrs rrl&_ImplicitBucketSubdirIterator.__iter__Es}} )#..9II 44J6 #=  " " 3 3J ? G G&*&@&@ H B$C (//111g/ !2   <<>>cl E KM M%sC(C*)rr>rrcNrr)rrrQrQ's 7$MrrQc\rSrSrSrSrSrg)CopyObjectInfoi\zARepresents the information needed for copying a single object. cURUlURUlURUlURUlURUlUR UlX lX0lg)a-Instantiates the object info from name expansion result and destination. Args: name_expansion_result: StorageUrl that was being expanded. exp_dst_url: StorageUrl of the destination. have_existing_dst_container: Whether exp_url names an existing directory, bucket, or bucket subdirectory. N)rrrrrr exp_dst_urlhave_existing_dst_container)rrrrs rrCopyObjectInfo.__init__`sg4FFD#8#P#PD ?? *0@@D 5 J JD0@@D"'B$r)rrrrrrrrN)r#r$r%r&r'rr(r)rrrr\s CrrDestinationInforr%NameExpansionIteratorDestinationTuplename_expansion_iter destinationc*\rSrSrSrSrSrSrSrg)CopyObjectsIteratoriaIterator wrapper for copying objects and keeping track of source URL types. This is used in the cp command for copying from multiple source to multiple destinations. It takes a list of NameExpansionIteratorDestinationTuple. It wraps them and return CopyObjectInfo objects that wraps NameExpansionResult with the destination. It's used also for collecting analytics PerformanceSummary info, because there may be multiple source URLs and we want to know if any of them are file URLs, if any of them are cloud URLs, if any of them require daisy chain operations, and if any use different providers. The source URL type information will be aggregated at the end of _SequentialApply or _ParallelApply. cX lSUlSUl/UlXl[ UR5nUR UlURUl g)aInstantiates the iterator. Args: name_expansion_dest_iter: NameExpansionIteratorDestinationTuple iterator. is_daisy_chain: The -D option in cp might have already been specified, in which case we do not need to check again for daisy chain operations. FN) is_daisy_chain has_file_src has_cloud_srcprovider_typesname_expansion_dest_iternextrcurrent_expansion_iterrcurrent_destination)rrrname_expansion_dest_tuples rrCopyObjectsIterator.__init__sU)DDD$<! $T%B%B C";"O"OD8DDDrcU$)Nr)r s rrlCopyObjectsIterator.__iter__s Krc [UR5n[UUR RUR R5nUR(d&URR5(aSUl UR(d&URR5(aSUlUR RR5(a!UR RR nOSnUR"(dCUb@URR5(a!URR U:waSUlURR UR$;a/UR$R'URR 5 U$![aJ [UR5nURUlUR UlUR5s$f=f)z@Keeps track of URL types as the command iterates over arguments.TN)rr StopIterationrrrr__next__rrrrrrIrrLschemerrappend)rrreltdst_url_schemes rrCopyObjectsIterator.__next__s"4#>#>? .11==11MM OC   !7!7!A!A!C!Cd   #"8"8"C"C"E"Ed ++6688//;;BBnn   N$> ))++ %%7 d $$D,?,??   !7!7!>!>? JA "&t'D'D"E # 7 7 !!:!F!Fd ]]_ sF//AHH)rrrrrrrN) r#r$r%r&r'rrlrr(r)rrrrs E$$rrrs))r' __future__rrrr collectionsr~r[rTsixapitools.base.pyrrogslib.exceptionrr r "gslib.plurality_checkable_iteratorr gslib.seek_ahead_threadr "gslib.third_party.storage_apitoolsr rgslib.wildcard_iteratorrobjectrr+rurrRrVrQr namedtuplerrIteratorrr)rrrs'%' % ,32I3W8+,&+,\FVFR5 V5 |(-15%)*/,104@!F6(/%/%d2MF2MjCVC6((  & )4(>(>+.)%G#,,Gr