G>SrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK J r SSK J r SS K Jr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJr SS/r/SQrS Sjr Sr!Sr"Sr#Sr$Sr%S!Sjr&Sr'"SS\(5r)g)"zCUtility functions and class for listing commands such as ls and du.)absolute_import)print_function)division)unicode_literalsN)EncryptionException)CommandException)PluralityCheckableIterator)GenerationFromUrlAndString)S3_ACL_MARKER_GUID)S3_DELETE_MARKER_GUID)S3_MARKER_GUIDS)UTF8) IS_WINDOWS)AclTranslation) text_util)StorageUrlFromStringmd5Hashcrc32c)acl cacheControlcomponentCountcontentDispositioncontentEncodingcontentLanguage contentType customTime kmsKeyNamecustomerEncryptionetageventBasedHold generationmetadatametagenerationretentionExpirationTimesize storageClass temporaryHold timeCreated timeDeletedtimeStorageClassUpdatedupdatedc\SRSU-S-U-S-RS5U5$)aReturns a string with a vertically aligned label and value. Labels of the same indentation level will start at the same column. Values will all start at the same column (unless the combined left-indent and label length is excessively long). If a value spans multiple lines, indentation will only be applied to the first line. Example output from several calls: Label1: Value (default indent of 1 was used) Sublabel1: Value (used indent of 2 here) Label2: Value Args: label: The label to print in the first column. value: The value to print in the second column. indent: (4 * indent) spaces will be placed before the label. Returns: A string with a vertically aligned label and value. z{}{} :)formatljust)labelvalueindents (platform/gsutil/gslib/utils/ls_helper.pyMakeMetadataLiner7Fs3( v)U2S8??CU KKcg)zDefault function for printing headers for buckets. Header is printed prior to listing the contents of the bucket. Args: bucket_listing_ref: BucketListingRef of type BUCKET. Nbucket_listing_refs r6PrintBucketHeaderr=]sr8cD[R"UR5 g)z{Default function for printing buckets or prefixes. Args: bucket_listing_ref: BucketListingRef of type BUCKET or PREFIX. N)r print_to_fd url_stringr;s r6PrintDirrAhs  *556r8cg)zOff-by-default function for printing buckets or prefix size summaries. Args: num_bytes: Number of bytes contained in the directory. bucket_listing_ref: BucketListingRef of type BUCKET or PREFIX. Nr:) num_bytesr<s r6PrintDirSummaryrDrsr8cb[R"SRUR55 g)zDefault function for printing headers for prefixes. Header is printed prior to listing the contents of the prefix. Args: bucket_listing_ref: BucketListingRef of type PREFIX. {}:N)rr?r1r@r;s r6PrintDirHeaderrG|s"  %7%B%BCDr8c.[R"5 g)zXyR8'MD U(a[R"[SS55 UHYn SRU R85n SRU R>5n [XSS9n [R"U 5 M[ UR@(aURB(d [R"[SS55 URD(d [R"[SS55 [R"[S UR@RF55 [R"[S!UR@RH55 URB(a*[R"[SURB55 URD(a*[R"[SURD55 [R"[S"URJRMS#555 URN(a5[QX4RN5n[R"[S$U55 URR(a*[R"[S%URR55 U(aURT(aA[R"[S&[VRX"URT555 Xe4$[ZU;a*[R"[S&U[Z55 Xe4$[R"[S&S'55 Xe4$)(aPrint full info for given object (like what displays for gsutil ls -L). Args: bucket_listing_ref: BucketListingRef being listed. Must have ref_type OBJECT and a populated root_object with the desired fields. incl_acl: True if ACL info should be output. Returns: Tuple (number of objects, object_length) Raises: Exception: if calling bug encountered. rzrFz Creation timez%a, %d %b %Y %H:%M:%S GMTz Update timezStorage class update timez Storage classzTemporary HoldEnabledzEvent-Based HoldzRetention ExpirationzKMS keyz Cache-ControlzContent-DispositionzContent-EncodingzContent-LanguagezContent-Lengthz Content-TypezComponent-Countz Custom-TimezNoncurrent timeMetadataz{})r5z Hash (crc32c) encryptedz Hash (md5)zEncryption algorithmzEncryption key SHA256ETagz"' GenerationMetagenerationACLz[]).r@r root_objectr"r additionalPropertiesr%rr?r1r(r7strftimer+r*r&r'r r$rrrrrrrrr)keyr appendr4rrrencryptionAlgorithm keySha256rstripr!r r#rrJsonFromMessager )r<incl_aclurl_str storage_urlobjrCnum_objs marker_propsnon_marker_propsadd_propapap_keyap_valuemeta_data_linegeneration_strs r6PrintFullInfoAboutObjectrks  ) )'$W-+&&# lls||@@@IH GIH  W-.__ 112MN PQ [[ --.IJ LM !! !!S__4  '  ' ' 0 01L M OP  *?C,>?A^^ *=#..IJ__ *112MN PQ,\\cll77LL55 _ ,)%-^^ \\" 6 ,Z<= "RVV$;;rxx()&1En- !   ::,_kJK ;;,\;GH ///CC EF0//99 ;< ZZ *?CJJGH[[ *<EF (1FGH^^/ ^^LN *<HI *+;+.+=+=?@  ww 5."@"@"I JL   | + 5,/A"B CE  ,UD9:  r8c[R"UR5 g![a&n[(aUR S:XdeSnAgSnAff=f)zDefault printing function for objects. Args: bucket_listing_ref: BucketListingRef of type OBJECT. Returns: (num_objects, num_bytes). rN)rKr)rr?r@IOErrorrerrno)r<es r6 PrintObjectrpsO  ,778    J177a< (   s # AAAc N\rSrSrSr\\\\\ \ SSSSS4 Sjr Sr S S jr S rS rg) LsHelperi$zHelper class for ls and du.FN)nameTcXlX lX0lX@lXPlX`lXplXlXlXl Xl Xl Xl g)a+Initializes the helper class to prepare for listing. Args: iterator_func: Function for instantiating iterator. Inputs- url_string- Url string to iterate on. May include wildcards. all_versions=False- If true, iterate over all object versions. logger: Logger for outputting warnings / errors. print_object_func: Function for printing objects. print_dir_func: Function for printing buckets/prefixes. print_dir_header_func: Function for printing header line for buckets or prefixes. print_bucket_header_func: Function for printing header line for buckets or prefixes. print_dir_summary_func: Function for printing size summaries about buckets/prefixes. print_newline_func: Function for printing new lines between dirs. all_versions: If true, list all object versions. should_recurse: If true, recursively listing buckets/prefixes. exclude_patterns: Patterns to exclude when listing. fields: Fields to request from bucket listings; this should include all fields that need to be populated in objects so they can be listed. Can be set to None to retrieve all object fields. Defaults to short listing fields. list_subdir_contents: If true, return the directory and any contents, otherwise return only the directory itself. N) _iterator_funclogger_print_object_func_print_dir_func_print_dir_header_func_print_bucket_header_func_print_dir_summary_func_print_newline_func all_versionsshould_recurseexclude_patternsbucket_listing_fieldslist_subdir_contents)self iterator_funcrvprint_object_funcprint_dir_funcprint_dir_header_funcprint_bucket_header_funcprint_dir_summary_funcprint_newline_funcr}r~rfieldsrs r6__init__LsHelper.__init__'sRX(K/)"7%="#9 1$(,!' 4r8c 0SnSnSnSnUR5(dUR(a@UR5(aURU5 URURSS9$UR 5(a URnOUR 5n[URX`RS9RSURS95nUR5nUR5 UGH.n UR!U 5(aMU R#5(aSn UR%U 5upSnOU R'5(aU(aUR)5 OSnU(a#UR*(aUR-U 5 OU(aUR*(dSn[/U R5R UR*(aS OSS9n URU 5upn UR1X5 O [3S 5eX+- nX:- nXL- nGM1 X2U4$![aV [URUR SS9URS9RS[S95nUR5nGNf=f) zIterates over the given URL and calls print functions. Args: url: StorageUrl to iterate over. Returns: (num_objects, num_bytes) total number of objects and bytes iterated. rF)print_initial_newliner}Texpand_top_level_bucketsrNwildcard_suffix*z?Sub-level iterator returned a CsBucketListingRef of type Bucket)IsBucketr~rz_RecurseExpandUrlAndPrintr@ HasGenerationCreatePrefixUrlr rur}IterAllr HasPlurality PeekExceptionrUNENCRYPTED_FULL_LISTING_FIELDS_MatchesExcludedPatternIsObjectrwIsPrefixr|rryrr{r)rurl num_objectsnum_dirsrC print_newline iteration_urltop_level_iterator pluralityblrndnonbexpansion_url_strs r6ExpandUrlAndPrintLsHelper.ExpandUrlAndPrintasKHIM ||~~,,  &&s+  + +CNNBG,II     ++- 5   *;*;  ==DW+/(,(B(B>E>DE %113i 6((*$#  ' ' , ,  <<>>"**3/&"- \\^^   $ $ & M 444  ' ' ,!:!:!M2nn-o)-)B)B#.N 556GH*""  & &r /!OQ Q  7$8I --Q! 68   ##D#9!..  007-1*I181KL '335  6s,H55AJJcSnSnSnURSU-URS9RSURS9GHnUR U5(aMUR 5(aSnUR U5upOUR5(aUR(asU(aUR5 OSnURU5 [UR5RSS9n URU 5upxn URX5 O"Supxn UR!U5 O [#S 5eXG- nX8- nXY- nGM XCU4$) akIterates over the given URL string and calls print functions. Args: url_str: String describing StorageUrl to iterate over. Must be of depth one or higher. print_initial_newline: If true, print a newline before recursively expanded prefixes. Returns: (num_objects, num_bytes) total number of objects and bytes iterated. rz%srTrrr)rKrrz=Sub-level iterator returned a bucketListingRef of type Bucket)rur}rrrrrwrr~r|ryrr@rrr{rxr) rr_rrrrCrrrrrs r6r"LsHelper._RecurseExpandUrlAndPrintsbKHI"" wT%6%6#88?%)"&"<"<9@9>>  % %c * *  ((-B <<>>    "  $ $ &$( !  % %c *2nn-ocoB 556GH*""  & &r /*""   s # KM Mnhkoi?>B ) ++r8cUR(ae[R"UR5nURH5n[R"U[R"U55(dM5 g g)zChecks bucket listing reference against patterns to exclude. Args: blr: BucketListingRef to check. Returns: True if reference matches a pattern and should be excluded. TF)rsix ensure_strr@fnmatch)rrtomatchpatterns r6r LsHelper._MatchesExcludedPatternsQ s~~.g**' ??7CNN7$; < <+ r8) rurzrxryr{r|rwr}rrrrvr~T)__name__ __module__ __qualname____firstlineno____doc__rprArGr=rDrIrrrr__static_attributes__r:r8r6rrrr$s@# "-&%3(9&5".!# $$(85tL.\0,dr8rr)rKr)*r __future__rrrrrsysrgslib.cloud_apirgslib.exceptionr"gslib.plurality_checkable_iteratorr gslib.storage_urlr gslib.utils.constantsr r r rgslib.utils.system_utilrgslib.utils.translation_helperr gslib.utilsrgslib.wildcard_iteratorrENCRYPTED_FIELDSrr7r=rArDrGrIrkrpobjectrrr:r8r6rsJ&%' /,I8471&.9!8 #6L.7E AH&KvKr8