rSrSSKJr SSKJr SSKJr SSKr/SQrSr"SS \5r "S S \5r S r g) z!A utility for tokenizing strings.)absolute_import)division)unicode_literalsN)Literal SeparatorTokenize\c\rSrSrSrg)rN__name__ __module__ __qualname____firstlineno____static_attributes__r )lib/googlecloudsdk/core/util/tokenizer.pyrrrrc\rSrSrSrg)r!r Nr r rrrr!rrrc/n[R"5n[R"U5nURS5nU(dOU[:XaWURS5nU(aUR U5 MO[ SR UR5S- U55eXQ;aXUR[UR555 UR[U55 [R"5nOUR U5 MUR[UR555 U$)aTokenizes the given string based on a list of separator strings. This is similar to splitting the string based on separators, except that this function retains the separators. The separators are wrapped in Separator objects and everything else is wrapped in Literal objects. For example, Tokenize('a:b,c:d', [':', ',']) returns [Literal('a'), Separator(':'), Literal('b'), Separator(','), Literal('c'), Separator(':'), Literal('d')]. Args: string: str, The string to partition. separators: [str], A list of strings on which to partition. Raises: ValueError: If an unterminated escape sequence is at the end of the input. Returns: [tuple], A list of strings which can be of types Literal or Separator. z)illegal escape sequence at index {0}: {1}) ioStringIOread _ESCAPE_CHARwrite ValueErrorformattellappendrgetvaluer)string separatorstokenscurrbufcs rrr%s2 & $ F#   A  l  ((1+a  1 DKK HHJNF$% %  mmGDMMO,- mmIaL! [[]d jjm# & -- () -r) __doc__ __future__rrrr__all__rstrrrrr rrr.s= (&' . c1r