g$SrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKJ r SSK J r SSK Jr S r"S S \5r"S S \5r"SS\5rSSjrg)aLogic for streaming logs. We implement streaming with two important implementation details. First, we use polling because Cloud Logging does not support streaming. Second, we have no guarantee that we will receive logs in chronological order. This is because clients can emit logs with chosen timestamps. However, we want to generate an ordered list of logs. So, we choose to not fetch logs in the most recent N seconds. We also decided to skip logs that are returned too late (their timestamp is more than N seconds old). )absolute_import)division)unicode_literalsN)common)log)timesz1970-01-01T01:00:00.000000000Zc4\rSrSrSrS SjrSrSrSrSr g) LogPosition+zTracks a position in the log. Log messages are sorted by timestamp. Within a given timestamp, logs will be returned in order of insert_id. NcHU=(d [UlSUlSUlg)NF)_UNIX_ZERO_TIMESTAMP timestamp insert_idneed_insert_id_in_lb_filter)selfrs -lib/googlecloudsdk/command_lib/logs/stream.py__init__LogPosition.__init__2s6"6DNDN',D$cXR:agXR:XaX R:aX lSUlggSUlX lXlg)aUpdate the log position based on new log entry data. Args: timestamp: the timestamp of the message we just read, as an RFC3339 string. insert_id: the insert id of the message we just read. Returns: True if the position was updated; False if not. FT)rrr)rrrs rUpdateLogPosition.Update7sO>>!  nn $ ^^ #"+/( */d& n n rcUR(a1SRURURUR5$SRUR5$)zThe log message filter which keeps out messages which are too old. Returns: The lower bound filter text that we should use. z9((timestamp="{0}" AND insertId>"{1}") OR timestamp>"{2}")ztimestamp>="{0}")rformatrr)rs rGetFilterLowerBoundLogPosition.GetFilterLowerBoundTsJ '' H O O ..$..$..:: & &t~~ 66rc[R"UR5RnUR US9nU[ R "SS9- nSR[R"US55$)zThe log message filter which keeps out messages which are too new. Args: now: The current time, as a datetime object. Returns: The upper bound filter text that we should use. )tzinfo)secondsztimestamp<"{0}"z%Y-%m-%dT%H:%M:%S.%6f%Ez) r ParseDateTimerrreplacedatetime timedeltarFormatDateTime)rnowr upper_bounds rGetFilterUpperBoundLogPosition.GetFilterUpperBoundasi  0 7 7F ++V+ $C**155K  # # [*DE GGr)rrrN) __name__ __module__ __qualname____firstlineno____doc__rrrr)__static_attributes__rrr r +s - : 7Grr c$\rSrSrSrSrSrSrg)_TaskIntervalTimerraTimer to facilitate performing multiple tasks at different intervals. Here's an overview of how the caller sees this class: >>> timer = _TaskIntervalTimer({'a': 5, 'b': 10, 'c': 3}) >>> timer.Wait() # sleeps 3 seconds, total time elapsed 3 ['c'] >>> timer.Wait() # sleeps 2 seconds, total time elapsed 5 ['a'] >>> timer.Wait() # sleeps 1 second, total time elapsed 6 ['c'] >>> timer.Wait() # sleeps 3 seconds, total time elapsed 9 ['c'] >>> timer.Wait() # sleeps 1 second, total time elapsed 10 ['a', 'c'] And here's how it might be used in practice: >>> timer = _TaskIntervalTimer({'foo': 1, 'bar': 10, 'baz': 3}) >>> while True: ... tasks = timer.Wait() ... if 'foo' in tasks: ... foo() ... if 'bar' in tasks: ... bar() ... if 'baz' in tasks: ... do_baz() Attributes: task_intervals: dict (hashable to int), mapping from some representation of a task to to the interval (in seconds) at which the task should be performed cNXlURR5Ulgr+)task_intervalscopy_time_remaining)rr7s rr_TaskIntervalTimer.__init__s(..335Drcd[URR55n[R"U5 [ 5nURH\nURU==U-ss'URUS:XdM/UR UURU'URU5 M^ U$)z`Wait for the next task(s) and return them. Returns: set, the tasks which should be performed r)minr9valuestimesleepsetr7add)r sleep_timetaskstasks rWait_TaskIntervalTimer.Waits T))0023JJJz EE$$ 4 J.  d #q (%)%8%8%>T" $ % Lr)r9r7N)r,r-r.r/r0rrEr1r2rrr4r4rs!F6rr4ch\rSrSrSr"SS\R 5rSrSSSSS4S jr S r S r S r g) LogFetcherzA class which fetches job logs.c\rSrSrSrSrSrg)LogFetcher._Tasksr2N)r,r-r.r/POLLCHECK_CONTINUEr1r2rr_TasksrKs DNrrQiN cg)NTr2)xs rLogFetcher.strcU=(d /UlX lU=(d UUlX0l[ X5n[ R "SRU55 [US9Ul g)aInitializes the LogFetcher. Args: filters: list of string filters used in the API call. polling_interval: amount of time to sleep between each poll. continue_func: One-arg function that takes in the number of empty polls and outputs a boolean to decide if we should keep polling or not. If not given, keep polling indefinitely. continue_interval: int, how often to check whether the job is complete using continue_function. If not provided, defaults to the same as the polling interval. num_prev_entries: int, if provided, will first perform a decending query to set a lower bound timestamp equal to that of the n:th entry. zstart timestamp: {})rN) base_filterspolling_intervalcontinue_intervalshould_continue_GetTailStartingTimestamprdebugrr log_position)rfiltersrY continue_funcrZnum_prev_entriesstart_timestamps rrLogFetcher.__init__sY" 2D,.B2BD(/JOII#**?;<#o>Drc[RR5nURR5nURR U5nUR X#/-n[ R"SRU5SURS9nUVs/sH<nURRURUR5(dM:UPM> sn$s snf)zRetrieves a batch of logs. After we fetch the logs, we ensure that none of the logs have been seen before. Along the way, we update the most recent timestamp. Returns: A list of valid log entries.  AND ASC log_filterorder_bylimit) r$utcnowr^rr)rXlogging_common FetchLogsjoinLOG_BATCH_SIZErrinsertId)rrk lower_filter upper_filter new_filterentriesentrys rGetLogsLogFetcher.GetLogss   % % 'F$$88:L$$88@L""l%AAJ&&<< +!!#G ' Gwe    $ $U__enn E w GG Gs 9CCc## [URRURURRUR 05nSnURRURR/nURRU;a,UR 5nU(aSnUHnUv M OUS- nURRU;aURU5nU(dgUR5nM7f)zPolls Get API for more logs. We poll so long as our continue function, which considers the number of periods without new logs, returns True. Yields: A single log entry. rrMN) r4rQrOrYrPrZrvr[rE)rtimer empty_pollsrClogs log_entryr[s r YieldLogsLogFetcher.YieldLogss  $// ""D$:$:  EK [[  t{{99 :E    U "||~ +iO   +  # #u ,..{; jjle sDD)rXrZr^rYr[) r,r-r.r/r0enumEnumrQrorrvr}r1r2rrrHrHs9'tyy.!B+t $?2G(rrHcU(dg[[R"SRU5SUS95n[ U5U:ag[U5SR $)a|Returns the starting timestamp to start streaming logs from. Args: filters: [str], existing filters, should not contain timestamp constraints. offset: int, how many entries ago we should pick the starting timestamp. If not provided, unix time zero will be returned. Returns: str, A timestamp that can be used as lower bound or None if no lower bound is necessary. NreDESCrg)listrlrmrnlenr)r_offsetrts rr\r\sX   ))W\\'5J39068 9' \F  gr  $ $$rr+)r0 __future__rrrr$r>rgooglecloudsdk.api_lib.loggingrrlgooglecloudsdk.corergooglecloudsdk.core.utilrrobjectr r4rHr\r2rrrsa '' C#*8DG&DGN77tSSl%r