SrSSKJr SSKJr SSKJr SSKJr SSKJr SSKrSSK r SSK r SSK J r SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJr SSKJ r SSKJ!r! \RDr#"SS\$5r%S(Sjr&"SS\$5r'"SS\'5r("S S!\'5r)"S"S#\$5r*"S$S%\$5r+"S&S'\ RX5r-g))zFBase classes for gsutil UI controller, UIThread and MainThreadUIQueue.)absolute_import)print_function)division)unicode_literals)dequeN)queue)LogPerformanceSummaryParams)LogRetryableError) FileMessage) FinalMessage)MetadataMessage)PerformanceSummaryMessage)ProducerThreadMessage)ProgressMessage)RetryableErrorMessage)SeekAheadMessage) StatusMessage)parallelism_framework_util) DecimalShort)HumanReadableWithDecimalPlaces)MakeHumanReadable) PrettyTimec(\rSrSrSrSrSrSrSrSr g) EstimationSource2zenum for total size source.N) __name__ __module__ __qualname____firstlineno____doc__PRODUCER_THREAD_FINALSEEK_AHEAD_THREADPRODUCER_THREAD_ESTIMATEINDIVIDUAL_MESSAGES__static_attributes__r &platform/gsutil/gslib/ui_controller.pyrr2s%#r+rc[UUS9nUR5n[[[ US555S:a[ US5U- S- nUSSUUS'SSR U5-$)zAdjusts proper width for printing num_bytes in readable format. Args: num_bytes: The number of bytes we must display. decimal_places: The standard number of decimal places. Returns: String of fixed width representing num_bytes. )decimal_placesrirNz%9s )rsplitintroundfloatlenjoin) num_bytesr.human_readable number_formatlast_characters r,BytesToFixedWidthStringr:Fs2)AOQ. &&(-u]1%& '(D0q)*^;a?N$Q'8M! #((=) **r+c\rSrSrSr"SS\5rSSjrSrSr S r S r S r S r S rSr\R"4SjrSr\R"4SjrSrg)StatusMessageManagerZzGeneral manager for common functions shared by data and metadata managers. This subclass has the responsibility of having a common constructor and the same handler for SeekAheadMessages and ProducerThreadMessages. c\rSrSrSrSrSrg)+StatusMessageManager._ThroughputInformationazClass that contains all information needed for throughput calculation. This _ThroughputInformation is used to track progress and time at several points of our operation. cXlX lg)zConstructor of _ThroughputInformation. Args: progress: The current progress, in bytes/second or objects/second. report_time: Float representing when progress was reported (seconds since Epoch). Nprogresstime)selfrC report_times r,__init__4StatusMessageManager._ThroughputInformation.__init__hsmir+rBNr!r"r#r$r%rGr*r r+r,_ThroughputInformationr?as  r+rJNc XlX lX0lX@lXPlX`lXplXl[RUl [RUl SUl SUl UR (a UR O[R"5UlURUlURUlSUl['5UlSUl/SQUlSUlSUlSUl SUlSUlg)aInstantiates a StatusMessageManager. Args: update_message_period: Minimum period for refreshing and displaying new information. A non-positive value will ignore any time restrictions imposed by this field, but it will affect throughput and time remaining estimations. update_spinner_period: Minimum period for refreshing and displaying the spinner. A non-positive value will ignore any time restrictions imposed by this field. sliding_throughput_period: Sliding period for throughput calculation. A non-positive value will make it impossible to calculate the throughput. first_throughput_latency: Minimum waiting time before actually displaying throughput info. A non-positive value will ignore any time restrictions imposed by this field. quiet_mode: If True, do not print status messages (but still process them for analytics reporting as necessary). custom_time: If a custom start_time is desired. Used for testing. verbose: Tells whether or not the operation is on verbose mode. console_width: Width to display on console. This should not adjust the visual output, just the space padding. For proper visualization, we recommend setting this field to at least 80. rg)/-\|FN)update_message_periodupdate_spinner_periodsliding_throughput_periodfirst_throughput_latency quiet_mode custom_timeverbose console_widthrr)num_objects_sourcetotal_size_source num_objects total_sizerDrefresh_message_time start_timerefresh_spinner_time throughputr old_progresslast_progress_timespinner_char_listcurrent_spinner_indexobjects_finishedobject_report_change final_message) rErPrQrRrSrTrUrVrWs r,rGStatusMessageManager.__init__ssH"7!6%>"$<! O"L& /BBD-AADDDO%)$4$4"&!1!1:>))+ //DO $ 9 9DDODD2D!"DDD!&DDr+c4URUR$)zdReturns the current spinner character. Returns: char_to_print: Char to be printed as the spinner )rbrcrEs r, GetSpinnerStatusMessageManager.GetSpinners  ! !$"<"< ==r+cXURS-[UR5-Ulg)z&Updates the current spinner character.rN)rcr4rbris r, UpdateSpinner"StatusMessageManager.UpdateSpinners(#'#=#=#A"%d&<&<"=#>Dr+cUR(aUR[R:a&[RUlURUlUR [R:a7UR (a&[RUlUR UlgUR[R:a&[RUlURUlUR [R:a9UR (a'[RUlUR Ulggg)zeHandles a ProducerThreadMessage. Args: status_message: The ProducerThreadMessage to be processed. N) finishedrXrr&rZrYsizer[r(rEstatus_messages r,_HandleProducerThreadMessage1StatusMessageManager._HandleProducerThreadMessages  $4$J$J J"2"H"H)55  $4$J$J J   !1!G!G(--  "2"K"KK 0 I Id'33d "2"K"KK/HHd&++do  Lr+cSUR-nUR(a_US[UR5-- nUR[R :a&[R UlURUlUR[R :a&[R UlURUlUS- nUR(dURU5 gg)zHandles a SeekAheadMessage. Args: status_message: The SeekAheadMessage to be processed. stream: Stream to print messages. z,Estimated work for this command: objects: %sz, total size: %s N) rZrqrrYrr'r[rXrTwrite)rErsstreamestimate_messages r,_HandleSeekAheadMessage,StatusMessageManager._HandleSeekAheadMessagesG&223-,^-@-@ABC  #3#E#E E!1!C!C(-- "2"D"DD 0 B Bd'33d ?? ll#$ r+c*[URS9 g)zmHandles a PerformanceSummaryMessage. Args: status_message: The PerformanceSummaryMessage to be processed. ) uses_sliceN)r r~rrs r, _HandlePerformanceSummaryMessage5StatusMessageManager._HandlePerformanceSummaryMessages  >+D+DEr+c8XR- UR:$)zDecides whether enough time has passed to start tracking throughput. Args: cur_time: current time. Returns: Whether or not we should track the throughput. )r]rSrEcur_times r,ShouldTrackThroughput*StatusMessageManager.ShouldTrackThroughputs oo %)F)F FFr+cXR- UR:nURnU=(d UR=(a U$)zDecides whether or not it is time for printing a new progress. Args: cur_time: current time. Returns: Whether or not we should print the progress. )r\rPrZre)rErsufficient_time_elapsednonzero_reports r,ShouldPrintProgress(StatusMessageManager.ShouldPrintProgresssL (*C*CC"&"<"< =%%N # &  % % <,:? ,0%0FG< !!$ -!.F-0JJ=r+r<cx^\rSrSrSrS U4SjjrSrSrSr\ R4Sjr Sr S r U=r$) MetadataManagerikaManages shared state for metadata operations. This manager is specific for metadata operations. Among its main functions, it receives incoming StatusMessages, storing all necessary data about the current and past states of the system necessary to display to the UI. It also provides methods for calculating metrics such as throughput and estimated time remaining. Finally, it provides methods for displaying messages to the UI. c 6>[[U] UUUUUUUUS9 g)z`Instantiates a MetadataManager. See argument documentation in StatusMessageManager base class. rPrQrRrSrTrUrVrWN)superrrG rErPrQrRrSrTrUrVrW __class__s r,rGMetadataManager.__init__vs4 / /D/D3L2J$.%0!('46r+cUR$)zUGets the progress for a MetadataManager. Returns: The number of finished objects. )rdris r, GetProgressMetadataManager.GetProgresss   r+cpU=RS- slUR[R:a*[RUlU=RS- slSUlUR UlURUR:Xa'UR[R:XaSUl ggg)zYHandles a MetadataMessage. Args: status_message: The MetadataMessage to be processed. rTN) rdrXrr)rZrerDrar&rfrrs r,_HandleMetadataMessage&MetadataManager._HandleMetadataMessages  Q "2"F"FF 0 D Dd ! $D,11D !1!11 #3#I#IId J 2r+cSUl[U[5(aURX5 O[U[5(aUR U5 On[U[ 5(aURU5 OG[U[5(a [U5 O&[U[5(aURU5 URRURURUR 55 g)zProcesses a message from _MainThreadUIQueue or _UIThread. Args: status_message: The StatusMessage item to be processed. stream: Stream to print messages. FN)re isinstancerr{rrtr rrr rrr`appendrJrdrDrErsrys r,ProcessMessageMetadataManager.ProcessMessages!&D."233 "">: N$9 : : ''7 NO 4 4 !!.1 N$9 : :' N$= > > ++N; ##D$9$9>;N;NOQr+c URUR- nUR(aX R- nOSnUR5nUR[ R ::aS[UR5-S-[UR5-S-nURUR:XaSnO;S[S[S[UR5-UR- 55-nUS -nOS[UR5-S-nS nURUR- UR:aPS UR-nUR[ R::a UR(aS [U5-n OS n OS nS n S n U R!UUUUU S9n UR"[%U 5- n UR&(d%UR)U [+U S5S--S-5 gg)aOPrints progress and throughput/time estimation. Prints total number of objects and number of finished objects with the percentage of work done, potentially including the throughput (in objects/second) and estimated time remaining. Args: stream: Stream to print messages. Usually sys.stderr, but customizable for testing. N[rLz objects]100%3dcd% Donez%.2f objects/sETA z\{char_to_print} {objects_completed} {percentage_completed} {throughput} {time_remaining_str}) char_to_printobjects_completedpercentage_completedr_time_remaining_strrr/r)rZrdr_rjrXrr'rminr1r3r\r]rSr(rformatrWr4rTrxr) rErytotal_remainingtime_remainingrr percentagerr_r format_strrrs r, PrintProgressMetadataManager.PrintProgresss&&)>)>>O &8nnOO%M "2"D"DDd.C.C!DDsJ'(8(89:M@r+rc^\rSrSrSr"SS\5rSU4SjjrSrSr Sr S r S r S r \R4S jrS rSrU=r$) DataManageriaManages shared state for data operations. This manager is specific for data operations. Among its main functions, it receives incoming StatusMessages, storing all necessary data about the current and past states of the system necessary to display to the UI. It also provides methods for calculating metrics such as throughput and estimated time remaining. Finally, it provides methods for displaying messages to the UI. c\rSrSrSrSrSrg) DataManager._ProgressInformationizClass that contains all progress information needed for a given file. This _ProgressInformation is used as the value associated with a file_name in the dict that stores the information about all processed files. c:SUlSUl0UlXlg)zPConstructor of _ProgressInformation. Args: size: The total size of the file. rN)new_progress_sumexisting_progress_sumdictrq)rErqs r,rG)DataManager._ProgressInformation.__init__ s$ d$%d diir+)rrrrqNrIr r+r,_ProgressInformationrs  r+rc >[[U] UUUUUUUUS9 SUlSUlSUlSUl0UlSUlSUl SUl g)z\Instantiates a DataManager. See argument documentation in StatusMessageManager base class. rTrN) rrrG first_itemtotal_progress new_progressexisting_progressindividual_file_progresscomponent_totalfinished_componentsexisting_componentsrs r,rGDataManager.__init__3sw + /D/D3L2J$.%0!('46DODDD%'D!D D Dr+cUR$)zbGets the progress for a DataManager. Returns: The number of processed bytes in this operation. )rris r,rDataManager.GetProgressZs   r+cUR(Gd`UR(aKUR(d:URUlURUlURUlSUlURRnUR(a UROSUl URUR5URU'UR[R:a*[RUl U=RS- slUR [R:a4[RUlU=R"UR- slSUlgU=R&S- slURRnURUnUR(UR*-nU=R,URU- - slU=R.URU- - slURUlURU SUlUR&UR:Xa'UR[R2:XaSUlggg)zgHandles a FileMessage that describes a file. Args: status_message: the FileMessage to be processed. FrrTN)rprrUrDr\r]last_throughput_timesrc_url url_stringrqrrrXrr)rZrYr[rerdrrrrrar&rf)rErs file_name file_progresstotal_bytes_transferreds r,_HandleFileDescription"DataManager._HandleFileDescriptionbs  " " " !1!1$2$7$7!33$($=$=!!((33i3A3F3FN//An151J1J   2d##I.  $4$H$H H#3"F"F A  #3#G#G G"2!E!E >..."&d q  ((33i33I>m!.!?!?!.!D!D"E ]//2III =--0GGG . 3 3d  ' ' 2"&d   4#3#3 3  ! !%5%K%K K! L 4r+cxURnU[R:H=(d U[R:H=(d{ U[R:H=(da U[R :H=(dG U[R :H=(d- U[R:H=(d U[R:H$)zTells whether or not this FileMessage represent a file. This is needed because FileMessage is used by both files and components. Args: file_message: The FileMessage to be analyzed. Returns: Whether or not this represents a file. ) message_typer FILE_DOWNLOAD FILE_UPLOADFILE_CLOUD_COPYFILE_DAISY_COPYFILE_LOCAL_COPY FILE_REWRITE FILE_HASH)rE file_messagers r,_IsFileDataManager._IsFiles ,,L K55 5 2 K33 3 2 K77 7 2 K77 7 2 K77 7  2 K44 4  2 K11 1 3r+cUR[R:XaUR(dU=RS- slUR R nURUnURUR4nSUR4URU'U=RUR- sl U=RUR- sl U=RUR- slgUR[R:XdUR[R :XGa$UR(dU=R"S- slUR[R :XaUR R nURUnU=RUR$- sl URUR4nSUR$4URU'U=RUR$- sl U=RUR$- slggU=R&S- slUR R nURUnURUR4nXCR;aURUOSnU=RUR[)U5- - sl U=R*UR[)U5- - slUR,UlU=R0UR[)U5- - slURUS- US4URU'gg)zlHandles a FileMessage that describes a component. Args: status_message: The FileMessage to be processed. rrrrN)rr EXISTING_COMPONENTrprrrr component_numdst_urlrqrrrrCOMPONENT_TO_UPLOADCOMPONENT_TO_DOWNLOADrbytes_already_downloadedrsumrrDrarrErsrrkey last_updates r,_HandleComponentDescription'DataManager._HandleComponentDescriptions ##{'E'EE  # # !# ((33i33I>m  ) )>+A+A Bc!"N$7$7 8m))^-@-@@) ^000  3 33  & &+*I*I I  & &+*K*K K  $ $ !  & &+*K*K K$,,77)77 B-  - -55 7 - --~/E/EF#%&(O(O$P-  S !   !H!H H   N$K$K K  L   A% "**55 55i@ ++^-C-CD259K9K2K}))#.  ~22S5EEE ^003{3CCC"0"5"5&&>+>+>+.{+;,< =#6#6Q#G#.q>#3 3? Lr+c*URRnURUnURUR4nXCR ;aUR UOSnU=R US-slU=RUR US- - slU=RUR US- - slU=RUR US- - sl UR US4UR U'URUl g)zHandles a ProgressMessage that tracks progress of a file or component. Args: status_message: The ProgressMessage to be processed. rrrN) rrrrrrprocessed_bytesrrrrDrar s r,_HandleProgressMessage"DataManager._HandleProgressMessages&&11I11)99KNJJ77+a.HH-=={1~NMs,11Dr+cSUl[U[5(aURU5 O[U[5(aUR X5 O[U[ 5(aCURU5(aURU5 OURU5 [US9 On[U[5(aURU5 OG[U[5(a [U5 O&[U[5(aUR!U5 UR"R%UR'UR(UR*55 g)zProcesses a message from _MainThreadUIQueue or _UIThread. Args: status_message: The StatusMessage item to be processed. stream: Stream to print messages. Here only for SeekAheadThread F)rN)rerrrtrr{r rrr r rrrr rrr`rrJrrDrs r,rDataManager.ProcessMessages!&D."788 ''7 N$4 5 5 "">: NK 0 0 n % % ##N3 ((8!~> NO 4 4 !!.1 N$9 : :' N$= > > ++N; ##D$5$5~7J7JKMr+c URUR- nUR(aX R- nOSnUR5nUR[ R ::a5S[UR5-S-[UR5-S-nOS[UR5-S-nS[UR5<S[UR5<S3nUR[ R ::a^URUR:XaSnO;S[S[S [UR5-UR- 55-nUS -nOS nURUR - UR":aY[UR5S -n UR[ R$::a UR(aS ['U5-n OS n OS n S n Sn U R)UUUUU U S9n UR*[-U 5- n UR.(d%UR1U [3U S5S--S-5 gg)aPrints progress and throughput/time estimation. If a ProducerThreadMessage or SeekAheadMessage has been provided, it outputs the number of files completed, number of total files, the current progress, the total size, and the percentage it represents. If none of those have been provided, it only includes the number of files completed, the current progress and total size (which might be updated), with no percentage as we do not know if more files are coming. It may also include time estimation (available only given ProducerThreadMessage or SeekAheadMessage provided) and throughput. For that to happen, there is an extra condition of at least first_throughput_latency seconds having been passed since the UIController started, and that either the ProducerThread or the SeekAheadThread have estimated total number of files and total size. Args: stream: Stream to print messages. Usually sys.stderr, but customizable for testing. NrrLz files]]rrrrrrz/srzl{char_to_print} {objects_completed}{bytes_progress} {percentage_completed} {throughput} {time_remaining_str})rrbytes_progressrr_rrr/r)r[rr_rjrXrr'rrdrZr:rYrr1r3r\r]rSr(rrrWr4rTrxr)rEryrrrrrrrr_rrrrs r,rDataManager.PrintProgress"s0,oo(;(;;O &8nnOO%M "2"D"DDd.C.C!DDsJ'(8(89:L!6!6%>"$<!DL O"D&(D"!&D"&D $()BD$Id!!r+cHURRX5 URRU5(a~URRU5(a3URR X0RR 55 URR U5 X0RlURRU5(a+URRU5 X0Rl [U[5(dURR(anURR(aRUR(d@SUl[!URRS9 URR#U5 gggg)aProcesses a message, updates throughput and prints progress. Args: status_message: Message to be processed. Could be None if UIThread cannot retrieve message from status_queue. stream: stream to print messages. Usually sys.stderr, but customizable for testing. cur_time: Message time. Used to determine if it is time to refresh output, or calculate throughput. T)num_objects_transferredN)r rrrrrrr\rrr^rr rfrZr"r r)rErsryrs r,_HandleMessageUIController._HandleMessages LL7 ||''11  + +H 5 5 %%h 0H0H0JK ll  (*2ll' ||&&x00 ll'*2ll' NL 1 1T\\5O5O LL $ $T-G-G#'d !"&,,":":< ll++F3 .H $6Pr+c [U[5(dU[:XaUR(d[ UR UR URURURURURURS9UlURHnURUUURS9 M gUR (a?UR R#[%U55 UR R#S5 U(d URnUR(Gdp[U[&5(d[U[(5(aURR+U5 g[U[,5(a[/UR UR URURURURURURS9UlURHnURXBU5 M O[ UR UR URURURURURURS9UlURHnURXBU5 M URR1U5(d[U[25(d[U[45(a[ UR UR URURURURURS9UlURHnURXBU5 M OgURXU5 g)aCoordinates UI manager and calls appropriate function to handle message. Args: status_message: Message to be processed. Could be None if UIThread cannot retrieve message from status_queue. stream: Stream to print messages. Usually sys.stderr, but customizable for testing. cur_time: Message time. Used to determine if it is time to refresh output, or calculate throughput. r)rNrw)rPrQrRrSrUrVrW)rr_ZERO_TASKS_TO_DO_ARGUMENTr rrPrQrRrSrTrUrVrWr!r)rDr#rxstrrrrr rrr r)rErsryrestimation_messages r,CallUIController.Calls nm 4 4 5 5dll#"&"<"<"&"<"<&*&D&D%)%B%B((LL,,. #'"@"@    0$'9'>'>  @#A  "" !!''N(;< !!''- $$h <<< ^%5 6 6 ^%: ; ; &&--n= no 6 6'"&"<"<"&"<"<&*&D&D%)%B%B((LL,,. #'"@"@    0( C#A$"&"<"<"&"<"<&*&D&D%)%B%B((LL,,. #'"@"@    0( C#A << ( ( 8 8 ^[ 1 1 ^_ 5 5$"&"<"<"&"<"<&*&D&D%)%B%B((LL,,. #'"@"@    0( C#A 9r+) rWrUr#r!rSr r"rTrRrPrQrVrN) r!r"r#r$r%rGr)r/r*r r+r,rrs3&'%()*(*)-/Jb46Z:r+rc6^\rSrSrSrU4SjrSSjrSrU=r$)MainThreadUIQueuei=aHandles status display and processing in the main thread / master process. This class emulates a queue to cover main-thread activity before or after Apply, as well as for the single-threaded, single-process case, i.e., _SequentialApply. When multiple threads or processes are used during calls to Apply, the main thread is waiting for work to complete, and this queue must remain unused until Apply returns. Code producing arguments for Apply (such as the NameExpansionIterator) must not post messages to this queue to avoid race conditions with the UIThread. This class sends the messages it receives to UIController, which decides the correct course of action. cB>[[U] 5 X lXlg)zInstantiates a _MainThreadUIQueue. Args: stream: Stream for printing messages. ui_controller: UIController to manage messages. N)rr3rG ui_controllerry)rEryr5rs r,rGMainThreadUIQueue.__init__Ls T+-&Kr+cNURRXR5 gr1)r5r/ry)rErstimeouts r,putMainThreadUIQueue.putYsNKK8r+)ryr5r1) r!r"r#r$r%rGr9r*rrs@r,r3r3=s  99r+r3c6^\rSrSrSrSU4SjjrSrSrU=r$)UIThreadi_aResponsible for centralized printing across multiple processes/threads. This class pulls status messages that are posted to the centralized status queue and coordinates displaying status and progress to the user. It is used only during calls to _ParallelApply, which in turn is called only when multiple threads and/or processes are used. This class sends the messages it receives to UIController, which decides the correct course of action. cz>[[U] 5 XlX lX@lX0lUR5 g)zInstantiates a _UIThread. Args: status_queue: Queue for reporting status updates. stream: Stream for printing messages. ui_controller: UI controller to manage messages. timeout: Timeout for getting a message. N)rr<rG status_queueryr8r5start)rEr>ryr5r8rs r,rGUIThread.__init__ks1 (D"$$KL&JJLr+cVURRURS9nUR R XR5 U[:XagMV![Ra SnMpf=f![a(nURRSU-5 SnAgSnAff=f)N)r8zException in UIThread: %s ) r>getr8QueueEmptyr5r/ryr, Exceptionrx)rErses r,run UIThread.run|s ;  ,,000F.  < 7 7  {{ .   ; kk59::;s?A6#A/A6A6A3/A62A33A66 B(B##B()r>ryr8r5r) r!r"r#r$r%rGrGr*rrs@r,r<r<_s " ; ;r+r<rI).r% __future__rrrr collectionsrr threadingrD six.movesrrC gslib.metricsr r gslib.thread_messager r r rrrrrr gslib.utilsrgslib.utils.unit_utilrrrrZERO_TASKS_TO_DO_ARGUMENTr,rrr:r<rrrr3Threadr<r r+r,rTsM&%'  $5+,-0:6061.2.@3,88v(+(N=6N=b`*`F{&{| n:6n:b99D*;y*;r+