qSrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK r SSK J r SSK Jr SS K Jr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJr SSKJr SSK r SSK J!r! \ RD(a\#r$\"S5r%"SS\&5r'"SS\&5r(Sr)"SS\&5r*Sr+"SS\ RX5r-"SS \ RX5r.g)!zEMedia helper functions and classes for Google Cloud Storage JSON API.)absolute_import)print_function)division)unicode_literalsN) http_client)urllib) cStringIO) exceptions)BadRequestException) LazyWrapper)ProgressCallbackWithTimeout)DEBUGLEVEL_DUMP_REQUESTS)SSL_TIMEOUT_SEC)TRANSFER_BUFFER_SIZE)UTF8) text_util) parse_uric.[R"S5$)Nz\d+)recompile'platform/gsutil/gslib/gcs_json_media.pyr2s RZZ%7rcR\rSrSrSrSr\S5r\RS5rSr g)BytesTransferredContainer5aContainer class for passing number of bytes transferred to lower layers. For resumed transfers or connection rebuilds in the middle of a transfer, we need to rebuild the connection class with how much we've transferred so far. For uploads, we don't know the total number of bytes uploaded until we've queried the server, but we need to create the connection class to pass to httplib2 before we can query the server. This container object allows us to pass a reference into Upload/DownloadCallbackConnection. cSUlg)Nr-_BytesTransferredContainer__bytes_transferredselfs r__init__"BytesTransferredContainer.__init__@s  DrcUR$Nrr!s rbytes_transferred+BytesTransferredContainer.bytes_transferredCs  # ##rcXlgr&rr"values rr'r(Gs$r)__bytes_transferredN) __name__ __module__ __qualname____firstlineno____doc__r#propertyr'setter__static_attributes__rrrrr5s:! $ $%%rrc2\rSrSrSr\SSSS4SjrSrSrg)$UploadCallbackConnectionClassFactoryLzCreates a class that can override an httplib2 connection. This is used to provide progress callbacks and disable dumping the upload payload during debug statements. It can later be used to provide on-the-fly hash digestion during upload. rNcLXlX lX0lX@lXPlX`lgr&)bytes_uploaded_container buffer_size total_sizeprogress_callbackloggerdebug)r"r9r:r;r<r=r>s rr#-UploadCallbackConnectionClassFactory.__init__Ts%%=!" O.KJrc^^^^^^URmURmURmURmURmUR m"UUUUUU4SjS[ R5nU$)z/Returns a connection class that overrides send.c|>\rSrSrSrYrSrYrSrYr Sr Sr Sr Sr SrS UU4SjjrUU4S jrS UU4S jjrS rg)YUploadCallbackConnectionClassFactory.GetConnectionClass..UploadCallbackConnectionkz&Connection class override for uploads.FNg?c^[US'[RR"U/UQ70UD6 gNtimeoutrhttplib2HTTPSConnectionWithTimeoutr#r"argskwargss rr#bUploadCallbackConnectionClassFactory.GetConnectionClass..UploadCallbackConnection.__init__z*+y++44TKDKFKrc`>URRS5 [R(a URnOa/nURHOn[ U[ 5(aUR U5 M+UR UR[55 MQ SRU5n[U5nT[:XaT(aTRSU-5 URSS2 [ U[5(aXQ- nSnURXVS9 UbURU5 gg)zSend the currently buffered request and clear the buffer. Appends an extra \r\n to the buffer. Args: message_body: if specified, this is appended to the request. )rrs zsend: %sN)num_metadata_bytes)_bufferextendsixPY2 isinstancebytesappendencoderjoinlenrr>strsend) r" message_bodyencode_chunkeditemsitemmsgrQ outer_debug outer_loggers r _send_outputfUploadCallbackConnectionClassFactory.GetConnectionClass..UploadCallbackConnection._send_outputs J' 77,,%%lld$&&ll4 ll4;;t,- # ll5! X 2 2|   Z#- . LLO lC ( ( #, # =  # ))L ! $rc >US:XaTSRUVs/sHn[U5PM sn5nX@lT[:XaT(aTR S5 OUS:Xa<[ SRUVs/sHn[U5PM sn55nX@lOUS:XaySRUVs/sHn[U5PM sn5n[5RU5n[U5S:a%[ US5[ US5- S-Ul URS:XaUR (a|UR(akUR[UR 5- Ul SUlS UlS Ul T[:Xa%T(aTR S UR-5 [RR "X/UQ76 g s snfs snf![a Nf=fs snf![a Nf=f) a%Overrides HTTPConnection.putheader. Send a request header line to the server. For example: h.putheader('Accept', 'text/html'). This override records the content encoding, length, and range of the payload. For uploads where the content-range difference does not match the content-length, progress printing will under-report progress. These headers are used to calculate a multiplier to correct the progress. For example: the content-length for gzip transport encoded data represents the compressed size of the data while the content-range difference represents the uncompressed size. Dividing the content-range difference by the content-length gives the ratio to multiply the progress by to correctly report the relative progress. Args: header: The header. *values: A set of values for the header. zcontent-encodingrDz4send: Using gzip transport encoding for the request.content-lengthz content-rangergzipNz&send: Setting progress modifier to %s.)rZr\header_encodingrr>int header_length ValueError DECIMAL_REGEXfindallr[ header_rangefloat size_modifierrHTTPSConnection putheader)r"headervaluesvr+rangesrcrds rrucUploadCallbackConnectionClassFactory.GetConnectionClass..UploadCallbackConnection.putheaders* ' '''626a3q6623%!&  4 4   F H ' '  8AQ 89:E!&  & GGV4VSVV45E"_,,U3F6{Q#&vay>Cq N#Ba"Gd  F *t/A/A   #0059K9K3LL$ !#$ #$ "$  4 4   G $ 2 2 4 5 ##--dDVDO3!9    5   sNG#G&7G! G&%G;4G6 AG;!G&& G32G36G;; HHc">UR(dNSUlT(a@[TT5UlURRURR 5 [ U[R5(a [U5nO8[ U[R5(a[R"U5nOUnURUR5nU(Ga)[R(a [R R#X5 Og[ U[$5(a [R R#X5 O2[R R#XR'[(55 [+U5nU(aX%::aXR-nSnOX%-nSnUR(a2[-XPR.-5nURRU5 URUR5nU(aGM(gg)zOverrides HTTPConnection.send. Args: data: string or file-like object (implements read()) of data to send. num_metadata_bytes: number of bytes that consist of metadata (headers, etc.) not representing the data being uploaded. TrN)processed_initial_bytesr callback_processorProgressr9r'rVrT text_typer binary_typeBytesIOreadGCS_JSON_BUFFER_SIZErUrIrJr]rWrYrr[rlrs)r"datarQ full_bufferpartial_buffersent_data_bytesouter_progress_callbackouter_total_sizes rr]^UploadCallbackConnectionClassFactory.GetConnectionClass..UploadCallbackConnection.sends++)-$ & $&A "9';D #  # # , ,--?? A dCMM * *!$+ coo . . D)++$))$*C*CD WW  / / 4 4T J.%001166tL1166--d35// !43o#$  3 !o  $ $"/4F4F"FGO  # # , ,_ =&++D,E,EF.7nr)r}rkrmrqr|rs)NF)r)r-r.r/r0r1r9r|rr}sizerkrmrqrsr#rerur]r4)outer_buffer_sizeouter_bytes_uploaded_containerrcrdrrsrUploadCallbackConnectionrBks]2!?!&. domlmL%"%"N=E~3G3Grr)r9r:r;r<r=r>rIrJ)r"rrrrcrdrrs @@@@@@rGetConnectionClass7UploadCallbackConnectionClassFactory.GetConnectionClassbsi%)%B%B"(("44;;L**KoGoG8#F#FoGb $#r)r:r9r>r=r<r; r-r.r/r0r1rr#rr4rrrr6r6Ls#0!% z$rr6c`^URmSSS[RS4U4SjjnXlg)zWraps upload_http so we only use our custom connection_type on PUTs. POSTs are used to refresh oauth tokens, and we don't want to process the data sent in those requests. Args: upload_http: httplib2.Http instance to wrap GETNc <>US:XdUS:XaUnOSnT"UUUUUUS9$)NPUTPOSTmethodbodyheaders redirectionsconnection_typer)urirrrrroverride_connection_type request_origs r NewRequest)WrapUploadHttpRequest..NewRequest*s? &F*!0!% %! '%1(@  BBr)requestrIDEFAULT_MAX_REDIRECTS) upload_httprrs @rWrapUploadHttpRequestrs5$$,&<\rSrSrSrYR rYRrYRr YRr Sr SrSrSSjrSrg) ]DownloadCallbackConnectionClassFactory.GetConnectionClass..DownloadCallbackConnectioniYz(Connection class override for downloads.FNc^[US'[RR"U/UQ70UD6 grFrHrKs rr#fDownloadCallbackConnectionClassFactory.GetConnectionClass..DownloadCallbackConnection.__init__brOrc^^[RRT5nUR[R[R 4;aU$UR mSUU4SjjnX2lU$)zWraps an HTTPResponse to perform callbacks and hashing. In this function, self is a DownloadCallbackConnection. Args: buffering: Unused. This function uses a local buffer. Returns: HTTPResponse object with wrapped read function. ct>U(a U[:a[SU<S[<S35eU=(d [nTR(dlSTlTR(aT[ TR TR5TlTR RTRR5 T"U5n[U5nTR (aTR RU5 TR(a1TRH!nTRURU5 M# U$)a@Overrides HTTPConnection.getresponse.read. This function only supports reads of TRANSFER_BUFFER_SIZE or smaller. Args: amt: Integer n where 0 < n <= TRANSFER_BUFFER_SIZE. This is a keyword argument to match the read function it overrides, but it is required. Returns: Data read from HTTPConnection. zInvalid HTTP read size z during download, expected .T) rr r|rr rr}r~ outer_bytes_downloaded_containerr'r[outer_digestersupdate)amtr read_lengthalgorig_read_funcr"s rrwDownloadCallbackConnectionClassFactory.GetConnectionClass..DownloadCallbackConnection.getresponse..readws22%*,- ---C--+/D (++(C'')E)E)Gd%%%..77IIK $$D +  $ $  # # , ,[ 9  ! !++""3'..t4,+rr&)rHTTPConnection getresponsestatusOKPARTIAL_CONTENTr)r" buffering orig_responserrs` @rriDownloadCallbackConnectionClassFactory.GetConnectionClass..DownloadCallbackConnection.getresponsefsi$22>>tD    (3(C(C(E E &++# # J"r)r}r|)F)r-r.r/r0r1r;rrrr<rrrr|r}r#rr4r!srDownloadCallbackConnectionrYsH4o $ 6 6)-)H)H& %L8rr)rIrJ)r"rs` rr9DownloadCallbackConnectionClassFactory.GetConnectionClassVs EX%H%HEN &%r)r:rrr<r;rrrrrr?s!0!% AJ&rrc^SnURmSSS[RS4U4Sjjn[R"X5UlX lU$)zOverrides download request functions for an httplib2.Http object. Args: download_http: httplib2.Http.object to wrap / override. Returns: Wrapped / overridden httplib2.Http object. c URV s/sH.n U RX$5(dMU RU5U 4PM0 n n U =(a [U 5SS=(d Sn U (aU R XTXv5 UR XXVU5upU (aDU R X5(a.U R XTXv5 UR XUXg5upSU lU RS:XaURX$X|U 5HhnUR XTXv5 UR XUXg5upU RS:wdM<URRU5 UR X5 O UR(dUS;dU RS:XGaUR(GaU RS;GaqU(GaRSU ;a'U RS :wa[R"S X5eSU ;a;U Sn[U5unnnnnUc"[ R"R%X?5U S'U RS :Xa:US;a4U SU S 'S U ;aX.OverrideRequests ;?:M:M2:M$d0.tzz+&-:ME2  (VE]1%a( 0DD  ll66,,T-46X x & & V'8"00F15@!"#44 W9-f7A"00F15@ ??c !    $ $] 3   0 9 !!f&? ??c !   8??7A$A  x 'HOOs,B22L# #8 # +H9B89L 6VYeX %+\\%9%9,%Qhz" __ #/(A4US:Xa T"XUX4SS9$T"XUX4US9$)Nrrr)rrrrrrrs rr+WrapDownloadHttpRequest..NewRequests= #4")*.00#4")*9;;r)rrIrtypes MethodType_request) download_httprrrs @rWrapDownloadHttpRequestrsP*Z~&&,"t&<>  ??4 jjl  ( ()J)-*3 44 <<   jjl << c F  a #gg ++ +  ,  $ $  jjl  LL+33 4  jjl  s&AB1G 1BG  AF!!)G  6HrN)r-r.r/r0r1rr4rrrrr+s  )rrcf^\rSrSrSrU4Sjr\S5r\RS5rSr Sr U=r $)HttpWithDownloadStreamieahttplib2.Http variant that only pushes bytes through a stream. httplib2 handles media by storing entire chunks of responses in memory, which is undesirable particularly when multiple instances are used during multi-threaded/multi-process copy. This class copies and then overrides some httplib2 functions to use a streaming copy approach that uses small memory buffers. Also disables httplib2 retries (for reasons stated in the HttpWithNoRetries class doc). cp>SUl[R"5Ul[[ U]"U0UD6 gr&)_streamlogging getLogger_loggersuperrr#)r"rLkwdsrs rr#HttpWithDownloadStream.__init__rs/DL$$&DL $0$?$?rcUR$r&rr!s rstreamHttpWithDownloadStream.streamws <<rcXlgr&r r*s rr!r"{sLrc[US5(aURcUR5 URX2XE5 UR'5nSn US:Xa)UR5 [R("U5nX4$US:XGaOUR*["R,["R.4;Ga Sn [US5(aUR1S 5n Un Sn U R3[45n U (aSUR6c[8R:"S 5e[<R>"UR6U 5 U [AU 5- n OOMrU bi[CU 5[CU 5:waQURDRG[HRJS XU5 URLS [OU 5URLS '[R("U5nX4$UR35n [R("U5n[RP"X5n X4$![R a e[R a4 UR5 [R"SUR-5e[RRa UR5 e[RaYnSn[US5(a[US5SnO URnU[RR :XaeSnAGNSnAf["R$a UR5 ef=f![R["R$4anUR5 eSnAff=f) NrrrrLrDrrrbrhz5Cannot exercise HttpWithDownloadStream with no streamzlOnly got %s bytes out of content-length %s for request URI %s. Resetting content-length to match bytes read.))rrrrrrGrrrIrrr r r r r rrrrrrrr getheaderrrr!apitools_exceptionsInvalidUserInputErrorr write_to_fdr[longrlogrDEBUGrbr\r)r"rrrrrrrrrcontent_length http_stream bytes_readnew_datas rr$HttpWithDownloadStream._conn_requests v  499#4  ll66.4A!!#h g 6  $$X.X  W U?x;>>3>3N3N3P P 8U # ##--.>?.   %%&:;( {{"'==IKK  ! !$++x 8 #h- 'J   &  ^ 4 4 ,,  mm' Fll+,+.z?(,,' ($$X.   --/$$X.--h@  W >>  ??4 jjl  ( ()J)-*3 44 <<   jjl << c F  a #gg ++ +  ,  $ $  jjl   LL+33 4  jjl  s2AHL6BL36AL  )L36$M0M++M0)rr) r-r.r/r0r1r#r2r!r3rr4 __classcell__)rs@rrresF @    ==PPrr)/r1 __future__rrrrrrrrrrT six.movesrrr apitools.base.pyr r&gslib.cloud_apir gslib.lazy_wrapperr gslib.progress_callbackr gslib.utils.constantsrrrr gslib.utilsrrIrPY3rlr)roobjectrr6rrrHttprrrrrr=sL&%'  !>/*?:16&!77 $89 %%.P$6P$f#@a&Va&HEP7 7tkX]]kr