HIfSrSSKJr SSKJr SSKJr SSKJr SSKrSSKrSSKrSSK r SSK r SSK J r SSK r SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SrSrSrSrSrSrSrSrSr/SQr Sr!Sr"Sr#Sr$Sr%Sr&S*Sjr'Sr(S r)S!r*S"r+S#r,S$r-S%r.S+S&jr/S,S'jr0"S(S)\15r2g)-z+Helper functions for hashing functionality.)absolute_import)print_function)division)unicode_literalsN)config)CommandException)UsingCrcmodExtension)DEFAULT_FILE_BUFFER_SIZE)MIN_SIZE_COMPUTE_LOGGING)TRANSFER_BUFFER_SIZE)UTF8z WARNING: You have requested checksumming but your crcmod installation isn't using the module's C extension, so checksumming will run very slowly. For help installing the extension, please see "gsutil help crcmod". a WARNING: gsutil rsync uses hashes when modification time is not available at both the source and destination. Your crcmod installation isn't using the module's C extension, so checksumming will run very slowly. If this is your first rsync since updating gsutil, this rsync can take significantly longer than usual. For help installing the extension, please see "gsutil help crcmod". a| WARNING: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod". To disable slow integrity checking, see the "check_hashes" option in your boto config file. a: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod". To download regardless of crcmod performance or to skip slow integrity checks, see the "check_hashes" option in your boto config file. NOTE: It is strongly recommended that you not disable integrity checks. Doing so could allow data corruption to go undetected during uploading/downloading.aT WARNING: This download will not be validated since your crcmod installation doesn't use the module's C extension, so the hash computation would likely throttle download performance. For help in installing the extension, please see "gsutil help crcmod". To force integrity checking, see the "check_hashes" option in your boto config file. if_fast_else_failif_fast_else_skipalwaysnever)iiAoivE:iWiPYlT/Ul?OiNi.%i'hinTi\Fib&l",iGWL;iglb !l<cl!lN,Ql2!l=X iD3iKS}i2=lmyixBlAo= c8U(dU$[USU-5U- $)amComputes CRC32C for concat(A, B) given crc(A), crc(B) and len(B). An explanation of the algorithm can be found at crcutil.googlecode.com/files/crc-doc.1.0.pdf. Args: crc_a: A 32-bit integer representing crc(A) with least-significant coefficient first. crc_b: Same as crc_a. num_bytes_in_b: Length of B in bytes. Returns: CRC32C for concat(A, B) )_ExtendByZeros)crc_acrc_bnum_bytes_in_bs -platform/gsutil/gslib/utils/hashing_helper.py ConcatCrc32crjs#  L q>1 2U ::cSnS[-n[[5H-nUS-(aX!-nUS-nX-(a U[-nUS-nM/ U$)zMultiplies two polynomials together modulo CASTAGNOLI_POLY. Args: p: The first polynomial. q: The second polynomial. Returns: Result of the multiplication. r)DEGREErangeCASTAGNOLI_POLY)pqresulttop_bit_s r _CrcMultiplyr*sV & K' =a1u kf!GA{?a!GA  -rcSnU"U5nSnUS:wa?US-(a#[U[U[[5-5nUS- nUS-nUS:waM?U"U5nU$)zGiven crc representing polynomial P(x), compute P(x)*x^num_bits. Args: crc: crc respresenting polynomial P(x). num_bits: number of bits in crc. Returns: P(x)*x^num_bits cB[SRUSS9SSS2S5$)Nz{0:032b}r)widthr)intformat)crcs r_ReverseBits32&_ExtendByZeros.._ReverseBits32s( z  B /"5q 99rrr!)r*X_POW_2K_TABLElen)r1num_bitsr2is rrrsm: s#!A !| nQ^1D-DE FcFA NH A s# *rcSU0nURS5 [X5 URS5 USR5$)aCalculates a base64 digest of the contents of a seekable stream. This function resets the file pointer to position 0. Args: fp: An already-open file object. hash_alg: Instance of hashing class initialized to start state. Returns: Hash of the stream in hex string format. placeholderr)seekCalculateHashesFromContents hexdigest)fphash_alg hash_dicts r_CalculateHashFromContentsr@s@h')''!*b,''!* = ! + + --rc^UR[5nU(dg[R(a*[ U[ 5(aUR [5n[R"U5HnURU5 M U(aUR[U55 M)aoCalculates hashes of the contents of a file. Args: fp: An already-open file object (stream will be consumed). hash_dict: Dict of (string alg_name: initialized hashing class) Hashing class will be populated with digests upon return. callback_processor: Optional callback processing class that implements Progress(integer amount of bytes processed). N) readr sixPY3 isinstancestrencoder itervaluesupdateProgressr5)r=r?callback_processordatar>s rr;r;sw 77+ ,D   ww D#  {{4 NN9-ood.!!#d), rcT[U[RRS55$)zCalculates a base64 CRC32c checksum of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: CRC32c checksum of the file in base64 format. crc-32c)$_CalculateB64EncodedHashFromContentscrcmod predefinedCrcr=s r%CalculateB64EncodedCrc32cFromContentsrTs) .b.4.?.?.C.CI.N PPrc*[U[55$)zCalculates a base64 MD5 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: MD5 digest of the file in base64 format. )rOGetMd5rSs r"CalculateB64EncodedMd5FromContentsrWs .b&( ;;rc*[U[55$)zCalculates a base64 MD5 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. Returns: MD5 digest of the file in hex format. )r@rVrSs rCalculateMd5FromContentsrYs $B 11rc[R"[R"U55nUR S5R [ 5$)zAReturns the base64-encoded version of the input hex digest value. )base64 b64encodebinascii unhexlifyrstripdecoder ) digest_value encoded_bytess rBase64EncodeHashrds9""8#5#5l#CD-   e $ + +D 11rc[R"URS5R[55n[ R "U5$)zReturns the hex digest value of the input base64-encoded hash. Args: base64_hash: Base64-encoded hash, which may contain newlines and single or double quotes. Returns: Hex digest of the input argument. z "')r\ b64decodestriprGr r^hexlify) base64_hash decoded_bytess rBase64ToHexHashrk s;"";#4#4W#=#D#DT#JK-   - ((rc*[[X55$)a&Calculates a base64 digest of the contents of a seekable stream. This function sets the stream position 0 before and after calculation. Args: fp: An already-open file object. hash_alg: Instance of hashing class initialized to start state. Returns: Hash of the stream in base64 format. )rdr@)r=r>s rrOrOs 4RB CCrcZ[R"SS[5nUS:Xa0$S[0$)a3Returns a dict of hash algorithms for validating an uploaded object. This is for use only with single object uploads, not compose operations such as those used by parallel composite uploads (though it can be used to validate the individual components). Returns: dict of (algorithm_name: hash_algorithm) GSUtil check_hashesrmd5)rgetCHECK_HASH_IF_FAST_ELSE_FAILrV)check_hashes_configs rGetUploadHashAlgsrt(s2 8^#?AG# I rc[R"SS[5nU[:Xa0$0nU(a [US'U$U(a[ 5(aSUS'U$U(dlU[:Xa[ [5eU[:XaUR[5 U$U[:XaUR[5 SUS'U$[ S5eU$)azReturns a dict of hash algorithms for validating an object. Args: logger: logging.Logger for outputting log messages. consider_md5: If True, consider using a md5 hash. consider_crc32c: If True, consider using a crc32c hash. Returns: Dict of (string, hash algorithm). Raises: CommandException if hash algorithms satisfying the boto config file cannot be returned. rnrorpc@[RRS5$NrNrPrQrRrr%GetDownloadHashAlgs..UsF$5$5$9$9)$Drcrc32cc@[RRS5$rwrxryrrrzr{]sf&7&7&;&;I&Frz8Your boto config 'check_hashes' option is misconfigured.) rrqrrCHECK_HASH_NEVERrVr r_SLOW_CRC_EXCEPTION_TEXTCHECK_HASH_IF_FAST_ELSE_SKIPwarn_NO_HASH_CHECK_WARNINGCHECK_HASH_ALWAYS_SLOW_CRCMOD_DOWNLOAD_WARNING)logger consider_md5consider_crc32crs hash_algss rGetDownloadHashAlgsr9s 8^#?A,, I)Ie& %Di   < <788 "> > *+  "3 3 12F (  HJ J rcz[R"U5$![a [R"USS9s$f=f)a>Returns md5 object, avoiding incorrect FIPS error on Red Hat systems. Examples: GetMd5(b'abc') GetMd5(bytes('abc', encoding='utf-8')) Args: byte_string (bytes): String in bytes form to hash. Don't include for empty hash object, since md5(b'').digest() == md5().digest(). Returns: md5 hash object. F)usedforsecurity)hashlibrp ValueError) byte_strings rrVrVes8; ;;{ ## ; ;;{E :: ;s ::cj\rSrSrSrSr\S5rS SjrSr Sr \ R4Sjr S rS rg ) HashingFileUploadWrapperi|aWraps an input stream in a hash digester and exposes a stream interface. This class provides integrity checking during file uploads via the following properties: Calls to read will appropriately update digesters with all bytes read. Calls to seek (assuming it is supported by the wrapped stream) using os.SEEK_SET will catch up / reset the digesters to the specified position. If seek is called with a different os.SEEK mode, the caller must return to the original position using os.SEEK_SET before further reads. Calls to seek are fast if the desired position is equal to the position at the beginning of the last read call (we only need to re-hash bytes from that point on). c:U(d [S5eU(d [S5eXlX lX@lXPlSUl0UlURH-nURUR5UR U'M/ SUlSUl X0l g)aInitializes the wrapper. Args: stream: Input stream. digesters: dict of {string: hash digester} containing digesters, where string is the name of the hash algorithm. hash_algs: dict of {string: hash algorithm} for resetting and recalculating digesters. String is the name of the hash algorithm. src_url: Source FileUrl that is being copied. logger: For outputting log messages. z0HashingFileUploadWrapper used with no digesters.z0HashingFileUploadWrapper used with no hash_algs.Nr) r_orig_fp _digesters_src_url_logger _seek_away_digesters_previouscopy_digesters_previous_mark_digesters_current_mark _hash_algs)selfstream digestersrsrc_urlralgs r__init__!HashingFileUploadWrapper.__init__s  O PP  O PPMOMLDO!D&*ooc&:&?&?&Ads#$%D!#$D Orc0[URSS5$)z$//# # 00 0'+'D'D$??C!%!9!9#!>$//# # 4889 f<<<= 4778 f;;;< ==  f --rcfURR5UR:wa5[SURR5<SUR<35eURHhnU[ :a1UR RSUURR5 URUR5URU'Mj URUl Un[U[5nU(aURRU5n[!U["R$5(aUR'[(5nX4-nURH!nURUR+U5 M# [U[5nU(aMU=RU- slg)aSCatches up hashes, but does not return data and uses little memory. Before calling this function, digesters_current_mark should be updated to the current location of the original stream and the self._digesters should be current to that point (but no further). Args: bytes_to_read: Number of bytes to catch up from the original stream. z6Invalid mark when catching up hashes. Stream position z, hash position zCatching up %s for %s...N)rrrrrr rdebugr url_stringrrrminr rBrErCrrGr rI)r bytes_to_readrbytes_remainingbytes_this_roundrLs rr!HashingFileUploadWrapper._CatchUps[ }}t;;; ==--/1M1M O PP 2 2 5s==33 5&*ooc&:&?&?&Ads#  %)$@$@D!#O?,@A  ]]   0 1d D#-- ( ({{4 )o# ##D)!_.BC     M1 r) rrrrrrrrrN)r.)__name__ __module__ __qualname____firstlineno____doc__rpropertyrrBrrrrr:r__static_attributes__ryrrrr|sC  < 0 08 $!# ..` 2rr)N)FF)r)3r __future__rrrrr\r^rrrCbotorrPgslib.exceptionrgslib.utils.boto_utilr gslib.utils.constantsr r r r SLOW_CRCMOD_WARNINGSLOW_CRCMOD_RSYNC_WARNINGrrrrrrrr~r4r$r"rr*rr@r;rTrWrYrdrkrOrtrrVobjectrryrrrs2&%'   ,6::6& ! N 32  ;*. 4.&-. P < 22 ) D")X;.h2vh2r