BSSKJrJr SSKJrJrJrJr SSKJ r SSK r SSK JrJrJrJr SSKJrJrJr SSKJr SS KJrJr SS KJrJrJrJrJ r J!r! \ RD"S 5r#\#RI\ RJ5 \ RL"5r'\'RQ\ RR"S 55 \#RU\'5 SS \+S\,S\,S\-S\\ S\\ S\.S\.S\4Sjjr/SS\S\,S\,S\-S\\ S\\ S\.S\.S\4Sjjr0SS\ S\,S\,S\-S\\ S\\ S\.S\.S\4Sjjr1SS\ S\,S\,S\-S\\ S\\ S\.S\4Sjjr2g!\ a \ r GNIf=f))basenamesplitext)BinaryIOListOptionalSet)PathLikeN)coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCE) mess_ratio) CharsetMatchCharsetMatches)any_specified_encoding iana_nameidentify_sig_or_bom is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s sequencessteps chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc d[U[[45(d#[SR [ U555eU(d$[ R[R5 O#[ R[R5 [U5nUS:Xa/[ RS5 [[USSS/S5/5$UbB[ RS S RU55 UV s/sHn [!U S5PM nn O/nUbB[ RS S RU55 UV s/sHn [!U S5PM nn O/nXU-::a[ RS UUU5 S nUnUS :aX- U:a [#X- 5n[U5[$:n [U5[&:n U (a%[ RSR U55 O+U (a$[ R)SR U55 /n USLa [+U5OSn U b'U R-U 5 [ R)SU 5 [/5n/n/nSnSnSn[5n[1U5unnUb1U R-U5 [ R)S[U5U5 U R-S5 SU ;aU R-S5 U [2-GHnU(aUU;aMU(aUU;aM"UU;aM*UR5U5 SnUU:HnU=(a [7U5nUS;aUSLa[ R)SU5 My[9U5nU (a8USLa3[AUSLaUS[#S5OU[U5[#S5US9 O[AUSLaUO U[U5SUS9nSnUHn[GUU5(dMSn O U(a[ RSUW5 GM$[IUSLaSO [U5U[#X- 55nU=(a USL=(a [U5U:n U (a[ R)SU5 [#[U5S- 5n!U!S:aSn!Sn"/n#/n$UHn%UU%U%U-n&U(a USLaUU&-n&U&RKUSS9n'U(atU%S:anUU%S :aeUS!:aS!OUn(U(aTU'SU(U;aK[IU%U%S- S"5H7n)UU)U%U-n&U(a USLaUU&-n&U&RKUSS9n'U'SU(U;dM7 O U#R-U'5 U$R-[MU'U55 U$S"U:aU"S - n"U"U!:dU(dMUSLdM O U$(a[OU$5[U$5- n*OSn*U*U:dU"U!:adUR-U5 [ RS#UU"[QU*S$-S%S&95 USSU 4;a$[UUUS/U5n+UU :XaU+nO US:XaU+nOU+nGM>[ R)S'U[QU*S$-S%S&95 U(d [SU5n,O [UU5n,U,(a.[ R)S(R U[AU,555 /n-U#H9n'[WU'S)U,(aS*RU,5OS5n.U-R-U.5 M; [YU-5n/U/(a%[ R)S+R U/U55 UR-[UUU*UU/U55 UU SS4;a-U*S):a'[ R)S,U5 [UU/5s $UU:XdGM[ R)S-U5 [UU/5s $ [U5S:XaU(dU(dU(a[ RS.5 U(a3[ RS/URZ5 UR-U5 U$U(aUb+U(a!U(aUR\UR\:wdUb([ RS05 UR-U5 U$U(a&[ RS15 UR-U5 U$s sn fs sn f![:[<4a [ R?SU5 GMf=f![B[D4aRn[U[D5(d [ RSU[AU55 UR-U5 SnAGMFSnAff=f)2a Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. z4Expected object of type bytes or bytearray, got: {0}rzXGiven content is empty, stopping the process very early, returning empty utf_8 str matchutf_8gFNz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).Tz@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii>utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecodergA)encodingz9Code page %s does not fit given bytes sequence at ALL. %szW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.ignore)errorszc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d)ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z0%s is most likely the one. Stopping the process.z[%s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z#%s will be used as a fallback matchz&utf_8 will be used as a fallback matchz&ascii will be used as a fallback match)/ isinstance bytearraybytes TypeErrorformattypeloggersetLevelloggingCRITICALINFOlenwarningrrjoinrintrrinforappendsetrraddrrModuleNotFoundError ImportErrordebugstrUnicodeDecodeError LookupErrorrrangedecodersumroundr r r rr+ fingerprint)0rrrrr r!r"r#lengthcpis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresults sig_encoding sig_payload encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_count md_chunks md_ratiosi cut_sequencechunkchunk_partial_size_chkjmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergeds0 1platform/bq/third_party/charset_normalizer/api.py from_bytesr~&s[ 2 i)U!3 4 4 B I IY   (() % ^F { f |IwUBPRSTUU 5 IIl # 8DD| "e,| D   6 IIl # 8DD| "e,| D   u$% l       qyV^j0(  N-?? N.>> L S S   W ^ ^  .BT-Iy)t%$$%78 N  UF  NKG 3I >L+$$\2 W      )++$$W-.? M=  M\9  F "  =!+}</ 4K 5  0 05IU5R KKm   $:=$I ! $)>%)G'50kD *"3{#3c$i@* #&'50"3{#3#56* # %*!$; ],@AA,0)%< % NNi$   %.AC 4D     " .t+ .O$v-   KK-   B! , q !   A$QZ8L#(8E(A*\9  '' h'GE%Q9Q<43G%r/Bz' $556oM"1a!eR0'0Q^'D /4D4M+6+EL , 3 3M( 3 S !8"89_L!1   U #   Zy9 :} ) A%  $55$$)9U)BSV !)ns9~=O!O i '+;?P+P # * *= 9 NN0 o+Q7  '3E FF!-}iO"!$66)7&"g-%3N"0K  K  /C' 3 %1-@ 4]C   KK8??!3'7#8   E-s:JCHH%56PTO   _ - 2)<  KK299$m   $     0'7C C#% KKBM "7=#9":; ; L ( KKm "7=#9":; ;U@X 7|q .,> NNa   NN57I7R7R  NN- . N^3"++~/I/II' NNC D NN; ' N  NNC D NN> * No EEF$[1  LLDm    *#K0 a--O!F $ * *= 9  s=9bb b*>c )c &c  c  d/Ad**d/fpc B[UR5UUUUUUU5$)zz Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )r~read)rrrrr r!r"r#s r}from_fprs/      pathc t[US5n[UUUUUUUU5sSSS5 $!,(df  g=f)z Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. rbN)openr) rrrrr r!r"r#rs r} from_pathrs< dD R            s) 7c [UUUUUUU5n[U5n[[U55n [ U5S:Xa[ SR U55eUR5n U S==SU R-- ss'[SR [U5RUSRU 555S5n U RU R55 SSS5 U $!,(df  U $=f)za Take a (text-based) file path and try to create another file next to it, this time using UTF-8. rz;Unable to normalize "{}", no encoding charset seems to fit.-z{}r'wbN)rrlistrrBIOErrorr;bestr+rrMreplacerDwriteoutput) rrrrr r!r"rafilenametarget_extensionsresultrs r} normalizers  G~HXh/0 7|q I P P   \\^FaC&//11  CI%%h8I0JKLd   !  M    Ms  C11 D)皙?NNTF)rrrNNT)3os.pathrrtypingrrrrosr rKrMr?cdr r r rconstantrrrmdrmodelsrrutilsrrrrrr getLoggerr=r>DEBUG StreamHandlerhandler setFormatter Formatter addHandlerr9rEfloatboolr~rrrrr}rs&00 KJ0   / 0     !W&&'RST' ""!%JJ JJ J s) J s) JJJJ^ ""!%   s)  s) 8""!%      s)  s)     :""!%) ) )) ) s) ) s) )))CHsEE&%E&