a ‚oeOã@snddlmZddlmZddlmZmZddlmZm Z m Z ddl m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZGdd„dƒZGd d „d eƒZGd d „d eƒZGd d„deƒZ Gdd„deƒZ!Gdd„deƒZ"Gdd„deƒZ#Gdd„deƒZ$Gdd„deƒZ%Gdd„deƒZ&edd�ee'ee'e(dœdd„ƒZ)ed d�d'e'e*e(e*d#œd$d%„ƒZ+d&S)(é)Ú lru_cache)Ú getLogger)ÚListÚOptionalé)ÚCOMMON_SAFE_ASCII_CHARACTERSÚTRACEÚUNICODE_SECONDARY_RANGE_KEYWORD)Úis_accentuatedÚ is_arabicÚis_arabic_isolated_formÚis_case_variableÚis_cjkÚ is_emoticonÚ is_hangulÚ is_hiraganaÚ is_katakanaÚis_latinÚis_punctuationÚ is_separatorÚ is_symbolÚis_thaiÚis_unprintableÚ remove_accentÚ unicode_rangec@sPeZdZdZeedœdd„Zeddœdd„Zddœd d „Ze e dœd d „ƒZ dS) ÚMessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. ©Ú characterÚreturncCst‚dS)z@ Determine if given character should be fed in. N©ÚNotImplementedError©Úselfr©r#ú6C:\Program Files\Certbot\pkgs\charset_normalizer\md.pyÚeligible%szMessDetectorPlugin.eligibleNcCst‚dS)z‰ The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. Nrr!r#r#r$Úfeed+szMessDetectorPlugin.feed©rcCst‚dS)zB Permit to reset the plugin to the initial state. Nr©r"r#r#r$Úreset2szMessDetectorPlugin.resetcCst‚dS)z… Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. Nrr(r#r#r$Úratio8szMessDetectorPlugin.ratio) Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚstrÚboolr%r&r)ÚpropertyÚfloatr*r#r#r#r$rs rc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)Ú TooManySymbolOrPunctuationPluginNr'cCs"d|_d|_d|_d|_d|_dS)NrF)Ú_punctuation_countÚ _symbol_countÚ_character_countÚ_last_printable_charZ_frenzy_symbol_in_wordr(r#r#r$Ú__init__Bs z)TooManySymbolOrPunctuationPlugin.__init__rcCs| ¡S©N©Ú isprintabler!r#r#r$r%Jsz)TooManySymbolOrPunctuationPlugin.eligiblecCsp|jd7_||jkrf|tvrft|ƒr8|jd7_n.| ¡durft|ƒrft|ƒdurf|jd7_||_dS)NrFé) r6r7rrr4Úisdigitrrr5r!r#r#r$r&Msÿþ ÿþ ýz%TooManySymbolOrPunctuationPlugin.feedcCsd|_d|_d|_dS©Nr)r4r6r5r(r#r#r$r)_sz&TooManySymbolOrPunctuationPlugin.resetcCs0|jdkrdS|j|j|j}|dkr,|SdS)Nrçç333333Ó?)r6r4r5)r"Zratio_of_punctuationr#r#r$r*ds   þz&TooManySymbolOrPunctuationPlugin.ratio© r+r,r-r8r/r0r%r&r)r1r2r*r#r#r#r$r3As r3c@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚTooManyAccentuatedPluginNr'cCsd|_d|_dSr>©r6Ú_accentuated_countr(r#r#r$r8qsz!TooManyAccentuatedPlugin.__init__rcCs| ¡Sr9)Úisalphar!r#r#r$r%usz!TooManyAccentuatedPlugin.eligiblecCs(|jd7_t|ƒr$|jd7_dS©Nr)r6r rDr!r#r#r$r&xszTooManyAccentuatedPlugin.feedcCsd|_d|_dSr>rCr(r#r#r$r)~szTooManyAccentuatedPlugin.resetcCs*|jdkrdS|j|j}|dkr&|SdS)Nér?gffffffÖ?rC)r"Zratio_of_accentuationr#r#r$r*‚s  zTooManyAccentuatedPlugin.ratiorAr#r#r#r$rBps rBc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚUnprintablePluginNr'cCsd|_d|_dSr>)Ú_unprintable_countr6r(r#r#r$r8ŒszUnprintablePlugin.__init__rcCsdS©NTr#r!r#r#r$r%�szUnprintablePlugin.eligiblecCs(t|ƒr|jd7_|jd7_dSrF)rrIr6r!r#r#r$r&“szUnprintablePlugin.feedcCs d|_dSr>)rIr(r#r#r$r)˜szUnprintablePlugin.resetcCs|jdkrdS|jd|jS)Nrr?rG)r6rIr(r#r#r$r*›s zUnprintablePlugin.ratiorAr#r#r#r$rH‹s rHc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚSuspiciousDuplicateAccentPluginNr'cCsd|_d|_d|_dSr>©Ú_successive_countr6Ú_last_latin_characterr(r#r#r$r8¤sz(SuspiciousDuplicateAccentPlugin.__init__rcCs| ¡ot|ƒSr9)rErr!r#r#r$r%ªsz(SuspiciousDuplicateAccentPlugin.eligiblecCst|jd7_|jdurjt|ƒrjt|jƒrj| ¡rJ|j ¡rJ|jd7_t|ƒt|jƒkrj|jd7_||_dSrF)r6rNr ÚisupperrMrr!r#r#r$r&­sÿþýz$SuspiciousDuplicateAccentPlugin.feedcCsd|_d|_d|_dSr>rLr(r#r#r$r)»sz%SuspiciousDuplicateAccentPlugin.resetcCs|jdkrdS|jd|jS)Nrr?r<)r6rMr(r#r#r$r*Às z%SuspiciousDuplicateAccentPlugin.ratiorAr#r#r#r$rK£s rKc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚSuspiciousRangeNr'cCsd|_d|_d|_dSr>)Ú"_suspicious_successive_range_countr6Ú_last_printable_seenr(r#r#r$r8ÉszSuspiciousRange.__init__rcCs| ¡Sr9r:r!r#r#r$r%ÎszSuspiciousRange.eligiblecCsx|jd7_| ¡s&t|ƒs&|tvr0d|_dS|jdurD||_dSt|jƒ}t|ƒ}t||ƒrn|jd7_||_dSrF)r6ÚisspacerrrRrÚ is_suspiciously_successive_rangerQ)r"rÚunicode_range_aÚunicode_range_br#r#r$r&Ñs"ÿþý   zSuspiciousRange.feedcCsd|_d|_d|_dSr>)r6rQrRr(r#r#r$r)èszSuspiciousRange.resetcCs"|jdkrdS|jd|j}|S)Nér?r<)r6rQ)r"Zratio_of_suspicious_range_usager#r#r$r*ís  þzSuspiciousRange.ratiorAr#r#r#r$rPÈs rPc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚSuperWeirdWordPluginNr'cCs:d|_d|_d|_d|_d|_d|_d|_d|_d|_dS)NrFÚ) Ú _word_countÚ_bad_word_countÚ_foreign_long_countÚ_is_current_word_badÚ_foreign_long_watchr6Ú_bad_character_countÚ_bufferÚ_buffer_accent_countr(r#r#r$r8úszSuperWeirdWordPlugin.__init__rcCsdSrJr#r!r#r#r$r%szSuperWeirdWordPlugin.eligiblecCs6| ¡r�|j|7_t|ƒr,|jd7_|jdurŒt|ƒdusJt|ƒrŒt|ƒdurŒt|ƒdurŒt|ƒdurŒt |ƒdurŒt |ƒdurŒd|_dS|jsšdS|  ¡s´t |ƒs´t |ƒ�rü|j�rü|jd7_t|jƒ}|j|7_|dk�rP|j|dk�rd|_t|jdƒ�rP|jd ¡�rPtdd„|jDƒƒdu�rP|jd7_d|_|d k�r¸|j�r¸d d „t|jtd |ƒƒDƒ}d}|�ržt|ƒ|d k�ržd}|�s¸|jd7_d|_|j�rè|jd7_|jt|jƒ7_d|_d|_d|_d |_n6|dv�r2| ¡du�r2t|ƒ�r2d|_|j|7_dS)NrFTégÃõ(\�ÂÕ?éÿÿÿÿcss|]}| ¡VqdSr9©rO)Ú.0Ú_r#r#r$Ú -óz,SuperWeirdWordPlugin.feed..rWcSsg|]\}}| ¡r|‘qSr#rd)reÚcÚir#r#r$Ú 2sþz-SuperWeirdWordPlugin.feed..rr@rY>rfú=ú-ú|ú~ú<ú>)rEr`r rar^rrrrrrrSrrrZÚlenr6r]rOÚallr\ÚzipÚranger[r_r=r)r"rZ buffer_lengthZcamel_case_dstZprobable_camel_casedr#r#r$r& sŽÿ þþ ý ü û ú ù ÿÿÿþ   ÿ þýþÿ þýzSuperWeirdWordPlugin.feedcCs4d|_d|_d|_d|_d|_d|_d|_d|_dS)NrYFr)r`r]r^r[rZr6r_r\r(r#r#r$r)PszSuperWeirdWordPlugin.resetcCs$|jdkr|jdkrdS|j|jS)Né rr?)rZr\r_r6r(r#r#r$r*ZszSuperWeirdWordPlugin.ratiorAr#r#r#r$rXùs E rXc@s^eZdZdZddœdd„Zeedœdd„Zeddœd d „Zddœd d „Z e e dœd d„ƒZ dS)ÚCjkInvalidStopPluginu² GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and can be easily detected. Searching for the overuse of '丅' and '丄'. Nr'cCsd|_d|_dSr>©Ú_wrong_stop_countÚ_cjk_character_countr(r#r#r$r8hszCjkInvalidStopPlugin.__init__rcCsdSrJr#r!r#r#r$r%lszCjkInvalidStopPlugin.eligiblecCs4|dvr|jd7_dSt|ƒr0|jd7_dS)N>u丅u丄r)ryrrzr!r#r#r$r&os zCjkInvalidStopPlugin.feedcCsd|_d|_dSr>rxr(r#r#r$r)vszCjkInvalidStopPlugin.resetcCs|jdkrdS|j|jS)Nér?)rzryr(r#r#r$r*zs zCjkInvalidStopPlugin.ratio) r+r,r-r.r8r/r0r%r&r)r1r2r*r#r#r#r$rwbsrwc@sZeZdZddœdd„Zeedœdd„Zeddœdd „Zddœd d „Ze e dœd d „ƒZ dS)ÚArchaicUpperLowerPluginNr'cCs.d|_d|_d|_d|_d|_d|_d|_dS)NFrT)Ú_bufÚ_character_count_since_last_sepÚ_successive_upper_lower_countÚ#_successive_upper_lower_count_finalr6Ú_last_alpha_seenÚ_current_ascii_onlyr(r#r#r$r8‚sz ArchaicUpperLowerPlugin.__init__rcCsdSrJr#r!r#r#r$r%�sz ArchaicUpperLowerPlugin.eligiblecCs$| ¡ot|ƒ}|du}|r†|jdkr†|jdkrV| ¡durV|jdurV|j|j7_d|_d|_d|_d|_|j d7_ d|_dS|jdur¢|  ¡dur¢d|_|jdurþ|  ¡r¾|j  ¡sÐ|  ¡rø|j  ¡rø|jdurð|jd7_d|_qþd|_nd|_|j d7_ |jd7_||_dS)NFré@rTr<) rEr r~r=r‚r€rr�r}r6ÚisasciirOÚislower)r"rZ is_concernedZ chunk_sepr#r#r$r&’sFÿ þýÿ ÿÿ zArchaicUpperLowerPlugin.feedcCs.d|_d|_d|_d|_d|_d|_d|_dS)NrFT)r6r~rr€r�r}r‚r(r#r#r$r)¼szArchaicUpperLowerPlugin.resetcCs|jdkrdS|j|jS)Nrr?)r6r€r(r#r#r$r*Ås zArchaicUpperLowerPlugin.ratiorAr#r#r#r$r|�s  * r|c@sZeZdZddœdd„Zddœdd„Zeedœdd „Zeddœd d „Ze e dœd d „ƒZ dS)ÚArabicIsolatedFormPluginNr'cCsd|_d|_dSr>©r6Ú_isolated_form_countr(r#r#r$r8Îsz!ArabicIsolatedFormPlugin.__init__cCsd|_d|_dSr>r‡r(r#r#r$r)ÒszArabicIsolatedFormPlugin.resetrcCst|ƒSr9)r r!r#r#r$r%Ösz!ArabicIsolatedFormPlugin.eligiblecCs(|jd7_t|ƒr$|jd7_dSrF)r6r rˆr!r#r#r$r&ÙszArabicIsolatedFormPlugin.feedcCs|jdkrdS|j|j}|S)NrGr?r‡)r"Zisolated_form_usager#r#r$r*ßs  zArabicIsolatedFormPlugin.ratio) r+r,r-r8r)r/r0r%r&r1r2r*r#r#r#r$r†Ís r†é)Úmaxsize)rUrVrcCs”|dus|durdS||kr dSd|vr4d|vr4dSd|vsDd|vrHdSd|vsXd|vrld|vshd|vrldS| d¡| d¡}}|D]}|tvr”q†||vr†dSq†|dv|dv}}|s¾|rÒd |vsÎd |vrÒdS|rÞ|rÞdSd |vsðd |v�r d |v�sd |v�rdS|d k�s|d k�r dSd |v�sHd |v�sH|d v�r�|d v�r�d |v�s\d |v�r`dSd|v�std|v�rxdS|d k�sŒ|d k�r�dSdS)za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFZLatinZ EmoticonsZ Combiningú )ÚHiraganaÚKatakanaZCJKZHangulz Basic Latin)r�rŒZ PunctuationZForms)Úsplitr )rUrVZkeywords_range_aZkeywords_range_bÚelZrange_a_jp_charsZrange_b_jp_charsr#r#r$rTéslÿÿÿþÿúÿÿÿþrTiçš™™™™™É?F)Údecoded_sequenceÚmaximum_thresholdÚdebugrc CsXdd„t ¡Dƒ}t|ƒd}d}|dkr0d}n|dkr>d}nd }t|d t|ƒƒD]d\}}|D]} |  |¡r`|  |¡q`|d krŽ||d ksš||dkrTtd d „|Dƒƒ}||krTqºqT|�rNtdƒ} |   t d|›d|›d|›�¡t|ƒdk�r(|   t d|dd…›�¡|   t d|dd…›�¡|D] } |   t | j ›d| j ›�¡�q,t |dƒS)zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. cSsg|] }|ƒ‘qSr#r#)reZmd_classr#r#r$rk:szmess_ratio..rr?ié r‰rƒé€Ú rcss|] }|jVqdSr9)r*)reÚdtr#r#r$rgQrhzmess_ratio..Zcharset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=r{zStarting with: Nz Ending with: iðÿÿÿz: é)rÚ__subclasses__rrrtrur%r&ÚsumrÚlogrÚ __class__r*Úround) r‘r’r“Z detectorsÚlengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcrÚindexZdetectorÚloggerr—r#r#r$Ú mess_ratio2sRÿ   ÿ ÿ þÿÿþþr¡N)r�F),Ú functoolsrZloggingrÚtypingrrZconstantrrr Zutilsr r r r rrrrrrrrrrrrrrr3rBrHrKrPrXrwr|r†r/r0rTr2r¡r#r#r#r$Ús0  L"/%1iL þHÿþ