PNG  IHDR;IDATxܻn0K )(pA 7LeG{ §㻢|ذaÆ 6lذaÆ 6lذaÆ 6lom$^yذag5bÆ 6lذaÆ 6lذa{ 6lذaÆ `}HFkm,mӪôô! x|'ܢ˟;E:9&ᶒ}{v]n&6 h_tڠ͵-ҫZ;Z$.Pkž)!o>}leQfJTu іچ\X=8Rن4`Vwl>nG^is"ms$ui?wbs[m6K4O.4%/bC%t Mז -lG6mrz2s%9s@-k9=)kB5\+͂Zsٲ Rn~GRC wIcIn7jJhۛNCS|j08yiHKֶۛkɈ+;SzL/F*\Ԕ#"5m2[S=gnaPeғL lذaÆ 6l^ḵaÆ 6lذaÆ 6lذa; _ذaÆ 6lذaÆ 6lذaÆ RIENDB`  oY4@sddZddlZddlZdgZGdddZGdddZGdddZdS) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt NRobotFileParserc@seZdZdZdddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ dS)rzs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. cCs>g|_d|_d|_d|_|j|d|_dS)NFr)entries default_entry disallow_all allow_allset_url last_checked)selfurlr */opt/python35/lib/python3.5/robotparser.py__init__s      zRobotFileParser.__init__cCs|jS)zReturns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r )r r r r mtimeszRobotFileParser.mtimecCsddl}|j|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)timer )r rr r r modified(s zRobotFileParser.modifiedcCs5||_tjj|dd\|_|_dS)z,Sets the URL referring to a robots.txt file.N)r urllibparseurlparsehostpath)r r r r r r0s zRobotFileParser.set_urlcCsytjj|j}Wnmtjjk r}zG|jdkrOd|_n'|jdkrv|jdkrvd|_WYdd}~Xn)X|j }|j |j dj dS) z4Reads the robots.txt URL and feeds it to the parser.TiiNzutf-8)rr) rZrequesturlopenr error HTTPErrorcoderrreadrdecode splitlines)r ferrrawr r r r5s  zRobotFileParser.readcCs>d|jkr*|jdkr:||_n|jj|dS)N*) useragentsrrappend)r entryr r r _add_entryBs zRobotFileParser._add_entrycCsd}t}|jx|D]}|sr|dkrJt}d}n(|dkrr|j|t}d}|jd}|dkr|d|}|j}|sq |jdd}t|dkr |djj|ds z+RobotFileParser.__str__..)joinr)r r r r __str__szRobotFileParser.__str__N) __name__ __module__ __qualname____doc__rrrrrr)rrArGr r r r rs    4 c@s:eZdZdZddZddZddZdS) r5zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cCsY|dkr| rd}tjjtjj|}tjj||_||_dS)NrT)rrr;rr=rr?)r rr?r r r rs zRuleLine.__init__cCs|jdkp|j|jS)Nr%)r startswith)r filenamer r r r>szRuleLine.applies_tocCs|jrdndd|jS)NZAllowZDisallowz: )r?r)r r r r rGszRuleLine.__str__N)rHrIrJrKrr>rGr r r r r5s   r5c@sFeZdZdZddZddZddZdd Zd S) r-z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_dS)N)r&r4)r r r r rs zEntry.__init__cCsjg}x'|jD]}|jd|dgqWx*|jD]}|jt|dgq:Wdj|S)Nz User-agent: rBr)r&extendr4rCrF)r retagentr8r r r rGs z Entry.__str__cCs]|jddj}x=|jD]2}|dkr9dS|j}||kr#dSq#WdS)z2check if this entry applies to the specified agentr:rr%TF)r0r2r&)r r@rPr r r r>s   zEntry.applies_tocCs.x'|jD]}|j|r |jSq WdS)zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r4r>r?)r rMr8r r r r?s zEntry.allowanceN)rHrIrJrKrrGr>r?r r r r r-s    r-)rKZ urllib.parserZurllib.request__all__rr5r-r r r r  s