ƁKc @sdZdZdididdZdZddklZlZdd k l Z dd k Z dd k Z d Z d Zd efdYZdddZdZeeeeddZdZdZeZdeeedhdZdfdYZdZdZedZe ieZe idZ dZ!dZ"e#de$d DZ%e%i&e#d!d"d#d$Dd%Z'd&Z(d'Z)e ie)eZ*e ie)ed(e i+Z,e ie)ed)e i+Z-e id*e i+Z.he id+d,<e id-e i+d.<e id/e i+d0Z>d?Z?eZ@d S(@sPython module for web browsing and scraping. Done: - navigate to absolute and relative URLs - follow links in page or region - find strings or regular expressions: first, all, split - find tags: first, last, next, previous, all, splittag - find elements: first, last, next, previous, enclosing, all - set form fields - submit forms - strip tags from arbitrary strings of HTML - support HTTPS - handle entities > 255 and Unicode documents - accept and store cookies during redirection - store and send cookies according to domain and path To do: - split by element - detect ends of elements in most cases even if matching end tags are missing - make the line breaks in striptags correctly reflect whitespace significance - handle - submit forms with file upload - use Regions in striptags instead of duplicating work? - remove dependency on urllib.urlencode sKa-Ping Yee s$Date: 2010-02-21 23:49:22 $it/t-s$Revision: 1.48 $i(turlsplitturljoin(tname2codepointNcOsX|idti}x0t|D]"\}}|id||}q"Wti||S(s'Compile a regular expression, substituting in any passed parameters for placeholders of the form __0__, __1__, __2__, etc. in the template. Specify the named argument 'flags' to set regular expression compilation flags; by default, DOTALL is set ('.' matches anything including ' ').tflagss__%d__(tgettretDOTALLt enumeratetreplacetcompile(ttemplatetparamstkwargsRtitparam((s ./scrape.pytregex1s  cOs-|iddtiB|d ( tosttempfiletmkstempt isinstanceR!t urlencodeRFR%R&tsystemtopenR1tremove( turlR8tagenttreferrertcookiesR9RIRJtfdttempnametcommandtreply((s ./scrape.pytcurlis(  cCsh}x~|D]v}d|i|o\xY||D]I}|i|o3x0|||iD]\}}|||tpairsR;R<tattrs((s ./scrape.pyt setcookiess @!cCst|\}}} } } |idd}| pd} |odpd} |o!t|t ot|}ndig} t||| iD]}| d|q~ }|oti dI|IJndd k }|d jp|d jot |d o| o| d | 7} nh|d<dd<}|od|dd|ijo.t ||idd}}|d8}qVqnPqV|i i|t|i|_|iS(s)Navigate to a given URL. If the URL is relative, it is resolved with respect to the current URL. If 'data' is provided, do a POST; otherwise do a GET. Follow redirections up to 'redirects' times. If 'referrer' is given, send it as the referrer; if 'referrer' is True (default), send the current URL as the referrer; if 'referrer' is a false value, send no referrer. If 'charset' is given, it overrides any received 'charset' parameter; setting 'charset' to RAW leaves the content undecoded in an 8-bit string. If the document is successfully fetched, return a Region spanning the entire document. Any relevant previously stored cookies will be included in the request, and any received cookies will be stored for future use.i-i.tlocationRi(i-i.(RQRRR7RRtresolvetTrueRRRR9R]RRR0tRegion(RRQR8t redirectsRSR}t historyentry((s ./scrape.pytgo s"   *   cCs:|ii\|_|_|_|_|_|_|iS(s>Restore the state of this session before the previous request.(RtpopRQRRR7RR(R((s ./scrape.pytback,s3cCsr|p|iidd|}|ptd|n|iddptd|n|i|dS(sFind the first link that has the given anchor text, and follow it. The anchor may be given as a string or a compiled RE. If 'region' is specified, only that region is searched for a matching link, instead of the whole document.taRslink %r not foundthrefRslink %r has no href(RtfirstRRR(Rtanchortregiontlink((s ./scrape.pytfollow2s c Ks<|idjo|p |id}|ptd|n|d j o|i}n |i}d|jo|idd||ds &     % cCsE|io7|idp |id ot|i|}n|S(s3Resolve a URL with respect to the current location.shttp://shttps://(RQR\R(RRQ((s ./scrape.pyRZs !cCsXt|i\}}}}}|iddidd}t|i||gdS(sPut a cookie in this session's cookie jar. 'cookieline' should have the format "=; domain=; path=".RiRiN(RRQRRlR](Rt cookielineR3R5R6R~R((s ./scrape.pyt setcookieas N( RRt__doc__RRRRRRRRR(((s ./scrape.pyRs  !   s7(('[^']*'|"[^"]*"|--([^-]+|-[^-]+)*--|-(?!-)|[^'">-])*)cCsd|tdS(Nt(t tagcontent_re(t tagname_re((s ./scrape.pyttag_relss(\?|!\w*|/?[a-zA-Z_:][\w:.-]*)s!&(#(\d+|x[\da-fA-F]+)|[\w.:-]+);?cCsBt|tjo t}n d}|d}ti||S(s'Decode HTML entities in the given text.cSs$|djo t|p t|S(i(tunichrtchr(R<((s ./scrape.pyt{scSs|id}|ido|t|ddSnU|ido|t|dSn-|tjo|t|Sn|idSdS(Nis#xiit#i(tgroupR\R(R(Rtuchrtentity((s ./scrape.pyt entitydecode|s (ttypetunicodeRt charrefpattsub(RERR((s ./scrape.pyt htmldecodevs    cCsL|idd}|idd}|idd}|idd}|S( sAUse HTML entities to encode special characters in the given text.t&s&t"s"Rs<Rs>(R (RE((s ./scrape.pyt htmlencodes ccs)x"|]}t|d|fVqWdS(s%%%02XN(R(t.0R((s ./scrape.pys s iccsx|]}||fVqWdS(N((Rtc((s ./scrape.pys s tABCDEFGHIJKLMNOPQRSTUVWXYZtabcdefghijklmnopqrstuvwxyzs 0123456789._-cCsditti|S(NR(Rtmapt urlquotedR(RE((s ./scrape.pyturlquotescCs\g}|iD]5\}}|dt|t|iddfq~}di|S(Ns%s=%ss%20t+R(RRR R(R R=R`R<Rj((s ./scrape.pyRMsHcCs|iddiddS(Nt(s(?:s(?:?s(?(R (R((s ./scrape.pyt no_groupsss(p|table|form)s (div|br|tr)s<(!\s*--|style\b|script\b)s--\s*>t!s]*>tscripts ]*>tstylec Csgd}}xti||}|pPn|ididi}|iid}t|i||i}|pPn|i |||i !|i}q|i ||di |}g}xt i |D]}g}xti |D]x} di ti | } t| } t| tjodpd} | i| d} |i di | i qWd i |}t|tjo.y|id Wq|id }qXn|i |qWtid d d i |iS(sStrip HTML tags from the given string, yielding line breaks for DIV, BR, or TR tags and blank lines for P, TABLE, or FORM tags.iiRsutf-8Ru sRs tasciislatin-1s +s (tcdatapatRwRR+ReRztencodet endcdatapatRyR0RxRt parasplitterRt linesplittert tagsplitterRRRR R!R|RR( thtmlRCtpost startmatchRtendmatcht paragraphst paragraphRhRitnbsp((s ./scrape.pyt striptagssB    s0\s*([\w:.-]+)(\s*=\s*('[^']*'|"[^"]*"|[^\s>]*))?c Csh}d}xti||}|pPn|i}|id|idpd}}|d djo|dd!}nyt|}WnnXyt|}WnnXt|||is(RRRkRRxRy(RR=titemRk((s ./scrape.pyt__repr__?s  7cCs|iS(N(R(R((s ./scrape.pyt__str__Gsc Csg}d}xti||}|pPn|i\}}|idiid}|id}|i||||g|d joUt|i||}|pPn|i\}}|i||d|dgn|}q|S( s.Generate a list of all the tags in a document.iisutf-8iRRRR(RR(ttagpatRwtspanRRzRR0R( RRRRRRxRyRRk((s ./scrape.pyRKs$ ! cCs|i|d|i|d}}|d d jo||jp |djojt|toI|d d joh|i|d<}qt||i|d<}nt||SqndS( sAReturn 1 if the ith tag matches the given tagname and attributes.iiiRRRpRN(RR(RpR(RRRLRRR(RRRRktitagnametiattrs((s ./scrape.pytmatchtag`s#cCs(t|to|i}n|i|d}d}xt|dt|iD]}|i|dd|jo|d8}n|djp%|djo\|i|d|joD| o||ijo|Sn|o||ijo|SnPn|i|d|jo|d7}qTqTWdS(sFind the index of the tag that ends an element, given the index of its start tag, by scanning for a balanced matching end tag or a tag whose name is in 'enders'. 'enders' may contain plain tag names (for start tags) or tag names prefixed with '/' (for end tags). If 'outside' is 0, scan within the current region; if 'outside' is 1, scan starting from the end of the current region onwards.iiRiN(RLRRRtrangeR/R(RRtenderstoutsideRtdepthR((s ./scrape.pyt findendtagls 2cCs|i|||}|dj o\|i|d|i|d}}|djp tt|i||!|o|Sq~ndS(sIf the element with the given start tag matches the given content, return the index of the tag that ends the element. The end of the element is found by scanning for either a balanced matching end tag or tag whose name is in 'enders'. 'enders' may contain plain tag names (for start tags) or tag names prefixed with '/' (for end tags). If 'outside' is 0, scan within the current region; if 'outside' is 1, scan starting from the end of the current region onwards.iiN(RRRRRR(RRRRRRRxRy((s ./scrape.pyt matchelements  #cCs|i|i|i!S((RRxRy(R((s ./scrape.pyRscCs t|iS((RR(R((s ./scrape.pyRsc Cs|idjoh}x|idD]}d|jond|joa|iddi}|djp|djo+d |jo|id d ||d|i|i}|o!||i||i|!Sqn9|ii|}|djo|||t|!Sntd|dS(sSearch this Region for a string or a compiled RE and return a Region representing the match. If 'group' is given, it specifies which grouped subexpression should be returned as the match.Rwisno match found for %rN(R.RwRRxRytfindR/R(RttargetRRRx((s ./scrape.pyR#s% cCsd}|i}g}t|doQx|i||}|pPn|i|\}}|i|||!q(nOxK|i||}|djoPn|t|}|i|||!qy|S(swSearch this Region for a string or a compiled RE and return a sequence of Regions representing all the matches.iRw(RR.RwRR0R#R/(RR$RRRtmatchesRRx((s ./scrape.pytfindalls"  cCsd}|i}g}t|dohxP|i||}|pPn|id\}}|i|||!|}q(|i||nfxQ|i||}|djoPn|t|}|i|||!|}q|i|||S(sFind all occurrences of the given string or compiled RE and use them as separators to split this Region into a sequence of Regions.iRw(RR.RwRR0R#R/(Rt separatorRRR%RRxRy((s ./scrape.pyRs*    cKsxNt|i|idD]3}|i|||ot|dd|SqqW|djodpd|}|o d|pd}td||fdS( soReturn the Region for the first tag entirely within this Region with the given tag name and attributes.iiRs<%s> tags matching %rRs no %s found%sN(RRRRRRR(RRRkRRR((s ./scrape.pytfirsttag6scKsxQt|i|iddD]3}|i|||ot|dd|SqqW|d jodpd|}|o d|pd}td||fd S( snReturn the Region for the last tag entirely within this Region with the given tag name and attributes.iiiRs<%s> tags matching %rRs no %s found%sN(RRRRRRR(RRRkRRR((s ./scrape.pytlasttag@scKsdg}xWt|i|idD]<}|i|||o |it|dd|q q W|S(suReturn a list of Regions for all the tags entirely within this Region with the given tag name and attributes.ii(RRRRR0R(RRRkRR((s ./scrape.pyR Js $cKs|ii||S(srReturn the Region for the nearest tag after the end of this Region with the given tag name and attributes.(R!R((RRRk((s ./scrape.pytnexttagSscKs|ii||S(suReturn the Region for the nearest tag before the start of this Region with the given tag name and attributes.(R"R)(RRRk((s ./scrape.pyt previoustagXscKsYgd}}x4|i||D] }|i|||!|}q W|i|||S(sqSplit this Region into a list of the subregions separated by tags with the given tag name and attributes.i(R R0(RRRkt subregionsRxR((s ./scrape.pytsplittag]s  c Ksxwt|i|idD]\}|i|||o@|i|||}|d j ot|dd||SqvqqW|d jodpd|}|o d|pd}|d j o d|pd} td||| fd S( sReturn the Region for the first element entirely within this Region with the given tag name, content, and attributes. The element ends at a balanced matching end tag or any tag listed in 'enders'. 'enders' may may contain plain tag names (for start tags) or tag names prefixed with '/' (for end tags). The element content is passed through striptags() for comparison. If 'content' has a match() method, the stripped content is passed to it; otherwise it is compared directly as a string.iitelements <%s> elements matching %rRs with content %rsno %s found%s%sN(RRRRRRRR( RRRRRkRRRRR((s ./scrape.pyRhs "c Ksxzt|i|iddD]\}|i|||o@|i|||}|d j ot|dd||SqyqqW|d jodpd|}|o d|pd}|d j o d|pd} td ||| fd S( sReturn the Region for the last element entirely within this Region with the given tag name, content, and attributes. The element ends at a balanced matching end tag or at any tag listed in 'enders'.iiiR.s <%s> elements matching %rRs with content %rsno %s found%s%sN(RRRRRRRR( RRRRRkRRRRR((s ./scrape.pytlastzs "cKsg}|i}x||ijow|i|||oS|i|||}|dj o-|it|dd|||d}qn|d7}qW|S(sReturn Regions for all non-overlapping elements entirely within this Region with the given tag name, content, and attributes, where each element ends at a balanced matching end tag or any tag listed in 'enders'.iiN(RRRRRR0R(RRRRRktelementsRR((s ./scrape.pyR s  cKs|ii||||S(sReturn the Region for the nearest element after the end of this Region with the given tag name, content, and attributes. The element ends at a balanced matching end tag or any tag listed in 'enders'.(R!R(RRRRRk((s ./scrape.pytnextscKs|ii||||S(sReturn the Region for the nearest element before the start of this Region with the given tag name, content, and attributes. The element ends at a balanced matching end tag or any tag listed in 'enders'.(R"R/(RRRRRk((s ./scrape.pytprevioussc Ks|io|io|id}n|id}xvt|ddD]b}|i|||oF|i|||dd}|d j ot|dd||SqqEqEW|d jodpd|}|o d|pd} |d j o d |pd} td || | fd S( sReturn the Region for the nearest element that encloses this Region with the given tag name, content, and attributes. The element ends at ends at a balanced matching end tag or any tag listed in 'enders'.iiRiR.s <%s> elements matching %rRs with content %rsno %s found%s%sN( RRRRRRRRR( RRRRRkt laststarttagRRRRR((s ./scrape.pyRs  "N(+RRRRRRRRRRRtpropertyRRERRR tbuttonsRtnumberRRRRRRR!R"R#R&RR(R)R R*R+R-RR/R R1R2R(((s ./scrape.pyRsN $                        cCst|iS(sBRead and return the entire contents of the file at the given path.(ROR1(R6((s ./scrape.pyR1scCs*t|d}|i||idS(s1Write the given text to a file at the given path.twN(ROR-tclose(R6RERB((s ./scrape.pyR-s cCsddk}|it|S(s?Return the deserialized contents of the file at the given path.iN(tmarshaltloadRO(R6R9((s ./scrape.pyR:s cCs9ddk}t|d}|i|||idS(sBSerialize the given data and write it to a file at the given path.iNR7(R9ROtdumpR8(R6R8R9RB((s ./scrape.pyR;s cCs|xet|iddiddiD]9}yt|SWq+yt|SWqdq+qdXq+Xq+Wtd|dS(sFind and parse an integer or floating-point number in the given text, ignoring commas, percentage signs, and non-numeric words.t,Rt%Rsno number found in %rN(RR RR(tfloatR(REtword((s ./scrape.pyRs+(ARt __author__RR t__date__t __version__turlparseRRthtmlentitydefsRR%RRRt ExceptionRR:RFRRYRaRltobjectR{RRRRt anytag_reR RRRRRfRRRRRMRRtIRRRRRtattr_reRRRtPRESENTtANYRRRRR1R-R:R;RRt(((s ./scrape.pys'sr    \u       !! )