| | |
- exceptions.Exception(exceptions.BaseException)
-
- ScrapeError
- Region
- Session
class Region |
| |
A Region object represents a contiguous region of a document (in terms
of a starting and ending position in the document string) together with
an associated HTML or XML tag and its attributes. Dictionary-like access
retrieves the name-value pairs in the attributes. Various other methods
allow slicing up a Region into subregions and searching within, before,
or after a Region for tags or elements. For a Region that represents a
single tag, the starting and ending positions are the start and end of
the tag itself. For a Region that represents an element, the starting
and ending positions are just after the starting tag and just before the
ending tag, respectively.
|
| |
Methods defined here:
- __contains__(self, name)
- # Provide a dictionary-like interface to the tag attributes.
- __getitem__(self, name)
- __getslice__(self, start, end)
- # Access subregions by slicing. The starting and ending positions of a
# slice can be given as string positions within the region (just like
# slicing a string), or as regions. A slice between two regions begins
# at the end of the start region and ends at the start of the end region.
- __init__(self, parent, start=0, end=None, starttag=None, endtag=None)
- Create a Region. The 'parent' argument is a string or another
Region. The 'start' and 'end' arguments, if given, are non-negative
indices into the original string (not into the parent region). The
'starttag' and 'endtag' arguments are indices into an internal array
of tags, intended for use by the implementation only.
- __len__(self)
- # Report the length of the region.
- __repr__(self)
- __str__(self)
- after(self)
- Return the Region for everything after this Region.
- all(self, tagname=None, content=None, enders=[], **attrs)
- Return Regions for all non-overlapping elements entirely within
this Region with the given tag name, content, and attributes, where
each element ends at a balanced matching end tag or any tag listed
in 'enders'.
- alltags(self, tagname=None, **attrs)
- Return a list of Regions for all the tags entirely within this
Region with the given tag name and attributes.
- before(self)
- Return the Region for everything before this Region.
- enclosing(self, tagname=None, content=None, enders=[], **attrs)
- Return the Region for the nearest element that encloses this Region
with the given tag name, content, and attributes. The element ends at
ends at a balanced matching end tag or any tag listed in 'enders'.
- find(self, target, group=0)
- Search this Region for a string or a compiled RE and return a
Region representing the match. If 'group' is given, it specifies
which grouped subexpression should be returned as the match.
- findall(self, target, group=0)
- Search this Region for a string or a compiled RE and return a
sequence of Regions representing all the matches.
- findendtag(self, starttag, enders=[], outside=0)
- Find the index of the tag that ends an element, given the index of
its start tag, by scanning for a balanced matching end tag or a tag
whose name is in 'enders'. 'enders' may contain plain tag names (for
start tags) or tag names prefixed with '/' (for end tags). If
'outside' is 0, scan within the current region; if 'outside' is 1,
scan starting from the end of the current region onwards.
- first(self, tagname=None, content=None, enders=[], **attrs)
- Return the Region for the first element entirely within this Region
with the given tag name, content, and attributes. The element ends at
a balanced matching end tag or any tag listed in 'enders'. 'enders' may
may contain plain tag names (for start tags) or tag names prefixed with
'/' (for end tags). The element content is passed through striptags()
for comparison. If 'content' has a match() method, the stripped content
is passed to it; otherwise it is compared directly as a string.
- firsttag(self, tagname=None, **attrs)
- Return the Region for the first tag entirely within this Region
with the given tag name and attributes.
- get(self, name, default)
- get_buttons(self)
- Get a list of all the form submission buttons.
- get_number(self)
- # Provide access to numeric content.
- get_params(self)
- Get a dictionary of default values for all the form parameters.
- keys(self)
- last(self, tagname=None, content=None, enders=[], **attrs)
- Return the Region for the last element entirely within this Region
with the given tag name, content, and attributes. The element ends at
a balanced matching end tag or at any tag listed in 'enders'.
- lasttag(self, tagname=None, **attrs)
- Return the Region for the last tag entirely within this Region
with the given tag name and attributes.
- matchelement(self, starttag, content=None, enders=[], outside=0)
- If the element with the given start tag matches the given content,
return the index of the tag that ends the element. The end of the
element is found by scanning for either a balanced matching end
tag or tag whose name is in 'enders'. 'enders' may contain plain tag
names (for start tags) or tag names prefixed with '/' (for end tags).
If 'outside' is 0, scan within the current region; if 'outside' is 1,
scan starting from the end of the current region onwards.
- matchtag(self, i, tagname, attrs)
- Return 1 if the ith tag matches the given tagname and attributes.
- next(self, tagname=None, content=None, enders=[], **attrs)
- Return the Region for the nearest element after the end of this
Region with the given tag name, content, and attributes. The element
ends at a balanced matching end tag or any tag listed in 'enders'.
- nexttag(self, tagname=None, **attrs)
- Return the Region for the nearest tag after the end of this Region
with the given tag name and attributes.
- previous(self, tagname=None, content=None, enders=[], **attrs)
- Return the Region for the nearest element before the start of this
Region with the given tag name, content, and attributes. The element
ends at a balanced matching end tag or any tag listed in 'enders'.
- previoustag(self, tagname=None, **attrs)
- Return the Region for the nearest tag before the start of this
Region with the given tag name and attributes.
- scantags(self, document)
- Generate a list of all the tags in a document.
- split(self, separator)
- Find all occurrences of the given string or compiled RE and use
them as separators to split this Region into a sequence of Regions.
- splittag(self, tagname=None, **attrs)
- Split this Region into a list of the subregions separated by tags
with the given tag name and attributes.
Data descriptors defined here:
- buttons
- Get a list of all the form submission buttons.
- content
- number
- params
- Get a dictionary of default values for all the form parameters.
- text
|
class Session |
| |
A Web-browsing session. Exposed attributes:
agent - the User-Agent string (clients can set this attribute)
url - the last successfully fetched URL
status - the status code of the last request
message - the status message of the last request
headers - the headers of the last request as a dictionary
content - the content of the last fetched document
doc - the Region spanning the last fetched document
|
| |
Methods defined here:
- __init__(self, agent=None, verbose=0)
- Specify 'agent' to set the User-Agent. Set 'verbose' to 1, 2, or
3 to display status messages on stderr during document retrieval.
- back(self)
- Restore the state of this session before the previous request.
- follow(self, anchor, region=None)
- Find the first link that has the given anchor text, and follow it.
The anchor may be given as a string or a compiled RE. If 'region' is
specified, only that region is searched for a matching link, instead
of the whole document.
- go(self, url, data='', redirects=10, referrer=True, charset=None)
- Navigate to a given URL. If the URL is relative, it is resolved
with respect to the current URL. If 'data' is provided, do a POST;
otherwise do a GET. Follow redirections up to 'redirects' times.
If 'referrer' is given, send it as the referrer; if 'referrer' is
True (default), send the current URL as the referrer; if 'referrer'
is a false value, send no referrer. If 'charset' is given, it
overrides any received 'charset' parameter; setting 'charset' to RAW
leaves the content undecoded in an 8-bit string. If the document is
successfully fetched, return a Region spanning the entire document.
Any relevant previously stored cookies will be included in the
request, and any received cookies will be stored for future use.
- resolve(self, url)
- Resolve a URL with respect to the current location.
- setcookie(self, cookieline)
- Put a cookie in this session's cookie jar. 'cookieline' should
have the format "<name>=<value>; domain=<domain>; path=<path>".
- submit(self, region, paramdict=None, url=None, redirects=10, **params)
- Submit a form, optionally by clicking a given button. The 'region'
argument can be the form itself or a button in the form to click.
Obtain the parameters to submit by (a) starting with the 'paramdict'
dictionary if specified, or the default parameter values as returned
by get_params; then (b) adding or replacing parameters in this
dictionary according to the keyword arguments. The 'url' argument
overrides the form's action attribute and submits the form elsewhere.
After submission, follow redirections up to 'redirects' times.
| |