htmlscraper.py (8145B)
1 #Embedded file name: C:\BuildAgent\work\20d73f1b24eed06e\eve\release\V24.01\packages\carbonui\control\editutils\htmlscraper.py 2 import re 3 import blue 4 import telemetry 5 namefind = re.compile('(\\S+)\\s*(.*)', re.DOTALL) 6 attrfind = re.compile('\\s*([a-zA-Z_][-:.a-zA-Z_0-9]*)(\\s*=\\s*(\\\'[^\\\']*\\\'|"[^"]*"|[-a-zA-Z0-9./,:;+*%?!&$\\(\\)_#=~\\\'"@]*))?') 7 8 class HtmlScraper: 9 __guid__ = 'hparser.HtmlScraper' 10 11 def __init__(self): 12 self.buffer = '' 13 self.outfile = '' 14 15 def reset(self): 16 self.buffer = '' 17 self.outfile = '' 18 19 def push(self): 20 data = self.outfile 21 self.outfile = '' 22 return data 23 24 def close(self): 25 data = self.push() + self.buffer 26 self.buffer = '' 27 return data 28 29 @telemetry.ZONE_METHOD 30 def feed(self, data): 31 self.tempindex = 0 32 self.buffer = self.buffer + data 33 if self.buffer.find('<!--') != -1: 34 tempbuffer = [] 35 for cs in self.buffer.split('<!--'): 36 commentline = cs.split('-->') 37 if len(commentline) == 1: 38 tempbuffer.append(commentline[0]) 39 else: 40 commentline[0] = commentline[0].replace('<', '<').replace('>', '>') 41 tempbuffer.append('<!--' + '-->'.join(commentline)) 42 43 self.buffer = ''.join(tempbuffer) 44 outlist = [] 45 thischunk = [] 46 self.index = 0 47 bufferList = self.buffer.split('>') 48 for thischunk in bufferList[:-1]: 49 blue.pyos.BeNice() 50 s = thischunk.split('<') 51 if len(s[0]): 52 outlist.append(self.pdata(s[0])) 53 if len(s) > 1: 54 self.tagstart(s[-1]) 55 56 self.buffer = bufferList[-1] 57 58 @telemetry.ZONE_METHOD 59 def tagstart(self, thetag): 60 if thetag.startswith('!'): 61 return self.pdecl(thetag) 62 if thetag.startswith('?'): 63 return self.ppi() 64 if thetag.startswith('/'): 65 return self.endtag(thetag) 66 nt = namefind.match(thetag) 67 if not nt: 68 return self.emptytag(thetag) 69 name, attributes = nt.group(1, 2) 70 matchlist = attrfind.findall(attributes) 71 attrs = [] 72 for entry in matchlist: 73 attrname, rest, attrvalue = entry 74 if not rest: 75 attrvalue = attrname 76 elif attrvalue[:1] == "'" == attrvalue[-1:] or attrvalue[:1] == '"' == attrvalue[-1:]: 77 attrvalue = attrvalue[1:-1] 78 attrs.append((attrname.lower(), attrvalue)) 79 80 return self.handletag(name.lower(), attrs, thetag) 81 82 def pdata(self, inchunk): 83 return inchunk 84 85 def pdecl(self, thetag): 86 return '<' 87 88 def ppi(self): 89 return '<' 90 91 def endtag(self, thetag): 92 return '<' + thetag + '>' 93 94 def emptytag(self, thetag): 95 return '<' + thetag + '>' 96 97 def handletag(self, name, attrs, thetag): 98 return '<' + thetag + '>' 99 100 101 __doc__ = "\nScraper is a class to parse HTML files.\nIt contains methods to process the 'data portions' of an HTML and the tags.\nThese can be overridden to implement your own HTML processing methods in a subclass.\nThis class does most of what HTMLParser.HTMLParser does - except without choking on bad HTML.\nIt uses the regular expression and a chunk of logic from sgmllib.py (standard python distribution)\n\nThe only badly formed HTML that will cause errors is where a tag is missing the closing '>'. (Unfortunately common)\nIn this case the tag will be automatically closed at the next '<' - so some data could be incorrectly put inside the tag.\n\nThe useful methods of a Scraper instance are :\n\nfeed(data) - Pass more data into the parser.\n As much as possible is processed - but nothing is returned from this method.\npush() - This returns all currently processed data and empties the output buffer.\nclose() - Returns any unprocessed data (without processing it) and resets the parser.\n Should be used after all the data has been handled using feed and then collected with push.\n This returns any trailing data that can't be processed.\nreset() - This method clears the input buffer and the output buffer.\n\nThe following methods are the methods called to handle various parts of an HTML document.\nIn a normal Scraper instance they do nothing and are intended to be overridden.\nSome of them rely on the self.index attribute property of the instance which tells it where in self.buffer we have got to.\nSome of them are explicitly passed the tag they are working on - in which case, self.index will be set to the end of the tag.\nAfter all these methods have returned self.index will be incremented to the next character.\nIf your methods do any future processing they can manually modify self.index\nAll these methods should return anything to include in the processed document.\n\npdata(inchunk)\n Called when we encounter a new tag. All the unprocessed data since the last tag is passed to this method.\n Dummy method to override. Just returns the data unchanged.\n\npdecl()\n Called when we encounter the *start* of a declaration or comment. <!.....\n It uses self.index and isn't passed anything.\n Dummy method to override. Just returns '<'.\n\nppi()\n Called when we encounter the *start* of a processing instruction. <?.....\n It uses self.index and isn't passed anything.\n Dummy method to override. Just returns '<'.\n\nendtag(thetag)\n Called when we encounter a close tag. </...\n It is passed the tag contents (including leading '/') and just returns it.\n\nemptytag(thetag)\n Called when we encounter a tag that we can't extract any valid name or attributes from.\n It is passed the tag contents and just returns it.\n\nhandletag(name, attrs, thetag)\n Called when we encounter a tag.\n Is passed the tag name and attrs (a list of (attrname, attrvalue) tuples) - and the original tag contents as a string.\n\n\nTypical usage :\n\nfilehandle = open('file.html', 'r')\nparser = Scraper()\nwhile True:\n data = filehandle.read(10000) # read in the data in chunks\n if not data: break # we've reached the end of the file - python could do with a do:...while syntax...\n parser.feed(data)\n## print parser.push() # you can output data whilst processing using the push method\nprocessedfile = parser.close() # or all in one go using close\n## print parser.close() # Even if using push you will still need a final close\nfilehandle.close()\n\n\n\nTODO/ISSUES\nCould be sped up by jumping from '<' to '<' rather than a character by character search (which is still pretty quick).\nNeed to check I have all the right tags and attributes in the tagdict in approxScraper.\nThe only other modification this makes to HTML is to close tags that don't have a closing '>'.. theoretically it could close them in the wrog place I suppose....\n(This is very bad HTML anyway - but I need to watch for missing content that gets caught like this.)\nCould check for character entities and named entities in HTML like HTMLParser.\nDoesn't do anything special for self clsoing tags (e.g. <br />)\n\n\nCHANGELOG\n06-09-04 Version 1.3.0\nA couple of patches by Paul Perkins - mainly prevents the namefind regular expression grabbing a characters when it has no attributes.\n\n28-07-04 Version 1.2.1\nWas losing a bit of data with each new feed. Have sorted it now.\n\n24-07-04 Version 1.2.0\nRefactored into Scraper and approxScraper classes.\nIs now a general purpose, basic, HTML parser.\n\n19-07-04 Version 1.1.0\nModified to output URLs using the PATH_INFO method - see approx.py\nCleaned up tag handling - it now works properly when there is a missing closing tag (common - but see TODO - has to guess where to close it).\n\n11-07-04 Version 1.0.1\nAdded the close method.\n\n09-07-04 Version 1.0.0\nFirst version designed to work with approx.py the CGI proxy.\n\n"