Sorry, I am taking this off topic.

Jonathon, Your implementation looks interesting and useful for our startup that is scrapping stock prices from all over internet. However we are planning to use third party XQuery and I need org.w3c.dom.Document. Your implementation has a custom Document. How to convert your document model to org.w3c.Document tree? Any pointers much appreciated.

Self plug: I’ve just released a new open source HTML parser called jsoup @ http://jsoup.org/ . Its goal is to deal with all real-world HTML. The interface is designed a bit differently than the above parsers; it combines the best of DOM, CSS, and jquery-like accessor methods.

I ran the URLs from the showdown through it:

List urls = Arrays.asList(“http://www.theqarena.com”, “http://cleveland.indians.mlb.com”,
“http://www.clevelandbrowns.com”, “http://www.cbgarden.org”, “http://www.clemetzoo.com”,
“http://www.cmnh.org/site/”, “http://www.clevelandart.org”, “http://www.mocacleveland.org”,
“http://www.glsc.org”, “http://www.rockhall.com”);

for (String urlString : urls) {
URL url = new URL(urlString);
Document doc = Jsoup.parse(url, 3*1000);
Elements links = doc.select(“a”);

System.out.println(String.format(“Found %d links at %s”, links.size(), urlString));
}

The results: (all parse successfully)
Found 71 links at http://www.theqarena.com
Found 330 links at http://cleveland.indians.mlb.com
Found 105 links at http://www.clevelandbrowns.com
Found 135 links at http://www.cbgarden.org
Found 86 links at http://www.clemetzoo.com
Found 320 links at http://www.cmnh.org/site/
Found 103 links at http://www.clevelandart.org
Found 63 links at http://www.mocacleveland.org
Found 34 links at http://www.glsc.org
Found 90 links at http://www.rockhall.com

Probably all of these sites have updated since the original post so it’s not totally comparable, but I think it’s a pretty good showing, and the interface is (imho) a lot easier to use.

CleanerTransformations transformations = new CleanerTransformations();
TagTransformation tagTransformation = new TagTransformation(“table”, “TABLE”, true);
transformations.addTransformation(tagTransformation);
cleaner.setTransformations(transformations);

this code does not transform “table” into “TABLE” but…. if I try to transform it from “table” into “TABLET” or any value other than “table” it will transform…

is there a way where in I can transform all tags into Uppercase retaining all the tag attributes… in an instant? ^_^

I have a noob question regarding HTMLCleaner this time.

How can I transform from lower-case into upper-case? I want all the tags to be in Upper-case since the
xsl file that will use it is expecting all tags in upper-case form…. and I dont have idea about XSLT ^_^

Thank you and Godbless ^_^

Just a question, how about if I have the content that I want to parse? which among them are able to parse a string content directly not from a URL?

I am using this code..

DocumentBuilderFactory dbfac = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = dbfac.newDocumentBuilder();
doc = docBuilder.parse( new java.io.StringBufferInputStream(content) );

but I have no luck whenever the content is a malformed one. I need to convert this content into a Document or something in a form of nodes.

QUESTION:
————
Which among the tools u mentioned is capable of parsing the content directly and do some corrections if needed?
It would really be a great help.

Thanks and Godbless ^_^
————

So had to put in jTidy to clean up. But it required to save the intermediate file and then parse again from it. Alas my earlier xpath is no longer valid as now the DOM structure of temp html is different. Doesnt suit me.
Finally Tried Mozilla parser well in my office i dont hv rights to path variable so gave up. As other options to start was too time consuming to discover.
HTML Cleaner cleaner did the job for me no intermediate file saving. No creating xpath from intermediate files. It did parse malformed HTML as its parsed by IE or Mozilla(I know this because my JavaScript works for Mozilla as well as IE without any changes)