Web scraping with Java

I'm not able to find any good web scraping Java based API. The site which I need to scrape does not provide any API as well; I want to iterate over all web pages using some `pageID` and extract the HTML titles / other stuff in their DOM trees.

Are there ways other than web scraping?

Look at an HTML parser such as TagSoup, HTMLCleaner or NekoHTML.

Your best bet is to use Selenium Web Driver since it

1. Provides visual feedback to the coder (see your scraping in action, see where it stops)
2. Accurate and Consistent as it directly controls the browser you use.
3. Slow. Doesn't hit web pages like HtmlUnit does but sometimes you don't want to hit too fast.

Htmlunit is fast but is horrible at handling Javascript and AJAX.

**HTMLUnit** can be used to do web scraping, it supports invoking pages, filling & submitting forms. I have used this in my project. It is good java library for web scraping.
[read here for more][1]


[1]:

[To see links please register here]

mechanize for Java would be a good fit for this, and as Wadjy Essam mentioned it uses JSoup for the HMLT. mechanize is a stageful HTTP/HTML client that supports navigation, form submissions, and page scraping.

[To see links please register here]

(and the GitHub here

[To see links please register here]

)

There is also Jaunt Java Web Scraping & JSON Querying -

[To see links please register here]

If you wish to automate scraping of large amount pages or data, then you could try [Gotz ETL][1].

It is completely model driven like a real ETL tool. Data structure, task workflow and pages to scrape are defined with a set of XML definition files and no coding is required. Query can be written either using Selectors with JSoup or XPath with HtmlUnit.


[1]:

[To see links please register here]

# jsoup

Extracting the title is not difficult, and you have many options, search here on Stack Overflow for "_Java HTML parsers_". One of them is [Jsoup](

[To see links please register here]

).

You can navigate the page using DOM if you know the page structure, see

[To see links please register here]

It's a good library and I've used it in my last projects.

For tasks of this type I usually use Crawller4j + Jsoup.

With crawler4j I download the pages from a domain, you can specify which ULR with a regular expression.

With jsoup, I "parsed" the html data you have searched for and downloaded with crawler4j.

Normally you can also download data with jsoup, but Crawler4J makes it easier to find links.
Another advantage of using crawler4j is that it is multithreaded and you can configure the number of concurrent threads

[To see links please register here]

Normally I use selenium, which is software for testing automation.
You can control a browser through a webdriver, so you will not have problems with javascripts and it is usually not very detected if you use the full version. Headless browsers can be more identified.