Web scraping with Java

You might look into [jwht-scraper](

[To see links please register here]

)!

This is a complete __scraping__ framework that has all the features a developper could expect from a web __scraper__ :

- [Proxy support](

[To see links please register here]

)
- [Warning Sign Support to detect captchas and more](

[To see links please register here]

)
- [Complex link following features](

[To see links please register here]

)
- [Multithreading](

[To see links please register here]

)
- [Various scraping delays when required](

[To see links please register here]

)
- [Rotating User-Agent](

[To see links please register here]

)
- [Request auto retry and HTTP redirections supports](

[To see links please register here]

)
- [HTTP headers, cookies and more support](

[To see links please register here]

)
- [GET and POST support](

[To see links please register here]

)
- [Annotation Configuration](

[To see links please register here]

)
- [Detailed Scraping Metrics](

[To see links please register here]

)
- [Async handling of the scraper client](

[To see links please register here]

)
- [jwht-htmltopojo fully featured framework to map HTML to POJO](

[To see links please register here]

)
- [Custom Input Format handling and built in JSON -> POJO mapping](

[To see links please register here]

)
- [Full Exception Handling Control](

[To see links please register here]

)
- [Detailed Logging with log4j](

[To see links please register here]

)
- [POJO injection](

[To see links please register here]

)
- Custom processing hooks
- Easy to use and well documented API

It works with (jwht-htmltopojo)[

[To see links please register here]

) lib which itsef uses Jsoup mentionned by several other people here.

Together they will help you built awesome scrapers mapping directly HTML to POJOs and bypassing any classical scraping problems in only a matter of minutes!

Hope this might help some people here!

Disclaimer, I am the one who developed it, feel free to let me know your remarks!