Sorry for delay in answer, i did not enable mail notifications.
1. Crawling your site, especially if it is quite a complex sites with many links, WILL make extensive resources use. There is at the moment no "solution" for this.
2. The "Internal Server Error" is most likely due to a timeout. In FastCGI environment, the PHP ini directives about execution time got ignored so it will stop after about 40 seconds.
Fact is, the only way to avoid 2. is to increment parallel connections and using more resources, thus getting back to 1; but it's the only reasonable solution.
We have two possible workarounds:
- in JCrawler 1.x, adding a "chaining" structure which calls different pages instead of doing the crawling in a single shot
- in JCrawler 2.x, which is in an early development stage, a new crawling engine, possibly Ajax-based, so we may insert all kind of priority / delay configuration options.
