Hi using Jcrawler 1.8 beta at default settings I get this report in red:
Curl error on url www.lirmoi.com/component/contact/12-contacts/1-name.html: Operation timed out after 20 seconds with 0 bytes received
httpcode: 404 on url www.lirmoi.com/mailto%3A
Curl error on url www.lirmoi.com/kontakt-os.html: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/info-inden-du-beslutter-dig.html: Operation timed out after 20 seconds with 24106 bytes received
Curl error on url www.lirmoi.com/styregruppe-for-projektet.html: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/samarbejdspartnere.html: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/videnskabelige-publikationer.html: Operation timed out after 20 seconds with 0 bytes received
httpcode: 508 on url www.lirmoi.com/nyheder-og-status-om-proj...-af-resveratrol.html
httpcode: 508 on url www.lirmoi.com/
httpcode: 508 on url www.lirmoi.com/hvad-er-lirmoi.pdf
httpcode: 508 on url www.lirmoi.com/baggrund-/1-ars-behandling-ved-overvaegt.pdf
httpcode: 508 on url www.lirmoi.com/baggrund-/studie-2-ekstra...er-til-studie-1.html
httpcode: 508 on url www.lirmoi.com/baggrund-/studie-4-unders...gastrisk-bypass.html
httpcode: 508 on url www.lirmoi.com/baggrund-/6-mdr-behandlin...fedtlever-sygdom.pdf
httpcode: 508 on url www.lirmoi.com/baggrund-for-forsogene-detaljeret.pdf
httpcode: 508 on url www.lirmoi.com/formal.pdf
httpcode: 508 on url www.lirmoi.com/perspektiver.pdf
httpcode: 508 on url www.lirmoi.com/detaljeret-beskrivelse-af...nkelte-projekter.pdf
httpcode: 508 on url www.lirmoi.com/baggrund-.pdf
httpcode: 508 on url www.lirmoi.com/info-inden-du-beslutter-dig.pdf
httpcode: 508 on url www.lirmoi.com/phd-studerende.pdf
httpcode: 508 on url www.lirmoi.com/phd-studerende/thomas-kjaer.html
httpcode: 508 on url www.lirmoi.com/phd-studerende/sara.html
httpcode: 508 on url www.lirmoi.com/phd-studerende/berthild.html
httpcode: 508 on url www.lirmoi.com/phd-studerende/phd-ole.html
Curl error on url www.lirmoi.com/nyheder-og-status-om-proj...forsog-fedtvaev.pdf: Operation timed out after 20 seconds with 0 bytes received
httpcode: 508 on url www.lirmoi.com/nyheder-og-status-om-proj...ansk-universitet.pdf
httpcode: 508 on url www.lirmoi.com/nyheder-og-status-om-proj...g-af-resveratrol.pdf
httpcode: 508 on url www.lirmoi.com/nyheder-og-status-om-proj...e-soges-i-odense.pdf
Curl error on url www.lirmoi.com/projektet-i-medierne.pdf: Operation timed out after 20 seconds with 65088 bytes received
Curl error on url www.lirmoi.com/phd-studerende/anders-dahl-knudsen.pdf: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/phd-studerende/marie-ornstrup.pdf: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/nyheder/95-resveratrol-phd-forsog-fedtvaev.pdf: Operation timed out after 20 seconds with 8192 bytes received
Curl error on url www.lirmoi.com/nyheder/93-klaringsrappor...-af-resveratrol.pdf: Operation timed out after 20 seconds with 0 bytes received
Curl error on url www.lirmoi.com/nyheder/75-ph-d-studerende-soges-i-odense.pdf: Operation timed out after 20 seconds with 7744 bytes received
httpcode: 404 on url www.lirmoi.com/mailto%3Amarie.juul.ornstrup%40ki.au.dk
httpcode: 404 on url www.lirmoi.com/mailto%3Athomas.kjaer%40ki.au.dk
httpcode: 508 on url www.lirmoi.com/index.php
httpcode: 508 on url www.lirmoi.com/phd-studerende.html
httpcode: 508 on url www.lirmoi.com/phd-studerende/phd-lars.pdf
httpcode: 508 on url www.lirmoi.com/baggrund-for-forsogene-detaljeret.html
httpcode: 508 on url www.lirmoi.com/formal.html
httpcode: 508 on url www.lirmoi.com/perspektiver.html
httpcode: 508 on url www.lirmoi.com/detaljeret-beskrivelse-af...kelte-projekter.html
httpcode: 508 on url www.lirmoi.com/baggrund-.html
httpcode: 404 on url www.lirmoi.com/mailto%3A


Many of these webpages are OK - so somehow there is a problem.

Another issue is; If I try to decrease the number of parallel connetctions from 50 to anything else I get this error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please open a ticket at support department and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.



My webpage www.lirmoi.com is hosted at cloudaccess.net

Regards
Steen

I think this is what's happening:
1. if you have too many connections, your sh404SEF or your server configuration blocks the crawling because it thinks this is a DOS attack;
2. if you reduce your parallel connections, the crawling takes too long to complete and the script gets a timeout.

Only possible solution is to analyze your site / hosting security thresholds.

We hope to find a way around this issues in the next versions of JCrawler.

Hi

I just want to report, that I tried version 1.9, and all my problems that I described here in the first post in this thread are solved.

SO THANK YOU VERY MUCH

@sbp:
that is unexpected I mean, we did some bugfixing and general code maintenance for this release, but the procedure hasn't really changed... it seems we did just the RIGHT small changes
Glad to hear we're doing fine!