You are here

Error downloading robots data: The remote server returned an error: (404) Not Found

2 posts / 0 new
Last post
arthurd123
Offline
Has donated long time ago
Joined: 9 years
Last seen: 2 years
Error downloading robots data: The remote server returned an error: (404) Not Found

Channel Cartoon Network site -- NAGRYWANIE.UPC.PL -- update mode incremental

 

Error downloading robots data: The remote server returned an error: (404) Not Found.

skipped robots check

 

Error downloading robots data: The remote server returned an error: (404) Not Found.

skipped robots check

 

   Summary for update of Cartoon Network

     missing shows added       0

     changed shows updated     0

     new shows added           141

     unchanged shows inspected 0

     total after update        141

     elapstime / updated show  0.34 seconds

 

Can anyone help here with this?

 

francis
Offline
francis's picture
Has donated long time agoWG++ Team member
Joined: 9 years
Last seen: 1 month
Is the support helpful?
support us

Ok,
First, this is not an error. Just a warning.
Explanation:
Before WG++ grabs on a website, it will check if the webserver has set its robots.txt file. (this is a simple .txt file in the root of the website)
By setting this file, a website can inform grabers (like WG++, Yahoo, Google, ...) what pages it doesn't want to be grabbed.
If the file is not found, you get this warning in the logging. And because it is missing, this means the site allows grabbing the entire website.
More info about the robots.txt can be found on : http://www.robotstxt.org/wc/robots.html
 
I think you get a correct listing after the grabbing? See you have 141 show grabed.
 

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl