RCM - RETRO COMPUTER MUSEUM

GENERAL => OFF TOPIC => Topic started by: gooner on October 07, 2008, 05:39:05 PM

Title: Spiderwatch
Post by: gooner on October 07, 2008, 05:39:05 PM
Google spider      17:35:40     Viewing the topic RCM Website Contact Form Submissions / Questions.
Yahoo spider    17:33:40    Viewing the board index of RCM - RETRO COMPUTER MUSEUM.
Google spider    17:33:13    Viewing the topic Do you believe in UFO?.
Title: Re: Spiderwatch
Post by: billdooruk on October 07, 2008, 06:14:45 PM
Eh?

Have I missed something?
Title: Re: Spiderwatch
Post by: AndyRCM on October 07, 2008, 06:41:59 PM
Here you go . . . this now shows what search engines are looking at our site . . . thanks to Nick!

http://retrocomputermuseum.co.uk/forum/index.php?topic=280.msg3176#new
Title: Re: Spiderwatch
Post by: thx1138 on October 07, 2008, 07:01:04 PM
www.stopforumspam.com  is a good resource for stuff like that.
Title: Re: Spiderwatch
Post by: gooner on October 07, 2008, 07:18:25 PM
Wahhey Yahoo Spider is coming to the open day unless its a webb of lies LOL :D

Yahoo spider      19:09:58     Viewing the topic RCM Open Day - Tickets On Sale!.
Title: Re: Spiderwatch
Post by: Shockwave on October 07, 2008, 10:22:00 PM
www.stopforumspam.com  is a good resource for stuff like that.

It's important to differentiate between good spiders and bad ones.
A lot of them like askjeeves, googlebot, msnbot, inktomi slurp etc you want on the site because you need them to list you in their search engines.

Most of these robots are well programmed and a correctly written robots.txt file, (or even meta tags can contain instructions for them) will ensure that they respect your bandwidth.

The site thx1138 posed is a database of "rogue" web bots ip's. if you are losing a lot of bandwidth to robots, or if you suspect that they are harvesting email addresses or other details  and you get the IP from your server logs, the database helps you to identify what the robot is.

Mind you, google would do the same. If you copy a suspicious IP and google it, chances are that someone will have come across it.

You can then block the IP at server level if needs be.
Title: Re: Spiderwatch
Post by: gooner on October 09, 2008, 06:28:17 PM

Yahoo spider    18:14:37    Viewing andy313's profile. :D
Title: Re: Spiderwatch
Post by: Infernospence on October 10, 2008, 08:27:12 PM
Google spider  08:19:56 PM Viewing the topic Street of RAGE Remake.

LOL
Title: Re: Spiderwatch
Post by: Infernospence on October 11, 2008, 03:17:50 PM
Google spider  03:11:43 PM Viewing the topic Movie legend Paul Newman dies, 83.
Title: Re: Spiderwatch
Post by: gooner on October 11, 2008, 03:23:53 PM
Google spider      15:17:28     Viewing the topic Family Video Game 168.
Google spider    15:11:43    Viewing the topic Movie legend Paul Newman dies, 83.
Title: Re: Spiderwatch
Post by: Infernospence on October 11, 2008, 03:49:31 PM
Google spider  03:45:39 PM Viewing the topic Happy Birthday.
Google spider  03:41:08 PM Printing the topic "What Kind of fonts is in the picture".
Title: Re: Spiderwatch
Post by: gooner on October 11, 2008, 04:07:28 PM
Google spider      16:03:45     Viewing the topic I got Trojan Virus.
Google spider    15:57:15    Viewing the topic I got Trojan Virus.
 Poor spiders :'(
Title: Re: Spiderwatch
Post by: gooner on October 13, 2008, 08:20:53 PM
Google spider      20:18:46     Viewing the topic New Systems . . . not added yet . . ..
Google spider    20:11:44    Viewing the topic New Systems . . . not added yet . . ..
Yahoo spider    20:06:12    Logging into the forum.
Hope he4doesn't forget his password ;)
Title: Re: Spiderwatch
Post by: gooner on October 19, 2008, 09:44:59 PM
Google spider      21:37:17     Printing the topic "General bits . . .".
Title: Re: Spiderwatch
Post by: gooner on November 07, 2008, 05:06:34 AM
Google spider      05:03:31     Viewing the topic Hangman . . ..
Google spider    05:01:31    Viewing the topic Time Game.
Google spider    04:56:58    Viewing the topic Do you believe in Ghost?.
Yahoo spider    04:51:57    Viewing the topic RCM Open Day - Tickets On Sale!.

 :D
Title: Re: Spiderwatch
Post by: zerohour on November 07, 2008, 03:10:11 PM
Google spider      05:03:31     Viewing the topic Hangman . . ..
Google spider    05:01:31    Viewing the topic Time Game.
Google spider    04:56:58    Viewing the topic Do you believe in Ghost?.
Yahoo spider    04:51:57    Viewing the topic RCM Open Day - Tickets On Sale!.

 :D

don't you ever sleep... far too early for me to keep an eye on the spiders lol :)
Title: Re: Spiderwatch
Post by: gooner on November 07, 2008, 05:59:02 PM
don't you ever sleep... far too early for me to keep an eye on the spiders lol :)
I work longer hours than the spiders and get less reward. :'(
I wish i could crawl around websites all day doing absolutely sod all :D
Title: Re: Spiderwatch
Post by: gooner on November 11, 2008, 06:46:55 PM
Google spider      18:33:06     Viewing the topic I'll be there in spirit (not vodka)  :D
Title: Re: Spiderwatch
Post by: gooner on November 18, 2008, 07:39:17 PM
Yahoo spider      19:36:22     Viewing the help page.
Google spider    19:31:07    Viewing Andyroo's profile. Another Andy? ;D
Google spider    19:29:50    Viewing the topic Pandora - The Next Amiga? .
Google spider    19:26:20    Viewing the topic VIC-20 PC.....