Topic: Spiderwatch  (Read 5776 times)

Author Message

0 Members and 1 Guest are viewing this topic.

gooner

  • Guest
Spiderwatch
« on: October 07, 2008, 05:39:05 PM »
Google spider      17:35:40     Viewing the topic RCM Website Contact Form Submissions / Questions.
Yahoo spider    17:33:40    Viewing the board index of RCM - RETRO COMPUTER MUSEUM.
Google spider    17:33:13    Viewing the topic Do you believe in UFO?.

Offline billdooruk

  • Administrator
  • Amiga 4000
  • ******
  • Posts: 2476
  • Kudos 26
  • Gender: Male
  • I could murder a curry
    • View Profile
    • Our site:-
Re: Spiderwatch
« Reply #1 on: October 07, 2008, 06:14:45 PM »
Eh?

Have I missed something?

Offline AndyRCM

  • >=))))º> GO FEED THE FISH! <º((((=<
  • Administrator
  • Amiga 4000
  • ******
  • Posts: 9675
  • Kudos 50
  • Gender: Male
  • Manic Jet Set Willy
    • View Profile
    • Retro Computer Museum
Re: Spiderwatch
« Reply #2 on: October 07, 2008, 06:41:59 PM »
Here you go . . . this now shows what search engines are looking at our site . . . thanks to Nick!

http://retrocomputermuseum.co.uk/forum/index.php?topic=280.msg3176#new
"I could see the faces of those who led pissing themselves laughing" - Funeral Pyre by The Jam

Offline thx1138

  • CPC 464
  • ***
  • Posts: 231
  • Kudos 4
  • Gender: Male
    • View Profile
    • Football Forum UK
Re: Spiderwatch
« Reply #3 on: October 07, 2008, 07:01:04 PM »
www.stopforumspam.com  is a good resource for stuff like that.

gooner

  • Guest
Re: Spiderwatch
« Reply #4 on: October 07, 2008, 07:18:25 PM »
Wahhey Yahoo Spider is coming to the open day unless its a webb of lies LOL :D

Yahoo spider      19:09:58     Viewing the topic RCM Open Day - Tickets On Sale!.

Offline Shockwave

  • RCM Programmer.
  • user
  • C128
  • *
  • Posts: 307
  • Kudos 14
  • Gender: Male
    • View Profile
    • My Home Page
Re: Spiderwatch
« Reply #5 on: October 07, 2008, 10:22:00 PM »
www.stopforumspam.com  is a good resource for stuff like that.

It's important to differentiate between good spiders and bad ones.
A lot of them like askjeeves, googlebot, msnbot, inktomi slurp etc you want on the site because you need them to list you in their search engines.

Most of these robots are well programmed and a correctly written robots.txt file, (or even meta tags can contain instructions for them) will ensure that they respect your bandwidth.

The site thx1138 posed is a database of "rogue" web bots ip's. if you are losing a lot of bandwidth to robots, or if you suspect that they are harvesting email addresses or other details  and you get the IP from your server logs, the database helps you to identify what the robot is.

Mind you, google would do the same. If you copy a suspicious IP and google it, chances are that someone will have come across it.

You can then block the IP at server level if needs be.

gooner

  • Guest
Re: Spiderwatch
« Reply #6 on: October 09, 2008, 06:28:17 PM »

Yahoo spider    18:14:37    Viewing andy313's profile. :D

Offline Infernospence

  • Vic 20
  • **
  • Posts: 58
  • Kudos 3
  • Gender: Male
    • View Profile
Re: Spiderwatch
« Reply #7 on: October 10, 2008, 08:27:12 PM »
Google spider  08:19:56 PM Viewing the topic Street of RAGE Remake.

LOL
Tweet Tweet Time to Sleep...Or are you Waking up?

Offline Infernospence

  • Vic 20
  • **
  • Posts: 58
  • Kudos 3
  • Gender: Male
    • View Profile
Re: Spiderwatch
« Reply #8 on: October 11, 2008, 03:17:50 PM »
Google spider  03:11:43 PM Viewing the topic Movie legend Paul Newman dies, 83.
Tweet Tweet Time to Sleep...Or are you Waking up?

gooner

  • Guest
Re: Spiderwatch
« Reply #9 on: October 11, 2008, 03:23:53 PM »
Google spider      15:17:28     Viewing the topic Family Video Game 168.
Google spider    15:11:43    Viewing the topic Movie legend Paul Newman dies, 83.

Offline Infernospence

  • Vic 20
  • **
  • Posts: 58
  • Kudos 3
  • Gender: Male
    • View Profile
Re: Spiderwatch
« Reply #10 on: October 11, 2008, 03:49:31 PM »
Google spider  03:45:39 PM Viewing the topic Happy Birthday.
Google spider  03:41:08 PM Printing the topic "What Kind of fonts is in the picture".
Tweet Tweet Time to Sleep...Or are you Waking up?

gooner

  • Guest
Re: Spiderwatch
« Reply #11 on: October 11, 2008, 04:07:28 PM »
Google spider      16:03:45     Viewing the topic I got Trojan Virus.
Google spider    15:57:15    Viewing the topic I got Trojan Virus.
 Poor spiders :'(

gooner

  • Guest
Re: Spiderwatch
« Reply #12 on: October 13, 2008, 08:20:53 PM »
Google spider      20:18:46     Viewing the topic New Systems . . . not added yet . . ..
Google spider    20:11:44    Viewing the topic New Systems . . . not added yet . . ..
Yahoo spider    20:06:12    Logging into the forum.
Hope he4doesn't forget his password ;)

gooner

  • Guest
Re: Spiderwatch
« Reply #13 on: October 19, 2008, 09:44:59 PM »
Google spider      21:37:17     Printing the topic "General bits . . .".

gooner

  • Guest
Re: Spiderwatch
« Reply #14 on: November 07, 2008, 05:06:34 AM »
Google spider      05:03:31     Viewing the topic Hangman . . ..
Google spider    05:01:31    Viewing the topic Time Game.
Google spider    04:56:58    Viewing the topic Do you believe in Ghost?.
Yahoo spider    04:51:57    Viewing the topic RCM Open Day - Tickets On Sale!.

 :D