Protecting text from robots

[Zcumbag]

Hi,

I was wondering how to make my webpages protected from third party aplications reading my site? The page is login-protected but that I suppose is not protection enough if there's a valid session.

The only two ways I can think of is to render the text in an image and display that - or - create a java-applet.

I dont really fancy any of those solutions. Is there a better one?

And by the way... is there a .NET equivalent to a applet?

Thanks!

[TheCPUWizard]

Simple, yet effective against many robots.... Put all of your content inside a Frame.

[Zcumbag]

Putting content in a frame will not help. It can just as easy be read. My problem is not with your every day day robot or crawler but with programs written explicitly to read my page.

[Igor Soukhov]

I was wondering how to make my webpages protected from third party aplications reading my site? The page is login-protected but that I suppose is not protection enough if there's a valid session.

Please define "protect" or it would be even better if you could explain why would you need such "protection".

How the information is presented to the visitor?
How visitors navigate between web pages?
How much users do you have and who they are?

To give you a practical advice we need to know more about your web app,
because giving away simple-with-no-prior-knowledge suggestions like these:

1.Putting a 30-seconds delay every time user navigate from
one page to another.
2.Adding a CAPTCHA test on each page (even if there's a valid session)

unlikely to help you.

And by the way... is there a .NET equivalent to a applet?

No. To my best knowlege the closest thing you could get is Windows Forms controls hosted inside Internet Explorer.

[Zcumbag]

Thanks for the reply.

What I need protected is simply a small part of one of my webpages. This webpage renders (among other things) a string of text every two minutes or so. It is essential for my app that theese strings are read by a user that is visiting my page. I do not want there to be a way for a program to countinusly "refresh" this page and alert the user when a new message is rendered.

Any clearer

[Igor Soukhov]

Thanks for the reply.
What I need protected is simply a small part of one of my webpages. This webpage renders (among other things) a string of text every two minutes or so. It is essential for my app that theese strings are read by a user that is visiting my page. I do not want there to be a way for a program to countinusly "refresh" this page and alert the user when a new message is rendered.

As I have mentined in my previous post - you could make use of a CAPTCHA image on your web page.

If the page gets requested in a less than a <defined period of time> since the last time - render a page with a CAPTCHA on it asking visitor to prove that he/she in not a machine.

[Zcumbag]

I don't think a CAPTCHA is the answer here. It would be considerably better just to have the server generate the text in a jpg instead.

[Igor Soukhov]

I don't think a CAPTCHA is the answer here. It would be considerably better just to have the server generate the text in a jpg instead.

You wrote "My problem is not with your every day day robot or crawler but with programs written explicitly to read my page".

If someone is so interested in the information at your web site, having a text rendered as JPG will not solve your problem at all.

I don't think it would take more than a couple of hours to add a text recognition capabilities into that custom-made-crawler-program.

[Zcumbag]

Ok. possibly. But a CAPTCHA is not an option since it would be to annoying to the everyday user.

What about a java-applet then, would it be just as easy to make a text recognition tool for this or would this be a better option?

[Igor Soukhov]

Ok. possibly. But a CAPTCHA is not an option since it would be to annoying to the everyday user.

Since you see that shortcomings of applying a CAPTCHA (although it is was invented to do exactly what you need) outweight it’s possible advantages – you could try tackle a problem following two methods:

1. Think of what is typical bot/robot is going to do on YOUR web site. Then think of typical user scenarios. The key here is to grasp the difference - how the typical robot’s behavior could be different from the behavior of typical user of YOUR system.

And then re-design your app in a way to make navigation, page refreshing, etc more difficult to the visitors which behavior falls into “suspicious” category. This will likely introduce a problems to a small percentage of human users as well, but not to “typical-everyday” users.

2. You may also decide to delegate some of anti-robots functions to a human. Track the unusual behavior and report it to person who is looking after the web site (“moderator” or “admin”). If the reported user seems like a robot, moderator could decide to ban his session or login.

If the numbers of robot attacks are small and it’s alright to have some delay in reaction to these attacks - this might work for you.

Try mixing these methods.

As for your question - java applet will not make a big difference as it could be easily grabbed as an image and OCR-ed.