How do they know?

[cenk01]

Hi! I created a program that sends some GET commands to get the page source of a webpage. I also send some cookies too. But there is one problem, the site I'm connecting to knows I'm using a bot, how do they know that? Is it because of my send commands? Here is what I get from fiddler- a program that filters commands when you connect to the internet.

GET http://www.neopets.com/objects.phtml?type=shop&obj_type=7 HTTP/1.1
Host: www.neopets.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Referer: http://www.neopets.com/objects.phtml
Cookie: np_uniq=2006-10-04; xt6Yr4e33D=52182799197066579497; npuid=3c3406730N09000eG07c39000369c399c609760033c0f7978003740910703090; _tz=300; np_uniq_=2006-10-04; s_cc=true; s_sq=vianeopets2%3D%2526pid%253D/petcentral.phtml%2526pidt%253D1%2526oid%253Dhttp%25253A//www.neopets.com/objects.phtml%2526ot%253DAREA; neoremember=lollyyyyx37; neologin=lollyyyyx37%2B89f07cc830c7626120362cb0d7947582; toolbar=lollyyyyx37%2BB%2B7d8bb9c391fda0a0457554111780b054; np_randseed=90084-63405971855010; cachetimer=8637b522ac1bd60aa08140684ed08bb1; np_uniq_lollyyyyx37=2006-10-04
Pragma: no-cache
Cache-Control: no-cache

Do you notice the cookie line and dates. That say 2006-10-4? I made a function so my program automatically makes all the dates current. So tomorrow my program will change the dates to 2006-10-5. I thaught of this since the cookies I sent are outdated, they can know that I'm using a bot. But they still seem to know that I'm using a bot.

Could someone tell me how do they know this?

Thanks!

[MrViggy]

Do you know what the cookies mean? Perhaps every time you make a connection with your browser, one of the cookie values changes.

Viggy

[cenk01]

So they know it by cookies... Then how can I prevent them from knowning that I'm using a bot?

[MrViggy]

I was just guessing.

If they are using the cookies, then you'd have to find out what algorithim they use to change the values each time. You'd have to connect to the site a bunch of times, and analyze the cookie values yourself. I don't know of any "standard" techniques to do web page bot checking...

Viggy

[cenk01]

Wow, seems to be very hard. I just realized the number and letters in the cookie.

Cookie: np_uniq=2006-10-04; xt6Yr4e33D=52182799197066579497; npuid=3c3406730N09000eG07c39000369c399c609760033c0f7978003740910703090; _tz=300; np_uniq_=2006-10-04; s_cc=true; s_sq=vianeopets2%3D%2526pid%253D/petcentral.phtml%2526pidt%253D1%2526oid%253Dhttp%25253A//www.neopets.com/objects.phtml%2526ot%253DAREA; neoremember=lollyyyyx37; neologin=lollyyyyx37%2B89f07cc830c7626120362cb0d7947582; toolbar=lollyyyyx37%2BB%2B7d8bb9c391fda0a0457554111780b054; np_randseed=90084-63405971855010; cachetimer=8637b522ac1bd60aa08140684ed08bb1; np_uniq_lollyyyyx37=2006-10-04

So you mean I have to create a function that will make up those letters and numbers?!?!

[MrViggy]

If this is how they are detecting that it is a bot, not a browser, yes. Based on whatever algorithm they use. You can't use random letters and numbers.

Viggy

[cenk01]

Wow, very tough job. But I don't think that a person can actually crack that random numbers since I know there are programs like mine out there which don't get detected somehow. But it might also be the fact that they actually cracked the random numbers. Any other suggestions?

[cenk01]

Mr. Viggy, you were right! Look at these observations I recorded.

I opened firefox and fiddler, then recorded what cookies were sent. Then I wrote the whole cookie line down into Microsoft Word, then close firefox and opened it again, did the same steps and recorded it in to Microsoft word again. Guess what... SOME of the values were different, no wonder they can know it's a bot... Here are the observations. I'm not going to post all the cookie info, just some of the values that got changed.
1st time I opened firefox

Cookie: np_uniq=2006-10-04; xt6Yr4e33D=99782500610266579497; _tz=300; npuid=6006930i3304990cs0033003900770cf0c00970700976069037900183e67439c;

Then I close firefox opened it again, and this is what I got:

Cookie: np_uniq=2006-10-4; xt6Yr4e33D=52182799197066579497; _tz=300; npuid=3c3406730N09000eG07c39000369c399c609760033c0f7978003740910703090;

xt6Yr4e33D and npuid values are different. So now it's clear, the site knows that I'm using a bot because of the cookie values. Then here is the hard question;
How can I crack this random number code? It seems impossible with all those numbers and letters... Any ideas?

Thanks Mr. Viggy for the info!

[MikeAThon]

...xt6Yr4e33D and npuid values are different. So now it's clear, the site knows that I'm using a bot because of the cookie values. Then here is the hard question; How can I crack this random number code? It seems impossible with all those numbers and letters.

It probably is, indeed, impossible. As MrViggy explained above, there are no "standard" techniques by which servers generate their cookies. This means that the server's programmer is free to do whatever he wants, and he probably is doing something deliberately complicated, so that no one can figure it out.

Your best bet is to visit the site programmatically, and somehow record the cookies that the site sends you. Then parrot these values back to the site in your subsequent communications.

Mike

[cenk01]

Huh?

Your best bet is to visit the site programmatically, and somehow record the cookies that the site sends you. Then parrot these values back to the site in your subsequent communications.

But then since I got those cookies... wouldn't their values change again?

[MrViggy]

I was really only guessing...

Thinking out loud, one way to do this would be:

a) Log into server
b) Server sends you a cookie of some pre-computed value
c) You log off
d) You log back on; server looks at the cookie you send compares it to the one it knows it sent you before.
e) Cookie values match; server computes a new cookie and sends it back to you.

This is a fairly standard security technique (like one-time passwords), however the algorithim that creates the new cookie value can be almost anything!

Viggy

[MikeAThon]

Huh?
But then since I got those cookies... wouldn't their values change again?

Maybe, and if they do then this technique won't work.

You can only find out by trying. If the cookies stay constant for each session, then you might be able to succeed. But there's no requirement for them to stay constant; as before it's up to the web programmer and he can pretty much do whatever he wants. If they don't stay constant then this technique won't work and you'll have to try something else.

Mike

[cenk01]

So you are saying it is pretty impossible?

[MikeAThon]

So you are saying it is pretty impossible?

No. I said that you will not know if it's possible until you try it.

Mike

[cenk01]

I tried it a lot of times, but it seems to be really hard, and the site warned me not to do it again. Totally different question. How can I create a program that will launch a url in different IPs everytime.