network protocols

[dimm_coder]

Originally posted by souldog

Platform independence
Well if the stream consists of newline deliminated ascii strings
then big endian little endian does not come into play ( am I
wrong about this?) It seems like a crude way around doing byte
swapping.

Seems so, until the size of char is byte. I mean U donot send string in UTF-8 or smth.

[Kheun]

Originally posted by dimm_coder
One common problem with 1-st method is that as someone donot know how much data he have to read to complete the packet , there can be some problems with buffer overflowing or memory using (if someone donot pay enough attention while codding that code).

I thought buffer overflow is always considered as one of the most common secuity issue. There is many security attacks that use buffer overflow to write into the stack, causing the server application to do something else.

[dimm_coder]

Originally posted by Kheun
I thought buffer overflow is always considered as one of the most common secuity issue. There is many security attacks that use buffer overflow to write into the stack, causing the server application to do something else.

Yes, that's the security issue.
But I've said the following before it.

I donot think he means some security issues corresponding to data encrypting.

So I've meant the specified subclass of security issues, not all of them.

[Mick]

which buffer are we overflowing?, recv reads up to size of buffer for connection oriented sockets, (then you recv again after parsing the data) and connectionless reads up to size of recv buffer, then tosses the rest. but as I've stated (or I thought I did) I've always prepended the size, but I have had to deal with HL7 via connection oriented.

[souldog]

Ok.. I guess I started this thread to see if there is some good
reason to use tokenized streams that I am not thinking of. I
think not.

Micks mention of using the parsing mechanism to identify bad
packets is a reason I did not think of.

[Mick]

Originally posted by souldog
Ok.. I guess I started this thread to see if there is some good
reason to use tokenized streams that I am not thinking of. I
think not.

Micks mention of using the parsing mechanism to identify bad
packets is a reason I did not think of.

it really doesn't matter either way I guess. Both of them still have to read the data, if someone is sending you malformed data, you still have to determine that it's not a legitamite packet.

you can look at your packet size, or your magic chocolate chip cookie, and say that's not mine, and close the connection. But your still going to get hit with another connection in which you have to determine it's not yours...DDOS....

parsing same thing, parse some, determine it's crap, close the connection.

Usually in the parsing code, the first bits of data is well defined eg: this is my version of this code, this is the delimiter I am using, this is the current timestamp. So you parse that first header part, determine it's not valid, send an ack and close the connection (or not send an ack, you never know if the sending system is gone loco, jibbering jabbering nic card et al)

then rinse and repeat.

[dimm_coder]

Originally posted by Mick
which buffer are we overflowing?, recv reads up to size of buffer for connection oriented sockets, (then you recv again after parsing the data) ...

Where do U store yet not-completed packet before calling recv() again? And U cannot know how many recv() calls U have to do to get the completed packet (in marker case).
U have to use some buffer or smth to store it.
If U know a size of packet U have to get, it's much more clear way to design safe memory using.

[Mick]

Originally posted by dimm_coder
Where do U store yet not-completed packet before calling recv() again? And U cannot know how many recv() calls U have to do to get the completed packet (in marker case).
U have to use some buffer or smth to store it.
If U know a size of packet U have to get, it's much more clear way to design safe memory using.

not if your passing the packet data off to a thread to be parsed, you don't store a single thing. except maybe for the last delimited field that you were parsing before the data ended., but then you should have the fields laid out such that you know the max size the field can represent, hence a Standard.

[dimm_coder]

Originally posted by Mick
not if your passing the packet data off to a thread to be parsed, you don't store a single thing. except maybe for the last delimited field that you were parsing before the data ended., but then you should have the fields laid out such that you know the max size the field can represent, hence a Standard.

Well, there are some cases when U cannot process data before getting it all (completed packet). Sometimes it's better to wait for the rest of data before putting it off to a separate thread for parsing or processing. (using a separate thread for parsing sometimes is not the best solution too, but that's related to server code design and circumstances). But that's not related to the original question.

[Sam Hobbs]

Originally posted by Mick
which buffer are we overflowing?

I have always assumed that buffer overruns were the result of sloppy programming and only the result of sloppy programming.

[Mick]

Originally posted by dimm_coder
Well, there are some cases when U cannot process data before getting it all (completed packet). Sometimes it's better to wait for the rest of data before putting it off to a separate thread for parsing or processing. (using a separate thread for parsing sometimes is not the best solution too, but that's related to server code design and circumstances). But that's not related to the original question.

yes it really does depend on what you are doing...

[Mick]

Originally posted by Sam Hobbs
I have always assumed that buffer overruns were the result of sloppy programming and only the result of sloppy programming.

was there a question somewhere in there Sam don't assume, it's a fact.

but the place where you haXor it is in the interface, so if the interface only recieves up to n as specified by the buffer size, then n is all you get.

which is why you should also specify n as a param to bepassed in to indicate how large the data being passed in is. You either push only n into your own internal buffer and bit bucket the rest, or you dynamically grow your buffer and push n in.

/N

[usmarine]

Okay guys lets assume we are only talking about stream protocols for the time being. I.e. TCP.

The tokenized messages which you guys are referring to are usually called "self describing messages". There are alot of third party middleware apps which use this method because you can expand your messages without breaking or recompiling everything which uses it. It works nicely but tends to be a bit slow. The FIX protocol utilizes this method if you guys are interested.

I personally send binary packets with a header attached to every message. I have seen alot of people/companies not use headers which is a very poor design and adds compatibilty problems. The header should always contain 2 items. The message length and and the type of message. You can add other information as needed depending on what exactly you are doing.

The headers used should always be of the same length. If you include the length of a message you can extract it from the socket and process it faster.

Sometimes a will add additional integrity measures to my messages by ensuring they all end with a specific char etc.. That way I can handle hacking attacks and not allow a buffer overflow to occur.

[Sam Hobbs]

Originally posted by Homestead
I misread the question. And actually I didnt know nything about his problem. If I knew I would try my best to help him, true.
Could you help him ? You are good at computer and internet, just give him your ideas to help him out.... That also means you help me because i am trying to learn what he is doing ....

I think the question is vague and that even souldog isn't sure what the question is. So therefore this thread will continue to grow and grow and grow and ......

[galathaea]

I want to explain my position some.

Really, I don't think there is a requirement to use either packet size in a header or a delimiter. Even when you have multiple sizes in your packets, all you really need is some type info. Instead of size, you could have an ANSI string that says something like "LogonPacket" or "MilkPacket", and your socket can be connected to a factory to serialise in your packet. You can put in versioning as well, or use a number to identify the packets. As long as your factory knows the ABI of each object type, you do not need to transmit that at all.

I do transmit size, though. Its just easier for me to get the bytes in memory right away to be processed and free the socket for any more transmissions. Thats really the only reason why I put the size on.