Printable View
Can someone please explain the advantage of using a tokenized
stream protocol (so packets are deliminated by say newline).
The only one I can think of is platform independence.
A major drawback seems to be security concerns.
Why, oh why, resort to such a method instead of prepending the
packet size?
Is it just lazy *** programmers?
What platform independence do U mean?Quote:
Originally posted by souldog
The only one I can think of is platform independence.
A major drawback seems to be security concerns.
Why, oh why, resort to such a method instead of prepending the
packet size?
Is it just lazy *** programmers?
I donot see any advantages on the first view as compared with other one (point the packet size).
That's merely the simplest way to implement not so a difficult client/server protocol but not so appropriate in many cases, though.
I prefer to use packets with special header which has a field describing a size of the packet.
I am not familiar with such things enough to help, but I don't understand why prepending the packet size enables security and a tokenized stream protocol does not. Also, is it possible to use both? I probably misunderstand, but if I understand well enough, it should be possible to use both and then the receiving algorithm could have two levels; one that ensures the data is the correct size and another that parses the data. The parsing could be done by existing software, such as std::stringstream.
Both methods are used to know the size of packet in the stream. So if U know a size of the packet (from header of packet), U need no any token to mark the end of packet. There is no need to use both of them.Quote:
Originally posted by Sam Hobbs
I am not familiar with such things enough to help, but I don't understand why prepending the packet size enables security and a tokenized stream protocol does not. Also, is it possible to use both? I probably misunderstand, but if I understand well enough, it should be possible to use both and then the receiving algorithm could have two levels; one that ensures the data is the correct size and another that parses the data. The parsing could be done by existing software, such as std::stringstream.
This is actually dependent on the question of why prepending the packet size enables security and a tokenized stream protocol does not. If a tokenized stream protocol can be as secure as without an additional prepended packet size, then it is my understanding that the size of the packet is not relevant to this thread.Quote:
Originally posted by dimm_coder
Both methods are used to know the size of packet in the stream. So if U know a size of the packet (from header of packet), U need no any token to mark the end of packet. There is no need to use both of them.
By looking at the title, I first thought it is about network routing protocols such as RIP and OSPF. :)Quote:
Originally posted by souldog
Can someone please explain the advantage of using a tokenized
stream protocol (so packets are deliminated by say newline).
The only one I can think of is platform independence.
A major drawback seems to be security concerns.
Why, oh why, resort to such a method instead of prepending the
packet size?
Is it just lazy *** programmers?
Can you explain further about the platform independence and security issues? From what I see, you are talking about the different way of packing data into packets, and I don't really understand what has it got to do with platform independency and security ?
Hmm....pardon my ignorance, can you explain what has the attached document to do with what souldog is asking? The paper is just introducing a platform independent C++ development environment, while souldog is asking about different ways of packing data into a stream for inter-process communication. I can't really corelate the two of them.Quote:
Originally posted by Homestead
Souldog, Nina have a file but not sure if it is useful for Souldog, perhaps Souldog already had it or read it...it s big, Nina couldnt attach it, so please come there :) and download
ForASoulOfPetLover
:)
Regards,
-Fiona
:)
I see no reason to ever use delimiters. That would just limit the character set you could use for the rest of the packet string. I've only ever prepended the size.
I've always been implementing my code the same way too.Quote:
Originally posted by galathaea
I see no reason to ever use delimiters. That would just limit the character set you could use for the rest of the packet string. I've only ever prepended the size.
The only one case to use markers I see is to send simple string lines (text data, not binary) and the marker here is '\n', because in this case the rest of string have no any '\n'.
But in any case, to point a size of packet is better way.
well I was waiting for souldogs response before I chimmed in...Quote:
Originally posted by galathaea
I see no reason to ever use delimiters. That would just limit the character set you could use for the rest of the packet string. I've only ever prepended the size.
because I guess I want clarification. It kinda all depends there guys...on what your doing.
are we talking tokenized as in...
foo|bar|num|longstring|bah
more|delimited|data|here
ENDOFPACKETTOKEN
?
as far as security, matters not if it's tokenized or sized packed in the first two bytes. It matters if it's encrypted or not. Sniff Sniff
Ok. First I don't have a problem, I was just belly aching because
a protocol I had written got changed to use deliminators (for
no good reason) and now I have to rewrite a server application
(No big deal)
Security (I use the word very loosely)
It seems to me a first line of defense ( a simplistic one) against
data corruption and maybe more is to limit the size of messages
you are willing to accept on a socket. This is clearly not possible
in a bare bones, simplistic, newline deliminated stream. You don't
know how much you are going to get until you have gotten all
of it. Obviuosly other methods can be used. I did not mean to
indicate the deliminated streams could not be made secure.
Platform independence
Well if the stream consists of newline deliminated ascii strings
then big endian little endian does not come into play ( am I
wrong about this?) It seems like a crude way around doing byte
swapping.
Most of all if you prepend the size then the stream is natuarlly
packetized as it is read in. There is no string parsing. I think it is
more elegant.
Anyway I started this b*tch without thinking much so take it or leave it. Personally I pass everthing in binary with a header that includes the size.
I didn't mean security in the proper way. I take it back. Just little
checks I do on packets coming in so my server/client doesn't
go down on bad or malicious data. I know it can be done in either
case.
Unfortunately this new protocol provides no convenient mechanisms.
well have to reach back in my memory banks and dust off that old chip...but a good example of why you would use tokienized streams would be the HL7 format.Quote:
Originally posted by souldog
Most of all if you prepend the size then the stream is natuarlly
packetized as it is read in. There is no string parsing. I think it is
more elegant.
not to say you couldn't do HL7 the other way, but it is more flexible to add new segments of data, and parse variable segments, whereas teh parsing code only needs to recognize that well that's not a segment it recognizes, and moves along until it finds one it can parse....
I was waiting too to know what he meant saying about platform independence and security issues. But seems he is out of there and cannot answer or smth.Quote:
Originally posted by Mick
well I was waiting for souldogs response before I chimmed in...
The question is related to different ways (ok, 2 ways here) of sending a variable sized packets via the streaming protocol (such as TCP).Quote:
are we talking tokenized as in...
foo|bar|num|longstring|bah
more|delimited|data|here
ENDOFPACKETTOKEN
?
as far as security, matters not if it's tokenized or sized packed in the first two bytes. It matters if it's encrypted or not. Sniff Sniff
There are 2 ways if U want to send the packet:
1) U choose some marker to pinpoint the end of packet (sometimes is useful for sending string lines in the simple client/server communication).
So client is sending data
"long long string line blalala \n"
'\n' is a marker.
Server has to read data from socket until he gets the char == '\n', now it knows there is the end of packet and it can process data.
2) we put some header with data size before the data.
The simplest case:
[header (4 bytes)][data (variable size == value of header]
So server first read 4 bytes and then it knows how many bytes in the following packet it has to read before processing it.
The 2-nd case is much more flexible and useful.
I donot think he means some security issues corresponding to data encrypting.
One common problem with 1-st method is that as someone donot know how much data he have to read to complete the packet , there can be some problems with buffer overflowing or memory using (if someone donot pay enough attention while codding that code).
just refreshing my memory on the byte ordering, and nearly spewed my coffee on this...
Should have read, you can easily bloat your code and slow it down considerably by porting it to MFC.Quote:
You can avoid the work of converting byte orders in the following situations:
blah...
blah...
blah...
You can port the server to MFC.
This is fairly easy to do, and the result is usually smaller, faster code. For information, see the MFC Migration Kit, which is included with Visual C++.
/Easily amused once more
Seems so, until the size of char is byte. I mean U donot send string in UTF-8 or smth.Quote:
Originally posted by souldog
Platform independence
Well if the stream consists of newline deliminated ascii strings
then big endian little endian does not come into play ( am I
wrong about this?) It seems like a crude way around doing byte
swapping.
I thought buffer overflow is always considered as one of the most common secuity issue. There is many security attacks that use buffer overflow to write into the stack, causing the server application to do something else.Quote:
Originally posted by dimm_coder
One common problem with 1-st method is that as someone donot know how much data he have to read to complete the packet , there can be some problems with buffer overflowing or memory using (if someone donot pay enough attention while codding that code).
Yes, that's the security issue.Quote:
Originally posted by Kheun
I thought buffer overflow is always considered as one of the most common secuity issue. There is many security attacks that use buffer overflow to write into the stack, causing the server application to do something else.
But I've said the following before it.
So I've meant the specified subclass of security issues, not all of them.Quote:
I donot think he means some security issues corresponding to data encrypting.
which buffer are we overflowing?, recv reads up to size of buffer for connection oriented sockets, (then you recv again after parsing the data) and connectionless reads up to size of recv buffer, then tosses the rest. but as I've stated (or I thought I did) I've always prepended the size, but I have had to deal with HL7 via connection oriented.
Ok.. I guess I started this thread to see if there is some good
reason to use tokenized streams that I am not thinking of. I
think not.
Micks mention of using the parsing mechanism to identify bad
packets is a reason I did not think of.
it really doesn't matter either way I guess. Both of them still have to read the data, if someone is sending you malformed data, you still have to determine that it's not a legitamite packet.Quote:
Originally posted by souldog
Ok.. I guess I started this thread to see if there is some good
reason to use tokenized streams that I am not thinking of. I
think not.
Micks mention of using the parsing mechanism to identify bad
packets is a reason I did not think of.
you can look at your packet size, or your magic chocolate chip cookie, and say that's not mine, and close the connection. But your still going to get hit with another connection :p in which you have to determine it's not yours...DDOS....
parsing same thing, parse some, determine it's crap, close the connection.
Usually in the parsing code, the first bits of data is well defined eg: this is my version of this code, this is the delimiter I am using, this is the current timestamp. So you parse that first header part, determine it's not valid, send an ack and close the connection (or not send an ack, you never know if the sending system is gone loco, jibbering jabbering nic card et al)
then rinse and repeat.
Where do U store yet not-completed packet before calling recv() again? And U cannot know how many recv() calls U have to do to get the completed packet (in marker case).Quote:
Originally posted by Mick
which buffer are we overflowing?, recv reads up to size of buffer for connection oriented sockets, (then you recv again after parsing the data) ...
U have to use some buffer or smth to store it.
If U know a size of packet U have to get, it's much more clear way to design safe memory using.
not if your passing the packet data off to a thread to be parsed, you don't store a single thing. except maybe for the last delimited field that you were parsing before the data ended., but then you should have the fields laid out such that you know the max size the field can represent, hence a Standard.Quote:
Originally posted by dimm_coder
Where do U store yet not-completed packet before calling recv() again? And U cannot know how many recv() calls U have to do to get the completed packet (in marker case).
U have to use some buffer or smth to store it.
If U know a size of packet U have to get, it's much more clear way to design safe memory using.
Well, there are some cases when U cannot process data before getting it all (completed packet). Sometimes it's better to wait for the rest of data before putting it off to a separate thread for parsing or processing. (using a separate thread for parsing sometimes is not the best solution too, but that's related to server code design and circumstances). But that's not related to the original question.Quote:
Originally posted by Mick
not if your passing the packet data off to a thread to be parsed, you don't store a single thing. except maybe for the last delimited field that you were parsing before the data ended., but then you should have the fields laid out such that you know the max size the field can represent, hence a Standard.
I have always assumed that buffer overruns were the result of sloppy programming and only the result of sloppy programming.Quote:
Originally posted by Mick
which buffer are we overflowing?
yes it really does depend on what you are doing...Quote:
Originally posted by dimm_coder
Well, there are some cases when U cannot process data before getting it all (completed packet). Sometimes it's better to wait for the rest of data before putting it off to a separate thread for parsing or processing. (using a separate thread for parsing sometimes is not the best solution too, but that's related to server code design and circumstances). But that's not related to the original question.
was there a question somewhere in there Sam :p don't assume, it's a fact.Quote:
Originally posted by Sam Hobbs
I have always assumed that buffer overruns were the result of sloppy programming and only the result of sloppy programming.
but the place where you haXor it is in the interface, so if the interface only recieves up to n as specified by the buffer size, then n is all you get.
which is why you should also specify n as a param to bepassed in to indicate how large the data being passed in is. You either push only n into your own internal buffer and bit bucket the rest, or you dynamically grow your buffer and push n in.
/N
Okay guys lets assume we are only talking about stream protocols for the time being. I.e. TCP.
The tokenized messages which you guys are referring to are usually called "self describing messages". There are alot of third party middleware apps which use this method because you can expand your messages without breaking or recompiling everything which uses it. It works nicely but tends to be a bit slow. The FIX protocol utilizes this method if you guys are interested.
I personally send binary packets with a header attached to every message. I have seen alot of people/companies not use headers which is a very poor design and adds compatibilty problems. The header should always contain 2 items. The message length and and the type of message. You can add other information as needed depending on what exactly you are doing.
The headers used should always be of the same length. If you include the length of a message you can extract it from the socket and process it faster.
Sometimes a will add additional integrity measures to my messages by ensuring they all end with a specific char etc.. That way I can handle hacking attacks and not allow a buffer overflow to occur.
I think the question is vague and that even souldog isn't sure what the question is. So therefore this thread will continue to grow and grow and grow and ......Quote:
Originally posted by Homestead
I misread the question. And actually I didnt know nything about his problem. If I knew I would try my best to help him, true.
Could you help him ? You are good at computer and internet, just give him your ideas to help him out...:). That also means you help me because i am trying to learn what he is doing ....:)
I want to explain my position some.
Really, I don't think there is a requirement to use either packet size in a header or a delimiter. Even when you have multiple sizes in your packets, all you really need is some type info. Instead of size, you could have an ANSI string that says something like "LogonPacket" or "MilkPacket", and your socket can be connected to a factory to serialise in your packet. You can put in versioning as well, or use a number to identify the packets. As long as your factory knows the ABI of each object type, you do not need to transmit that at all.
I do transmit size, though. Its just easier for me to get the bytes in memory right away to be processed and free the socket for any more transmissions. Thats really the only reason why I put the size on.
Well, after all this I have to conceed that I was just resisting the
change. No big deal either way, thanks for the discussion everyone.
I am not sure if anyone has pointed out this. Using the delimeter approach requires additional overhead in processing at both ends. The delimeter chosen should not appear inside the information content, or else portion of the information will be mistaken as the delimeter. To avoid this, we will have to do "byte stuffing" to differentiate delimeters from actual information content.
I would prefer to append a header for every messages sent, rather than using delimeters. It caters for both fixed length and variable length packets in a more organized and simpler way.
Perhaps I misunderstand you, but in the original question, the delimeter is a newline. This implies to me that the data is textual, and the delimiter is as effective as it is in millions of text files that exist.Quote:
Originally posted by Wombat
The delimeter chosen should not appear inside the information content, or else portion of the information will be mistaken as the delimeter.
Or do I misunderstand?
Okie, I am actually talking about the approach of using delimeter as a whole, rather than being specific to the context brought out by souldog. If the data is textual, then using newline as delimeter is acceptable without need of byte stuffing. (Actually I admit I did not pay much attection to the original post cos it errm.....sounds ambiguous :p)Quote:
Originally posted by Sam Hobbs
Perhaps I misunderstand you, but in the original question, the delimeter is a newline. This implies to me that the data is textual, and the delimiter is as effective as it is in millions of text files that exist.
Or do I misunderstand?
As for other system that contains not only textual data, byte stuffing would be required if use delimeter approach. If that is the case, I rather opt for appending header.
Yes, my first post is ambiguous :cry:
I didn't have a specific question. I just wanted to know if anyone
out there had some very good reason to prefer deliminated streams to using a header with the size. The type of token is not
important, but I had a particular example in mind.
SolarFlareeeeeeeeee:(Quote:
Originally posted by souldog
Yes, my first post is ambiguous
I didn't have a specific question. I just wanted to know if anyone
out there had some very good reason to prefer deliminated streams to using a header with the size. The type of token is not
important, but I had a particular example in mind.
You deleted my question to Souldog right?
I just wanted to know if he already solved the problem or not...True. (no joke this time, I dont want to joke like that either)
[Yves: it's not solarflare, it's me. I'm removing noise from the forum]
Soul Dog !!!!
Well, ur ache is common as the same thing aches me as well,
However wat i see the problem is that u want the goodies of TCP but want the communication as UDP, as packets, so u do not have the insert the message delimiters ur self,
However, i'd seriousely recomend u take a look into SCTP ( stream control transport protocol)
SCTP was originally made to replace SS7, and carry out telecom signalling info over IP networks, however it is so good and (REAL TIME) that it now being used every where.
This is a new baby, and not implemented in a whole lot OSs by default, however implementations are available for most of the OSs as pathces . U can fidn them over the internet.
The basics of SCTP are that It delivers "messages" and not a stream, with same gaurantees as TCP(stream), so we do not have to worry abt Message delimiters or Packet size, nor do we have to tink abt CRCs, that are a real bear to figure out,
As for compatibility, i know the linux flavours allready have the SCTP implementation, I also know that SCTP's implementation is available for Win platform as well,
As for some one said abt Litle endian and big endian prob, that has nothing to do with OS, it has to do with the Processor and how it deals with mem words given to it, so forget it, no one can ever achieve that independence, that is best left to be done in the Kernel in the protocol implementation.
I hope this is useful to u,
Best regards
Salman:wave:
No. If some application transfers binary data, it has to deal with it. Of course, if it wants to be completely cross-platform. All is good while both client and server are running on machines with the same architecture. If U cannot be sure that, U are seeking out problems.Quote:
Originally posted by salman108
As for some one said abt Litle endian and big endian prob, that has nothing to do with OS, it has to do with the Processor and how it deals with mem words given to it, so forget it, no one can ever achieve that independence, that is best left to be done in the Kernel in the protocol implementation.
There is a list of functions specified for that case:
htonl, htons, ntohl, ntohs - convert values between host and network byte order.
___________________________________________________
No. If some application transfers binary data, it has to deal with it. Of course, if it wants to be completely cross-platform. All is good while both client and server are running on machines with the same architecture. If U cannot be sure that, U are seeking out problems.
___________________________________________________
See, as far as i know, Network = Pair of wires, think of the byte
10101010.
now the bits have to go on the wire either starting from Left (1) or right (0).
The thing is what ever goes on the network has to be in Nework byte order, But prob is some processors do not comply with network byte order, they read the bits from the other hand first,
Intel processors are not in Nework byte order so we have to convert them
This is how the hyperDictionary.com describes this
" The order in which the bytes of a multi-byte number are transmitted on a network - most significant byte first (as in "big-endian" storage). This may or may not match the order in which numbers are normally stored in memory for a particular processor"
and this is how chaps at Novell like to describe this
Byte order is dependent on the CPU Word architecture. In processors compatible with Intel processors, 4 byte long integers are represented as a sequence of 4 bytes, with the less significant bytes of the integer stored in the more significant bytes. This byte order is called host byte order or little endian order (little-end-first).
I hope the problem is solved:D