need Java memcpy() equivalent

[Randy C]

Here's the situation. I'm tasked with "porting" and composing device communication drivers in JAVA. By communication drivers, I mean I'm going to need to both parse raw byte streamed messages coming from various IEDs (Intelligent Electronic Devices, not explosives!), and format similar byte streams to be transmitted back. At the very least I'm going to have to convert arrays of bytes (or portiona of such arrays) to JAVA primitives (such as shorts, ints, longs, and IEEE floats), and reverse the same process for transmitting data. Note that due to the many and varied IEDs I'll be dealing with, I can not even count on multibyte types being in big or small endian format.

Well its no big deal for me to swap around byte order in Java, and really its not that big a deal to convert byte streams to integral types such as shorts or ints, using various shift operators (and being careful about sign extension). But 32 and 64 bit real numbers (floats and doubles) are another matter. I suppose I could write routines to parse bit fields to interpret mantissa, exponent, sign bits, etc, and "rebuild" floating point numbers from scratch. I suppose I could also do the reverse for converting floats to bytes too. But my gut feeling is that doing so will result in horribly CPU hogging code. Clearly, some simple way to serialize and de-serialize primitives is the best approach. I've seen an article about "deep copying" of Java objects using such serialization, but it seemed geared toward working with entire Java objects at both ends of the conversion, so I don't think that will help. Obviously a C style memcpy() would be best, but I don't see any "legal" way to do such things in JAVA.

Any ideas?

[keang]

Have you considered using java.io.DataInputStream and java.io.DataOutputStream to convert byte streams to primitive types and visa versa.

[Randy C]

Have you considered using java.io.DataInputStream and java.io.DataOutputStream to convert byte streams to primitive types and visa versa.

Yes... thanks. That's exactly what I'm looking at now. I suppose, not understanding how much is going on under the hood with serialization, it seemed like something that might cause a major performance hit. But maybe I'm worried about nothing in this case. And i've lost a lot of my zeaol to invent/re-invent wheels over the years. :-)

[keang]

I don't think serialization is going to be useful to you. Serialization is for turning objects and their contents into a transmittable stream of data that can, at some future time, be deserialized back into an object.

[Randy C]

@Keang: That's exactly what I need to do. When raw binary data comes "in" to my application from an outside device, its going to come as a raw stream of bytes. If that stream includes ten 4 byte IEEE floating point numbers, then that's 40 bytes. And since I have no way to "memcpy()" each block of 4 bytes directly into the address of a float, I need those serialization routines. Ditto for sending a similar group of floats. I'll need to first transform them into an array of bytes to transmit them. That is, unless someone knows of some other hidden "memcpy()" equivalent.

The only remaining problem I see occurs when the external device I'm communicating with is sending its data in LO->HI byte format (little endian). In that case I somehow need to reverse objects within an incoming byte buffer BEFORE it is deserialized into a float, short, int, or double. The most immediate way that comes to mind is to maintain an additional stream. The first would take raw bytes coming in from the external device and place them in a buffer. I would then re-order the items within that buffer as needed, and then write that buffer as a byte stream to a local file. I could then read that data back using the dataStream method, pulling the now "endian corrected" objects into appropriate variables.

Now that is a solution that will work, but is terribly inefficient because of the need to maintain that second stream to and from a file. Id obviously have to use the random access abilities of the file IO stream class, and HOPE (if I'm lucky) the internal ram buffering will mean that there really is no (or very little) actual disk access to speak of.

What do you think?

[keang]

That's exactly what I need to do

No it isn't. You are dealing with primitive types and not objects.

If that stream includes ten 4 byte IEEE floating point numbers, then that's 40 bytes. And since I have no way to "memcpy()" each block of 4 bytes directly into the address of a float, I need those serialization routines.

No, you need to use something like DataInputStream, as I suggested earlier, which will read the byte stream and recreate a primitive type for you.

The first would take raw bytes coming in from the external device and place them in a buffer. I would then re-order the items within that buffer as needed, and then write that buffer as a byte stream to a local file. I could then read that data back using the dataStream method, pulling the now "endian corrected" objects into appropriate variables.

Wow that's overly complicated. If you are dealing with byte streams just write a Reordering byte stream class that sits between the input stream and the DataInputStream object. The class needs to maintain a buffer of size 2 which it fills in reverse order from the input stream and then stops accepting input until the buffer is read by the DataInputStream. On a read request it outputs the buffer in the correct order and blocks further reads until the buffer is refilled.

In fact if you search on-line you may even find someone has already done this.

[Randy C]

Keang:

From your last comment, I think you now understand my problem. I can't simply use dataInputStream's methods to pull data into various primitives without first ensuring the byte streams that represent them have been pre-corrected to properly represent the JAVA version of such primatives.

I certainly do think what you're suggesting is much more eloquent. My only problem at this point is that I do not sufficiently understand the inner workings of the java.io classes to understand how to write such a go-between class. I have experimented with the Oracle tutorials on Java.io enough to see an example where a DataInputStrem wraps a BufferedInputStream, which in turn wraps a FileInputStream, like this...

DataInputStream in= new DataInputStream(new
BufferedInputStream(new FileInputStream(dataFile)));
// where 'datafile' is a string indicating the path/file of the source data

Of course in my actual apps I'll likely not be using FileInputStream, because I'll be communicating and receiving data from external devices via a TCP/IP link. But assuming the rest of the food chain is the same, I take it you are advocating I replace the BufferedInputStream in the above with a ReOrderBufferedInputStream class of my own. which "extends" BufferedInputStream?

I'm probably way off base already, but like i said even if I'm on the right track, I have a low confidence level at this point that I understand these classes well enough to do such a thing. (No excuses, but I was a C/C++ programmer, and only started working with JAVA a month ago.). So I may have to either use something like my more ugly solution, or hope (as you suggest) I can find an example of a similar solution already coded as a starting point.

[keang]

I can't simply use dataInputStream's methods to pull data into various primitives without first ensuring the byte streams that represent them have been pre-corrected to properly represent the JAVA version of such primatives.

So you have two issues:
1. Byte Ordering
2. Primitive formatting

Byte ordering is easy to handle - If your IED requires byte reordering simply plug your ByteReorderingInputStream class into the input stream chain.

Formatting is a slightly harder problem. You will need to write a reformattting input stream for each possible format that you will receive and plug the appropriate one into the input stream chain.

You will need some way of identifying which IED you are receiving data from and which input stream classes are needed to correctly parse the data.

My only problem at this point is that I do not sufficiently understand the inner workings of the java.io classes to understand how to write such a go-between class.

It's not hard, there is a class provided for the purpose - FilterInputStream. This class wraps an InputStream and simply overrides all the InputStream methods to pass the calls onto the wrapped streams methods. You extend the FilterInputStream class and override the methods you need to do your work in.

For example: For the read(..) method you call the wrapped objects read(..) method, get the returned data, do your transform on it and pass the transformed data back to the calling method.

Note: As you are dealing with multiple bytes you may need to perform some level of buffering internally so you can perform the transform correctly.

If you are still unsure how to do this look at the way sun implemented the FilterInputStream and BufferredInputStream classes. The code is readily available.

[Randy C]

Thanks Keang! That doesn't sound so bad, and I'll definitely look into this! I may have used the wrong word when I spoke of "re-formatting". I mean, its not like I'll have to manually convert floats to doubles, or manually convert UTF strings. With the DataInputStream (and DataOutput) stream calls, and the reasonably complete set of JAVA primitives available, I'm pretty sure byte ordering is all I'll have to worry about. There is always the starting premise that you know enough about the IEDs message structure to know what data types to expect, and in what order. I just need to develop a flexible way of dealing with byte orders, in the JAVA world.

FilterInputStream sounds like just what the doctor ordered, but it makes me aware that I've certainly not yet discovered what I'd call the "real" detailed "reference" documentation for the standard class libraries (like java.io) on the Sun/Oracle website. So far I've mostly found tutorials, which probably avoid bogging down in too many details. Just so I can better help myself in the days ahead, what is the proper way to drill down through the Sun documentation, for example to really explore all that the various java.io (and likely java.nio) classes have to offer?

By the way, coming from the Microsoft VStudio C++ world, regularly wasting days hunting down obscure API methodologies from the "not so well structured" MSDN, I'm am EXTREMELY encouraged so far with JAVA, and the way Sun has gone about documenting it. I just haven't found where all the marbles are hidden yet!

--Randy

[keang]

Two things you need to get hold of (you may already have them depending on which jdk pack you downloaded):

1. Java API docs. These holds details of every public and protected class, method, variable etc and generally have pretty good descriptions of what they all do. The class descriptions often include examples and or links to tutorials. The API docs are available on-line but you'll use them so much it's worth downloading them.

2. The Java source. The source code for all the above classes is available in a rather large zip file so you can actually look at how the Java programmers went about solving a problem - why re-invent the wheel.

[Randy C]

@Keang, Hey if you don't mind me picking your brain a little more... In that situation I described last week, where I'll SOMETIMES need to reverse the byte order before DataInputStream assigns them to Java primitives, shouldn't I be writing a class to inherit from DataInputStream rather than using FilterInputStream to wrap InputStream? Looking at the calls for these classes, it seems it's not until we get to all the "read" calls of DataInputStream (readFloat, readInt, etc), that it is actually known how many bytes to pull from the InputStream. So it would seem that if I made my own class that inherited from DataInputStream, I could then make overrides of each 'read" call that included another argument, a boolean, to indicate whether or not to reverse the byte order in that case. But I don't know... maybe the calls in DataInputStream can't be overridden? Or are ALL public calls ALWAYS be overridable in JAVA?

[keang]

. In that situation I described last week, where I'll SOMETIMES need to reverse the byte order before DataInputStream assigns them to Java primitives, shouldn't I be writing a class to inherit from DataInputStream rather than using FilterInputStream to wrap InputStream?

I guess it depends on the data stream. If all data is sent as pairs of bytes that need to be in reverse order then do as I suggested, however if the number of bytes and their order depends on the type then you will need to override DataInputStream.

So it would seem that if I made my own class that inherited from DataInputStream, I could then make overrides of each 'read" call that included another argument, a boolean, to indicate whether or not to reverse the byte order in that case.

You would be better using a DataInputStream for those IED's that don't require byte reordering and your new class for those that do. Adding a boolean flag is pointless unless an IED sends some data types in normal ordering and some in reverse ordering, which I can't imagine being the case.

Or are ALL public calls ALWAYS be overridable in JAVA?

All non-private methods that are visible to the extending class can be overridden unless they have the 'final' modifier.

[Randy C]

I guess it depends on the data stream. If all data is sent as pairs of bytes that need to be in reverse order then do as I suggested, however if the number of bytes and their order depends on the type then you will need to override DataInputStream.

You would be better using a DataInputStream for those IED's that don't require byte reordering and your new class for those that do. Adding a boolean flag is pointless unless an IED sends some data types in normal ordering and some in reverse ordering, which I can't imagine being the case.

Unfortunately, its usually a very mixed bag. Parsing binary IED data has always been a messy business. Often, not only will there be many disparate data types within a message, but just as you thought unthinkable, some manufacturers actually DO use one endian format for integral data, and another for floats. So I really do need control on a "one primitive at a time" basis.

All non-private methods that are visible to the extending class can be overridden unless they have the 'final' modifier.

Yeah, and I just realized that all those read methods of DataInputStream (readInt(), readFloat(), etc) all seem to be "final". GRRRR.... Well, like you mentioned, I can always find the source code and make my own version.

But I realized another problem today that further complicates the problem. Binary IED data normally comes in blocks with header data and some kind of trailing CRC. I've already written JAVA methods to calculate what the CRC should be, but to do so I have to first read the whole message into a byte array. So what I intend to do is read in the block, and AFTER it is verified (CRC), I can then easily fix all the byte orders of all primitives within the byte array block, because I know the IED's data structure.

Once the block is verified and all embedded primitives are endian corrected, it would be nice if I could then reinterpret the block on a primative by primitive basis, with the original DataInputStream class. the only way I can think of doing that, however, is to write the block to a random access file on the local disk, reset the file pointer to zero, and then read the file again, using the DataInputStream (wrapping BufferedInputStream, wrapping FileInputStream). This sounds ugly, but it does work. To make it less ugly, it would be nice to eliminate the interim file. So maybe I'm back to that 'FilteredInputStream Class, where I could probably create a call to switch between drawing bytes from the original TCP/IP stream, or switching over to my byte array, where I've already qualified and endian corrected my message.

Well...I just hope all of FilteredInputStream's methods aren't all marked final too! :-)

thanks for putting up with my rambling!

[keang]

the only way I can think of doing that, however, is to write the block to a random access file on the local disk, reset the file pointer to zero, and then read the file again, using the DataInputStream (wrapping BufferedInputStream, wrapping FileInputStream). This sounds ugly, but it does work.

Create a ByteArrayInputStream with your reordered byte data and pass this object as the InputStream when creating the DataInputStream object. The DataInputStream will then read the bytes from your byte array without having to go via a file.

[Randy C]

Thanks again keang! That worked great! A perfect and very flexible solution! Working with that ByteArrayInputStream does seem a little weird, only because I construct it by passing a completely external buffer, which I still have to manipulate (fill or modify) apart from any method provided by the class. Had I written a class that way I've feel like I violated some JAVA rule, but I guess not. In any case, this works perfectly


BTW, Yesterday someone else pointed out to me that the java.nio ByteArray class has all kinds of flexible methods for dealing with big/little endian data, and at first glance it looks like that might be the case. Not unusual for working with a new language, here I am re-inventing wheels and solving problems others seem to have already addressed. But this has still been a very useful exercise for me. And hopefully I won't find out that "nio" makes "io" classes obsolete, or that they shouldn't be used or are deprecated!