parse a string to bytearray to struct

[cbpetro]

I'm trying to parse a string into individual variables defined by a struct into a byte array. I want to pass the byte array to a com server object and not the string because it's smaller size, correct?

I wrote a function "GetByteArray" to parse the string into the byte array. The problem I have is the byte array does not cast back the floats into my TD struct. I also noticed the size of the byte array written is different than the structure. Why is that?


If there is better approach, please suggest one.

#include "stdafx.h"
#include <stdio.h>
#include <string>

struct TD
{
	short sYear;
	short sMonth;
	short sDay;

        float fVal1;
	float fVal2;
};

void GetByteArray(const std::string& str, unsigned char* bytes)
{
	// get chars 0-3 from str to get string for year 
	std::string strYear = str.substr(0,4);
	short sYear = (short)atoi(strYear.c_str());
	// get chars 5-6 from str to get string for month 
	std::string strMonth = str.substr(5,2);
	short sMonth = (short)atoi(strMonth.c_str());
	// get chars 8-9 from str to get string for day 
	std::string strDay = str.substr(8,2);
	short sDay = (short)atoi(strDay.c_str());
	// get chars 11-17 from str to get string for Val1 
	std::string strVal1 = str.substr(11,7);
	float fVal1 = (float)atof(strVal1.c_str());
	// get chars 19-25 from str to get string for Val2 
	std::string strVal2 = str.substr(19,7);
	float fVal2 = (float)atof(strVal2.c_str());

	size_t i = 0;
	// cast strYear to short and append to byte array
	memcpy(&bytes[i], &sYear, sizeof(short));
	i += sizeof(short);

	// cast strMonth to short and append to byte array
	memcpy(&bytes[i], &sMonth, sizeof(short));
	i += sizeof(short);

	// cast strDay to short and append to byte array
	memcpy(&bytes[i], &sDay, sizeof(short));
	i += sizeof(short);

	// cast strAsk to float and append to byte array
	memcpy(&bytes[i], &fVal1, sizeof(float));
	i += sizeof(float);

	// cast strBid to float and append to byte array
	memcpy(&bytes[i], &fVal2, sizeof(float));
	i += sizeof(float);
}

int main(int argc, char* argv[])
{
	// string to parse
	std::string strData = "2011.06.30,1.40562,1.40559";

	unsigned char MyBytes[sizeof(TD)];
	GetByteArray(strData, MyBytes);

	// i = 14 from GetByteArray
	int i = sizeof(TD); // i = 16 from structure size

	// error here! my floats in td are undefined
	TD* td = (TD*)MyBytes;

	return 0;
}

[Paul McKenzie]

I wrote a function "GetByteArray" to parse the string into the byte array. The problem I have is the byte array does not cast back the floats into my TD struct.

What made you think that there isn't any padding bytes between those struct members?

In other words, adding sizeof(type) to your "i" variable isn't guaranteed to get you to the specific member. Bottom line is that unless the struct has no padding bytes in between those members (some compilers have a #pragma or some other means to pack the struct), your code isn't going to work.

Regards,

Paul McKenzie

[cbpetro]

ok thanks. This works for my microsoft 32-bit compiler.

#pragma pack(push)  // push current alignment to stack
#pragma pack(1)     // set alignment to 1 byte boundary
struct TD
{
	short sYear;
	short sMonth;
	short sDay;

        float fVal1;
	float fVal2;
};
#pragma pack(pop)   // restore original alignment from stack

[Paul McKenzie]

I also noticed the size of the byte array written is different than the structure

Going back, how did you "notice" this? Were you (erroneously) adding up the number of bytes of each member? You should never determine the size of the structure this way. Always use sizeof() to get the size of any type.

Also, your code did make the byte array the same size as the structure, even in your original non-working code. You used sizeof(TD) to determine the number of bytes in the array.

Regards,

Paul McKenzie

[cbpetro]

Going back, how did you "notice" this? Were you (erroneously) adding up the number of bytes of each member?

No, I don't believe so. The size of my TD structure is different with and without the pragma directives. Adding up the number of bytes always is 14.

	int j = sizeof(TD); // j = 14 from structure size with pragma
	int k = sizeof(TD); // k = 16 from structure size without pragma

[Paul McKenzie]

No, I don't believe so. The size of my TD structure is different with and without the pragma directives. Adding up the number of bytes always is 14.

No, you didn't understand.

In your original code and post, you stated that the sizeof the byte array was different than the size of the structure. That is false.

The size of the byte array was always the same size as the structure. Would you like proof? Take away the pragma, and do this:

#include <assert.h>

unsigned char MyBytes[sizeof(TD)];
//...
assert( sizeof(MyBytes) == sizeof(TD) );

You will see that assertion will always be true.

Regards,

Paul McKenzie

[cbpetro]

I see, you are correct. But I don't understand why at the end of my "GetByteArray" function the sum of the bytearray is 14. Is summing up the size of each variable for placement in the byte array correct?

void GetByteArray(const std::string& str, unsigned char* bytes)
{
	// get chars 0-3 from str to get string for year 
	std::string strYear = str.substr(0,4);
	short sYear = (short)atoi(strYear.c_str());
	// get chars 5-6 from str to get string for month 

        ....
        ...
        ..

	// cast strAsk to float and append to byte array
	memcpy(&bytes[i], &fVal1, sizeof(float));
	i += sizeof(float);

	// cast strBid to float and append to byte array
	memcpy(&bytes[i], &fVal2, sizeof(float));
	i += sizeof(float);

        // i = 14 ?
}

If I add another variable to the byte array it will be at location 14 when the byte array size is 16. This does not make sense to me.

[MrViggy]

Read Paul's original response:

What made you think that there isn't any padding bytes between those struct members?

The sizeof a struct isn't necessarily the sum of the sizeof it's parts.

Viggy

[ninja9578]

ok thanks. This works for my microsoft 32-bit compiler.

#pragma pack(push)  // push current alignment to stack
#pragma pack(1)     // set alignment to 1 byte boundary
struct TD
{
	short sYear;
	short sMonth;
	short sDay;

        float fVal1;
	float fVal2;
};
#pragma pack(pop)   // restore original alignment from stack

GCC way

#pragma pack(push, 1)
struct TD {
...
};
#pragma(pop)

[cbpetro]

Read Paul's original response:

The sizeof a struct isn't necessarily the sum of the sizeof it's parts.

Viggy

But we've determined that the size of struct = size of byte array. Therefore the bytearray is padded as well, correct?

Again, I ask do I have to consider the padding each time I add a new set of bytes to my byte array.

[MrViggy]

No. An array is a continuous chunk of memory, with each item in the array having the same size. It's not a grouping of different things. A struct is a grouping of (possibly differently sized) things.

Viggy

[Paul McKenzie]

Again, I ask do I have to consider the padding each time I add a new set of bytes to my byte array.

You have to keep your struct packed on the byte boundary to not introduce any gaps in-between the members.

The issue you need to consider is if you or someone else changes the order of the members of your struct. Then you have to change the reading code to match the order of the items. Another thing to consider is if you add a member in the middle of the struct.

Maybe you should consider "auto-generating" the struct and the code to read it, as right now, manually making changes could lead to hard-to-find bugs. Write a program that creates a header of the struct, and the C++ code to process it.

Regards,

Paul McKenzie

[VladimirF]

I'm trying to parse a string into individual variables defined by a struct into a byte array. I want to pass the byte array to a com server object and not the string because it's smaller size...

If you really want the smaller size, note that both month and day fit in single byte.
Coincidentally, since short and two bytes take exactly 4 bytes, you wouldn’t have a packing issue…

[monarch_dodra]

As mentioned, it is usually not possible to assume a struct will align correctly on any given byte array. That, or the amount of assumptions/restrictions you need to do are usually prohibitive.

What you are trying to do is not unlike serialization. The easiest method (and usually recommend method I believe), is to just take your attributes 1 by 1, and write them to the array. To unserialize, read your blocks and write them to your attributes 1 by 1.

It might sound like a hassle, but it is just writing 2 functions, and performance usually isn't so mind-bogglingly tight you can't afford it. Besides, you already wrote 1 of the two functions, why not just write the other?

#include <string>  //string

#include <iostream>  //cout
#include <iterator>  //ostream_iterator
#include <algorithm> //string
#include <iomanip>   //hex setw

const size_t ARRAY_SIZE = sizeof(short)*3 + sizeof(float)*2;

struct TD
{
  short sYear;
  short sMonth;
  short sDay;

  float fVal1;
  float fVal2;
};

void WriteByteArray(const TD& td, unsigned char* bytes)
{
  size_t i = 0;

  // cast strYear to short and append to byte array
  (short&) bytes[i] = td.sYear;
  i += sizeof(short);

  // cast strMonth to short and append to byte array
  (short&) bytes[i] = td.sMonth;
  i += sizeof(short);

  // cast strDay to short and append to byte array
  (short&) bytes[i] = td.sDay;
  i += sizeof(short);

  // cast strAsk to float and append to byte array
  (float&) bytes[i] = td.fVal1;
  i += sizeof(float);

  // cast strBid to float and append to byte array
  (float&) bytes[i] = td.fVal2;
  //i += sizeof(float);
}

void ReadByteArray(unsigned char const * bytes, TD& td)
{
  size_t i = 0;

  // cast strYear to short and append to byte array
  td.sYear  = (short&) bytes[i];
  i += sizeof(short);

  // cast strMonth to short and append to byte array
  td.sMonth = (short&) bytes[i];
  i += sizeof(short);

  // cast strDay to short and append to byte array
  td.sDay   = (short&) bytes[i];
  i += sizeof(short);

  // cast strAsk to float and append to byte array
  td.fVal1 = (float&) bytes[i];
  i += sizeof(float);

  // cast strBid to float and append to byte array
  td.fVal2 = (float&) bytes[i];
  //i += sizeof(float);
}

int main(int argc, char* argv[])
{
  TD td = {2011, 7, 15, 1.5, 1e10};
  unsigned char MyBytes[ARRAY_SIZE];

  std::cout << "Array size: " << ARRAY_SIZE << std::endl;

  std::cout << std::hex << std::setw(3);

  WriteByteArray(td, MyBytes);
  std::cout << "encoded data:" << std::endl;
  std::copy(MyBytes, MyBytes+ARRAY_SIZE, std::ostream_iterator<int>(std::cout, ", "));
  std::cout << std::endl;

  unsigned char MyBytes2[ARRAY_SIZE] =
  {
    0x9e, 0x07, // 1950
    0x0c, 0x00, // 12
    0x1e, 0x00, // 30
    0xf9, 0x02, 0x15, 0x50, //1e10
    0x00, 0x00, 0xc0, 0x3f  //1.5
  };

  ReadByteArray(MyBytes2, td);
  std::cout << std::dec << std::setw(0);
  std::cout << "decoded data:" << std::endl;
  std::cout << td.sYear << " " << td.sMonth << " " << td.sDay << " " << td.fVal1 << " " << td.fVal2 << std::endl;

  return 0;
}

Note that neither endianness nor the fact that "short" is not the same on all machines, were taken into account here. If endianness should be taken into account, then you have use this kind of per-object approach anyways.