Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
I'm getting PVOID data continuously
Where is this data coming from and how does it get set into the memory pointed to by pvData? Is one thread producing this data and another thread processing it?
Re: Processing Time For PVOID Into Byte Array Conversion
Hi Mr.2kaud & Mr.Paul,
Sample code like below,
Code:
byte TempArr[1024];
PVOID pvData = NULL; //store a line data into a pointer
pvData = GetCardData(); //I'm just using this function only. All functions are in the form of dll.
// Its getting from owner of card seller. using this Getting one line data = 1024 bytes @ every milliseconds
byte *bPoint = (byte*)pvData;
for(int i = 0; i < 1024; i++) TempArr[i] = (byte)(*(bPoint + i)); //Started to copy the data for drawing purpose
//For this transaction only I'm getting delay.
But using this I'm missing a line data for every line like 1,3,5,7,.... etc.
Quote:
Originally Posted by
Paul McKenzie
you have to ensure that you don't miss any data coming in. You don't do that by "coding the fastest routine" and hoping that the routine processes all the data before some real-time function refreshes the buffer with new data. That is a naive and wrong approach. You have to use the proper data structures (probably a queue), that caches the data that comes in so you don't miss anything.
after getting a line data, If I add any process means, the missing line increases like 1,5,11,etc... Sure Data are coming correctly. Myself only missing. Where is the problem?
Quote:
Originally Posted by
2kaud
Where is this data coming from and how does it get set into the memory pointed to by pvData? Is one thread producing this data and another thread processing it?
I'm using a card it gives the data at every millisec 1024 bytes. And the memory pointed to pvData. Using thread concept I'm pointing Data by pvData continuously and using paint method to draw an image.
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
saraswathisrinath
... after getting a line data, If I add any process means, the missing line increases like 1,5,11,etc... Sure Data are coming correctly. Myself only missing. Where is the problem?
It seems that your painting takes much more time than reading the data...
Quote:
Originally Posted by
saraswathisrinath
I'm using a card it gives the data at every millisec 1024 bytes. And the memory pointed to pvData. Using thread concept I'm pointing Data by pvData continuously and using paint method to draw an image.
Which "thread concept" do you mean? I don't see any "thread concept" in the code snippet you have posted.
Besides, like Paul wrote:
Quote:
Originally Posted by Paul McKenzie
You have to use the proper data structures (probably a queue), that caches the data that comes in so you don't miss anything.
Do you use something like that?
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
saraswathisrinath
[/code]But using this I'm missing a line data for every line like 1,3,5,7,.... etc.
after getting a line data, If I add any process means, the missing line increases like 1,5,11,etc... Sure Data are coming correctly. Myself only missing. Where is the problem?
The problem is that you didn't put much thought in your design.
The solution is very simple in concept, and I explained briefly before, I will try again.
1) You are trying to write the fastest routine possible so that you can process the data quickly enough before the buffer gets overwritten with new data. This is the wrong approach.. There is no guarantee that your fast routine will be fast enough, so get rid of this idea of trying to write code that attempts to be speedier than the hardware that is sending you the data. What is speedy today may miss a byte or two of data tomorrow.
2) The solution or one such solution is to cache the data coming in, so that you don't lose any data. You save the incoming data on the back of the queue, while at the same time, you're processing the data that is sitting in front of the queue.
For the solution, you make the queue big enough so that it fits in with the processing time it takes to do one batch of data, plus accept any new data coming in. This is where experiments and trial/error comes in to see how big to make the queue. Once you have that set up, there is very little chance at all of missing data. If your drawing code is slow, then you make the maximum queue size large, if it's fast drawing code, then you can make the maximum queue size smaller. So basically, the speed of your graphics code won't matter.
Think of this like a cashier at a supermarket. The customers want to pay for the items they bought, so they must stand in line (a queue), waiting to be processed. Yes, the cashier needs to process each customer quickly, but there is no way a cashier can process a customer every single second or every two seconds.
Your code is trying to be the cashier that can process a customer every one or two seconds, therefore process each customer as they are about to check out of the store (which means no line is ever formed). You know that is practically impossible for a cashier to process a customer every second. So the solution is the same as what a supermarket does -- you create a line (queue) that where incoming checkout customers line up to be processed. If the cashier is not as fast as expected, then the queue will be longer, but every customer will be processed.
In your case, the customers are the same as the data the card is giving you, your graphics/drawing code is the cashier. So can you envision this? You make a queue for the data that the card is sending you. Any new data the card sends you gets placed at the back of the queue, while your graphics code processes the data that is in front of the queue.
For the queue, you will need proper synchronization so that reading and writing to the queue are done in a safe manner. So the only major thing you would need to do now is to implement the queue. There is std::queue from STL that can work for this, but again, you'll have to add the synchronization to ensure that you're not writing to the queue while reading from the queue, etc.
If your drawing code is very fast, e.g. the queue is empty most of the time, then that's ok -- just wait for data to be placed on the queue by the card.
Again, all of this should have been envisioned if you truly wanted to process real-time data. In no way should you have started trying to beat the card's speed. What if the processor(s) you were writing for are slow? What if the compiler's optimizer was bad (I know it's not, but what if it was slow)? What then? Get a faster computer? Or do you write the code so that no matter how slow or "bad" the drawing code is, it will process all the data that's coming in.
Regards,
Paul McKenzie
Re: Processing Time For PVOID Into Byte Array Conversion
Other processes running would also affect the speed of your program.
I am just wondering if one line of data (1024 bytes) will be drawn to one line of the display that was created in Draw Image Using 2D Byte Array ?. Does this display scroll vertically after each line is drawn? Or does each successive line of data get drawn on a successive line of the display until the display is full and then the drawing starts at the top of the display again? What are the dimensions of the display? (Width = 1024?, Height = ?) With the rate of data you are getting (1,024,000 bytes per second), it seems that the entire display would be redrawn a few times a second. Is this data coming from a camera? The InvalidateRect() function only needs to be called at most 60 times per second. Calling it more often would be unnecessary since your monitor does not update that fast.
Is the DLL allocating the memory for the incoming data array? Is this memory location static? If not, where is it freed? If your code could allocate the memory and pass the DLL a pointer to that memory, you could pass the DLL a pointer to a line of your image data. This would do the drawing automatically.
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
VictorN
It seems that your painting takes much more time than reading the data.
I Already removed Paint and run my application. But no changes in the result.
Quote:
Originally Posted by
VictorN
Which "thread concept" do you mean? I don't see any "thread concept" in the code snippet you have posted.
First I used Timer. called at every 1 milli seconds. This case I missed 15 to 20 lines. So I go to threads.
Your reference,
Code:
CWinThread *conThread;
PVOID pDaqData = NULL;
RETURN_CODE rc;
byte *bPnt;
void CMyDlg::OnBnClickedRealTimeData()
{
if(m_bDaqContFlag)
{
UpdateData(TRUE);
pDaqData = NULL;
conThread = ::AfxBeginThread(RealDataThread, this);
}
else
{
//End Data Capture
conThread->SuspendThread();
DataAaqEnd(m_nDevNo); //Dll function from Card Seller
}
}
UINT CMyDlg::RealDataThread(LPVOID rd)
{
CMyDlg *obj = (CMyDlg *) rd;
obj->realData();
return true;
}
void CMyDlg::realData()
{
LARGE_INTEGER m_lnStart;
LARGE_INTEGER m_lnEnd;
LARGE_INTEGER m_lnFreq;
QueryPerformanceFrequency(&m_lnFreq);//Retrieves the frequency of the high-resolution performance counter
QueryPerformanceCounter( &m_lnStart);//Retrieves the current value of the high-resolution performance counter
pDaqData = GetData(m_nDevNo, &rc); //Dll function from Card Seller
bPnt = (byte*)pDaqData;
byte *bTo = TempRealDataArray;
memcpy(bTo, bPnt,2048);
for(int i = 0; i < 2048; i++)
{
TempRealDataArray[i] = (*(bTo + i)); //This array data passed to paint method to drawn an image.
if(i == 1030) Temp[cnt] = TempRealDataArray[i]; //To find the missed lines.
// Temp[] stored the single data per line and print a file after stop the process
}
cnt++;
QueryPerformanceCounter( &m_lnEnd );
__int64 nDiff(0) ;
double dMircoSecond( 0 ) ;
if( m_lnStart.QuadPart !=0 && m_lnEnd.QuadPart != 0)
{
dMircoSecond = (double)(m_lnFreq.QuadPart)/1000000;
nDiff = m_lnEnd.QuadPart-m_lnStart.QuadPart ;
dMircoSecond = nDiff/dMircoSecond;
CString str;
str.Format("%d", nDiff);
GetDlgItem(IDC_TIMEDIF)->SetWindowText(str); //Takes 1200 to 1500 micro seconds
}
InvalidateRect(&rect); //Already defined the rect { 0, 0, 1024, 256}
conThread = ::AfxBeginThread(RealDataThread, this);
}
I don't know, here Am I using the proper Data Structure ? Please guide me.
Quote:
Originally Posted by
Paul McKenzie
The solution or one such solution is to cache the data coming in, so that you don't lose any data. You save the incoming data on the back of the queue, while at the same time, you're processing the data that is sitting in front of the queue.
Am I cache the Real Data correctly? Please refer the code above which I post.
Mistake is in my side. But sure I tell Painting & the compiler's optimizer not reason for this. B'cos I removed paint method and run my code and also run my application with higher configuration PC.
Quote:
Originally Posted by
Coder Dave
What are the dimensions of the display? (Width = 1024?, Height = ?)
Width = 1024 & Height = 256.
Now, My problem is not drawing the image.
My first aim is getting the data correctly from camera with out any missing data.
If I achieve this then my problem solved 90%.
Pls take me at right path.....
Re: Processing Time For PVOID Into Byte Array Conversion
Code:
for(int i = 0; i < 2048; i++)
{
TempRealDataArray[i] = (*(bTo + i)); //This array data passed to paint method to drawn an image.
if(i == 1030) Temp[cnt] = TempRealDataArray[i]; //To find the missed lines.
// Temp[] stored the single data per line and print a file after stop the process
}
The highlighted line in red just copies TempRealDataArray to itself! - as bTo is set to point to the beginning of TempRealDataArray.
The green line is just an attempt at a kludge to get the data because you are not using the proper structures for obtaining the data that have been suggested previously by Paul. The above code is not needed and should be removed.
You need to implement properly a queue with the producer being your realdata method and the consumer being the method/function to process or save the data as needed. The producer adds to the queue at the back and the consumer takes from the queue at the front. The queue consists of elements of 2048 bytes. Everytime your realdata method in its thread calls getdata() it copies the data to a new queue element at the back of the queue (add queue element - push in STL terms). Your consumer function takes elements from the front of the queue if there are any and processes the data (remove queue element - pop in STL terms). This can be done using the STL queue class and appropriate thread syncronisation.
See below for details of the STL queue
http://www.cplusplus.com/reference/queue/queue/
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
saraswathisrinath
I Already removed Paint and run my application. But no changes in the result.
The retrieval of the data from the card should do nothing except place the data onto the queue. Once the data is copied to the queue, that's it -- the reading of the card data is done. Instead, you're calling InvalidateRect and all sorts of other things during the reading of the data -- this is wrong.
The producer would do something like this.
Code:
while (forever)
{
while (card_says_I_have_data)
AddDataFromCardToBackOfQueue();
}
If it's anything more than that, you're not understanding the concepts that I've given you as well as 2kaud and Victor. You shouldn't be invalidating rectangles, putting together drawing buffers, etc.
On another thread (the consumer), you should then do this (this is happening "at the same time" as the code above):
Code:
while (forever)
{
if ( queue_not_empty )
{
RemoveDataFromFrontOfQueue()
DrawMyImageWithTheData();
}
}
The while (forever) will quit at some time, either it is signalled that no more data will come in or by some other means. Of course, you have to use the proper synchronization between the consumer/producer threads so that the queue doesn't become corrupted. You could even "lock" the queue with the RemoveDataFromFromOfQueue (and even in this case, to make the read "fast", you only read at most, say 500 bytes at a time). Then when the Read is done, you, unlock the queue so that the producer (the card) places data into the queue.
If you don't understand the producer/consumer paradigm or do not understand what a queue is, just say so. It seems you're not quite getting why your method is flawed, and why I suggested that you should just scrap it and use proper techniques. Just copying data to an array in a "simple" loop from begin to end is not a queue. You can use an array as a queue (but not necessary as there is std::queue built for this), but unless you use that array correctly as a queue, you're not going to get anywhere.
A queue requires that you copy data to the back of the array, while the processing of the data uses the data in the front of the array.
As an example, let's say that the queue has room for 10,000 bytes. Then 2,000 bytes of data comes in from the card (the producer). You place this data in the queue, and at the same time, the drawing function (the consumer) is looking at the queue for any data -- it is not looking or even care what the card is doing.
So now let's say your drawing function has drawn 1,500 bytes of data, and then the card sends 2000 bytes of data. Your card reading thread takes this data and places the card data in the queue -- nothing is being overwritten, since the data is placed in the back of the line.
The queue had 9500 bytes of space since the drawing completed 1,500 bytes. At the same time, the drawing function finishes up with the remaining 500 bytes it had to do the first time, then it looks at the queue and sees that there are 2,000 more bytes of data, and then draws that data.
Let's say the drawing function now processes 1,999 bytes of data, and then the card sends another 2000 bytes of data right when the drawing program is going to draw the last byte. You place the new card data at the back of the queue (which can hold 9,999 bytes, since the drawing processed 1,999 bytes). The drawing function finishes drawing that last remaining byte, looks at the queue, and sees it needs to draw more data.
This producer/consumer interaction goes on forever, until the drawing ends. Do you now see that you never miss any data, provided that the queue is big enough? Do you see where your current method fails miserably? Do you see why having the fastest code in the world doesn't matter with using a queue? With your current method, all of the times when the drawing function didn't draw all the bytes, you got into trouble. With the queue, you will never get into trouble (and your function to draw doesn't even need to be fast, albeit it is good that it is fast).
Quote:
First I used Timer. called at every 1 milli seconds. This case I missed 15 to 20 lines. So I go to threads.
Threads are not going to help you -- you need to use the proper data structures, and you're not doing it. If you used the proper structure, it doesn't matter what the timings are.
Code:
TempRealDataArray[i] = (*(bTo + i)); //This array data passed to paint method to drawn an image.
There should be no drawing at all, or even mention anything about paint methods. This is the time where you place the data onto the queue and return.
Code:
if(i == 1030)[B] Temp[cnt] = TempRealDataArray[i]; //To find the missed lines.
// Temp[] stored the single data per line and print a file after
Code:
GetDlgItem(IDC_TIMEDIF)->SetWindowText(str); //Takes 1200 to 1500 micro seconds
And if it took 10,000 microseconds, then what do you do? What should be happening is that this painting routine doesn't care one bit about the timing, as long as the queue has room for more data coming in from the card. If the drawing routine takes a long time so that the queue is too small, then make the queue bigger -- that's how you fix the problem. You don't fix it by trying to write the fastest code in the world.
Quote:
Mistake is in my side. But sure I tell Painting & the compiler's optimizer not reason for this. B'cos I removed paint method and run my code and also run my application with higher configuration PC.
Again, forget about what the compiler's optimizer does, and forget about how fast your PC is, and honestly forget about QueryPerformanceCounter() -- pretend that function doesn't exist. None of those things would hardly matter if you coded using the proper data structures.
Quote:
My first aim is getting the data correctly from camera with out any missing data.
If I achieve this then my problem solved 90%.
If you took the advice of creating a queue, then 100% or close to it would be solved.
Quote:
Pls take me at right path.....
We are trying, but are you listening? So far you're still stuck on using the same broken design. Again, forget about what you wrote -- it won't work if you do not want to miss any data.
My feeling is that you're trying to code in a "simple" fashion instead of actually coming up with a design. The card has data, you draw the data to the screen, and if the data comes in too fast, you create the drawing code even faster. Sure, that's the easy way to think about it, and frankly any beginner would have done exactly what you've done. But it isn't correct and it is seriously flawed. You have to actually use certain design patterns to overcome the problem.
Regards,
Paul McKenzie
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
saraswathisrinath
Code:
memcpy(bTo, bPnt,2048);
Why are you copying 2048 bytes when the input buffer is only 1024 bytes?
So your application is basically displaying the output of a black and white digital camera in a window. You are receiving 1,024,000 bytes/pixels per second from the camera. Your display has 1024 x 256 = 262,144 pixels, so it takes 256 milliseconds to receive one full image from the camera. That is, you are displaying 1,024,000 / 262,144 = 3.90625 full images per second. Even at this relatively slow speed for a video camera, if you miss a few lines of pixels here or there, they will be overwritten anyway after just a little more than a quarter second. Unless the camera data is being used for something other than just the display, does it really matter if some of the data is lost? I think your real problem is synchronizing the input signal with the display so that every pixel, that gets received, is drawn at the correct spot on the display.
Re: Processing Time For PVOID Into Byte Array Conversion
Hi,
Thanks to support me. I'm beginner only.
I didn't use Queue before that. I learned the queue concept. By using the queue concept, feeling very difficulty.
I stored the data into the queue , now how can I retrieve the data from that?. I know the method pop(). But i want byte array format.
The sample code, i didn't try just i share my doubt before use the code.
Code:
#include <queue>
using namespace std;
queue<PVOID> mypvoidQ;
void CMyDlg::realData()
{
while (forever)
{
while (card_says_I_have_data)
AddDataFromCardToBackOfQueue();
if (!mypvoidQ.empty() )
{
RemoveDataFromFrontOfQueue(); //How is possible? how can i copy the data into byte array
DrawMyImageWithTheData();
}
}
}
void CMyDlg::AddDataFromCardToBackOfQueue()
{
pDaqData = GetData(m_nDevNo, &rc);
mypvoidQ.push(pDaqData);
mypvoidQ.back();
}
void CMyDlg::RemoveDataFromFrontOfQueue()
{
mypvoidQ.front();
mypvoidQ.pop();//How is possible? how can i copy the data into byte array
}
Really I felt very bad about me. before use codegure, no one here to help me. But now, all of then help me but my knowledge, I'm felt very difficult :(
I will study the concept and use and come back with you. Please give any sample code for me about queue.
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
queue<PVOID> mypvoidQ;
Er... No. queue needs a type of data that is to be stored. PVOID is not a suitable type as this is just a pointer to void.
I've knocked up some code to demonstrate the idea between a producer thread and a consumer thread for you. Its just win32 not MFC but you should get the idea about using a queue and sync between the threads. I don't have a camera to test so I don't know the timings for loss of data - but if this sort of program design still looses data from the camera then you really are in trouble as all the producer is doing is adding blocks of data to the queue.
Hope this helps
Code:
#include <iostream>
#include <queue>
#include <conio.h>
using namespace std;
#include <windows.h>
#include <process.h>
#define BUFSIZE 2048
//Structure with methods for camera data blocks
struct scam {
char* pdata;
scam()
{
pdata = (char*)new char[BUFSIZE];
memset(pdata, 0, BUFSIZE);
}
scam (const char* dat)
{
size_t ll = strlen(dat);
if (ll > BUFSIZE - 1) {
ll = BUFSIZE - 1;
}
pdata = (char*)new char[BUFSIZE];
memset(pdata, 0, BUFSIZE);
memcpy(pdata, dat, ll);
pdata[ll + 1] = 0;
}
scam(const char *dat, int no)
{
pdata = (char*)new char[BUFSIZE];
memcpy(pdata, dat, min(no, BUFSIZE));
}
~scam()
{
delete [] pdata;
}
scam(const scam& scm)
{
pdata = (char*)new char[BUFSIZE];
memcpy(pdata, scm.pdata, BUFSIZE);
}
scam& operator=(const scam& scm)
{
if (this != &scm) {
delete [] pdata;
pdata = (char*)new char[BUFSIZE];
memcpy(pdata, scm.pdata, BUFSIZE);
}
return *this;
}
void SetData(const char *dat, int no)
{
memset(pdata, 0, BUFSIZE);
memcpy(pdata, dat, min(no, BUFSIZE));
}
};
CRITICAL_SECTION ccrit;
HANDLE sync;
queue<scam> camdata;
//CONSUMER
unsigned __stdcall consume(void* pArguments)
{
scam dataOut;
while (true) {
dataOut = "";
WaitForSingleObject(sync, INFINITE);
EnterCriticalSection(&ccrit);
dataOut = camdata.front();
camdata.pop();
if (!camdata.empty()) {
SetEvent(sync);
}
LeaveCriticalSection(&ccrit);
//PROCESS CAMERA DATA BLOCK HERE
cout << dataOut.pdata << endl;
}
return 0;
}
int main()
{
unsigned thid;
sync = CreateEvent(NULL, FALSE, FALSE, "camev");
InitializeCriticalSection(&ccrit);
_beginthreadex(NULL, 0, &consume, NULL, 0, &thid);
//PRODUCER
scam dataIn1;
//test loop - would be 'while there is data from camera'
for (int i = 1; i < 50; i++) {
EnterCriticalSection(&ccrit);
//ADD DATA BLOCK HERE FROM CAMERA
dataIn1.SetData("dfdgrehgfdhghghgdhgdhghngfegfrzz", 32);
camdata.push(dataIn1);
SetEvent(sync);
LeaveCriticalSection(&ccrit);
}
_getch();
DeleteCriticalSection(&ccrit);
return (0);
}
Re: Processing Time For PVOID Into Byte Array Conversion
Quote:
Originally Posted by
saraswathisrinath
Hi,
Thanks to support me. I'm beginner only.
Well this is why in a real situation, it takes an experienced programmer to process "live" data correctly and efficiently. You can't really write a program with skills of a beginner to accomplish this.
First, get the idea of how a queue works by writing a simple program. Don't try and use your current program to learn how to use a queue.
For example, the queue::front() function returns a reference to the item that is first in the queue, but your sample program does nothing with the return value. All the queue::pop() does is remove the item from the queue -- it doesn't return to you a reference to the item. Second, why are you calling queue::back()? The push() automatically places the item in the back of the queue.
So even in a very simple situation, you're not using the queue correctly and you really are not sure what those member functions do.
Code:
#include <queue>
#include <cstring>
struct CardData
{
char buffer[2048];
};
typedef std::queue<CardData> CardDataQueue;
using namespace std;
int main()
{
CardData d1;
CardDataQueue qData;
strcpy(d1.buffer, "abc123");
qData.push( d1 );
strcpy(d1.buffer, "xyz");
qData.push( d1 );
//...
CardData& mData = qData.front(); // this should get the abc123
qData.pop(); // removes the top item
CardData& mData2 = qData.front(); // this should get the xyz
}
Something like this is what you should have first attempted, so you get the idea of how to use push(), pop(), and front.
Regards,
Paul McKenzie
Re: Processing Time For PVOID Into Byte Array Conversion
Thanks Mr.Paul. Really the above examples makes me to clear. If any doubt, i will come back.
Re: Processing Time For PVOID Into Byte Array Conversion
Hi,
After long time, I came back and also my problem solved :).
I used the Buffer to store the data (1000 line of data) & used the memcpy function to copy the data in to the byte pointer. After that, I used the file method to print the data.Now, I received the data without missing. Really happy.
Single time(1000 lines data) real time data received successfully. here after trying to get the real time data continuously.
Thanks to all Guru's.