fwrite ERRNO 22 with big (?) size

On linux the same call to fwrite works correctly (errno==0),
on windows i get ERRNO==22 if the "size" parameter is somewhat big (e.g. "103429"). :-/

see the code:

Code:

---

// first i open the file
FILE* request_file = fopen( pathname, "wb" );
if(request_file == NULL)
{
  // error
  return;
}

// second, try to write my buffer (already filled)
size_t written_bytes = 0;
if( parsed_request->body != NULL )
{
  errno=0;
  written_bytes = fwrite((parsed_request->body), 1, (size_t)(parsed_request->content_length), request_file);
  perror("fwrite");
}

// third, check fwrite return value
if( written_bytes < 0 )
{
        // write error
        return;
}
else if( written_bytes != (parsed_request->content_length) )
{
        // error - partial body received
        return;
}
// else fwrite succeed

---

NOTE:
if i replace "(size_t)(parsed_request->content_length)" with a small constant (e.g. "50"), fwrite succeed!

NOTE2:
I get the same behavior with "_write"...

P.S.
To give you more context details, i'm using the Mingw compiler.
I've defined the macro "_MT" to enable multi-threaded stdlib.
Maybe this has some side-effect on fwrite?
Since the call is not atomic with a big buffer, maybe fwrite fails to preserve thread-safeness?
I've tried to remove "_MT" in the makefile but fwrite still fails. :-(

The arguments 2 and 3 should be exchanged. This is a common mistake, and, usually, it does not matter, but maybe it matters here.

Instead of

Code:

---

written_bytes = fwrite((parsed_request->body), 1, (size_t)(parsed_request->content_length), request_file);

---

Write

Code:

---

written_bytes = fwrite((parsed_request->body), (size_t)(parsed_request->content_length), 1, request_file);

---

Check the value in parsed_request->content_length.
Is it the size of parsed_request->body?

N.B. Write your code between [code] and [/code]. It will be placed in a nice rectangle and spaces at the beginning of lines will be saved.

EDIT: Excuse-me, forget what I have said.

Quote:

---

Originally Posted by olivthill2

The arguments 2 and 3 should be exchanged. This is a common mistake, and, usually, it does not matter, but maybe it matters here.

Instead of

Code:

---

written_bytes = fwrite((parsed_request->body), 1, (size_t)(parsed_request->content_length), request_file);

---

Write

Code:

---

written_bytes = fwrite((parsed_request->body), (size_t)(parsed_request->content_length), 1, request_file);

---

---

I already tried that. Still get ERRNO==22. :-(
BTW, the correct way should be the first one. See http://www.opengroup.org/onlinepubs/...ns/fwrite.html):
I mean to write nitems=(parsed_request->content_length) bytes (size=1) from ptr=(parsed_request->body) to stream=request_file.

Quote:

---

Originally Posted by olivthill2

Check the value in parsed_request->content_length.
Is it the size of parsed_request->body?

---

Yes it is.

Don't call perror unless you detect an error.

gg

Quote:

---

Originally Posted by Codeplug

Don't call perror unless you detect an error.

gg

---

That call is for debug purpose only.
I'll remove that once i've fixed the issue.

What issue? You haven't shown any code that actually performs error checking.

http://www.opengroup.org/onlinepubs/...ons/errno.html

gg

Quote:

---

Originally Posted by Codeplug

What issue? You haven't shown any code that actually performs error checking.

gg

---

Ok, now i've added the checks.
I've omitted them since they did not seem relevant to me.

Quote:

---

Originally Posted by hi1

Ok, now i've added the checks.
I've omitted them since they did not seem relevant to me.

---

So where exactly in your code do you see that errno is 22?
Did you step through your code under debugger?
What is the value of written_bytes? content_length? And what is its type?
Please note that errno is a global variable, it could be set in some other place in your code.
For example, if your parsed_request->body is 0, you bypass the line

Code:

---

errno=0;

---

Quote:

---

Originally Posted by VladimirF

So where exactly in your code do you see that errno is 22?
Did you step through your code under debugger?

---

No, perror always output "Invalid Argument" (that is, errno 22).
I've also disabled multi-threading so there can't be any interference with other threads...

"content_length" is of "unsigned int" type. It was "103429" when fwrite failed.
BTW, with smaller values of "content_length" fwrite succeed.
I'm quite sure fwrite fails because of the "nitems" param, but i can't understand why.

How do you know that fwrite fails? Can we see the code?

gg

Quote:

---

Originally Posted by Codeplug

How do you know that fwrite fails? Can we see the code?

gg

---

Boy, it's just what i've posted:
If fwrite succeed errno remains 0 and perror outputs "No error",
otherwise errno!=0 and perror output something meaningful.

OK, last question: how sure are you that content_length matches the length of body?
If body is shorter, it will cause access violation during write() that will be caught inside that function and translated into EINVAL.

>> http://www.opengroup.org/onlinepubs/...ons/errno.html

Quote:

---

... The value of errno shall be defined only after a call to a function for which it is explicitly stated to be set and until it is changed by the next function call or if the application assigns it a value. The value of errno should only be examined when it is indicated to be valid by a function's return value. ...

---

>> http://www.opengroup.org/onlinepubs/...ns/fwrite.html

Quote:

---

RETURN VALUE

The fwrite() function shall return the number of elements successfully written, which may be less than nitems if a write error is encountered. If size or nitems is 0, fwrite() shall return 0 and the state of the stream remains unchanged. Otherwise, if a write error occurs, the error indicator for the stream shall be set, [CX] and errno shall be set to indicate the error.

---

So you can only check errno if frwite() returns a value less than the number of elements requested, or if ferror() returns non-zero.

gg

Quote:

---

Originally Posted by VladimirF

OK, last question: how sure are you that content_length matches the length of body?
If body is shorter, it will cause access violation during write() that will be caught inside that function and translated into EINVAL.

---

Yes i've thinked of it too, but the test case i've used for testing is build so that the content_length matches exactly the length of the body buffer.
So you can assume that this condition is always true.

Quote:

---

Originally Posted by Codeplug

So you can only check errno if frwite() returns a value less than the number of elements requested, or if ferror() returns non-zero.
gg

---

Ok, but the problem here is not if the errno value is valid or not.
The problem is that fwrite does not write anything (written_bytes==0) even if the arguments are correct.

Also note that (as i've already stated) that the very same code works correctly on Linux (written_bytes==103429),
so i'm quite sure that the program logic is correct.
I guess there must be "something" in the Windows context that makes fwrite behave differently.

I hope i've made myself clear. :-)

Quote:

---

Originally Posted by hi1

...the test case i've used for testing is build so that the content_length matches exactly the length of the body buffer.
So you can assume that this condition is always true.

---

I prefer to assume that it isn't.
Could you look at your "body" in the debugger? Can you scroll the Memory window all the way to body+103429? Does it look OK?
Bottom line is - I can assure you that fwrite() can handle 100K on Windows.
Could you package your test case and post it here?

Quote:

---

Originally Posted by VladimirF

I prefer to assume that it isn't.
Could you look at your "body" in the debugger? Can you scroll the Memory window all the way to body+103429? Does it look OK?
Bottom line is - I can assure you that fwrite() can handle 100K on Windows.
Could you package your test case and post it here?

---

Well, i think that if the whole thing works ok on linux (no segm.faults thrown and the file is written correctly), then the logic must be correct (body buffer must be of the correct size).
Dont you think?

Quote:

---

Originally Posted by hi1

Well, i think that if the whole thing works ok on linux (no segm.faults thrown and the file is written correctly), then the logic must be correct (body buffer must be of the correct size).
Dont you think?

---

No, I don't agree.
First, your Linux might be more forgiving to memory access violation. For example, it might add more padding.
Second, is it an isolated test? Who fills that "body"? How can you (or rather I) be sure that its content is the same between runs on Linux and Windows? Do you read this data from the file? Was that test file properly transferred cross platforms? Is it even the same size?
What does it cost you to write a simple self-contained test that writes 103429 bytes on Windows?
Here is what I did (and it worked OK):

Code:

---

void test()
{
        // first i open the file
        FILE* request_file = fopen( "R:\\test.dat", "wb" );
        if(request_file == NULL)
        {
                // error
                return;
        }

        // second, try to write my buffer (already filled)
        size_t written_bytes = 0;
        char* body = new char[103429];
        unsigned int content_length(103429);
        if( body != NULL )
        {
                errno=0;
                written_bytes = fwrite((body), 1, (size_t)(content_length), request_file);
                perror("fwrite");
        }

        // third, check fwrite return value
        if( written_bytes < 0 )
        {
                // write error
                return;
        }
        else if( written_bytes != (content_length) )
        {
                // error - partial body received
                return;
        }
        // else fwrite succeed
}

---

Quote:

---

Originally Posted by VladimirF

No, I don't agree.
First, your Linux might be more forgiving to memory access violation. For example, it might add more padding.

---

You are right about that.
But i expected that even if there was a mismatch between the buffer size and the "content_length" value the write operation should at least start and then throw an error, which it is not the case on Windows.
The fact that nothing is written on disk suggests me that fwrite returns during pre-conditions checking (this made me think about some OS influence...).
Besides, i don't think that fwrite can know the actual buffer size and make an internal check, since it receives just a pointer to the buffer.

Quote:

---

Originally Posted by VladimirF

No, I don't agree.
Second, is it an isolated test? Who fills that "body"? How can you (or rather I) be sure that its content is the same between runs on Linux and Windows? Do you read this data from the file? Was that test file properly transferred cross platforms? Is it even the same size?

---

The body is read from a network socket a chunk at a time and the whole buffer is reallocated multiple times with "realloc".

Quote:

---

Originally Posted by VladimirF

What does it cost you to write a simple self-contained test that writes 103429 bytes on Windows?

---

Thanks for your help.
Tomorrow i will run more tests (even with the debugger), and i'll see if i can package a test case.

Quote:

---

Originally Posted by hi1

...Besides, i don't think that fwrite can know the actual buffer size and make an internal check, since it receives just a pointer to the buffer.

---

Right. But it WILL cause access violation trying to copy the stuff that is not there, and will catch the exceprion.

Quote:

---

Originally Posted by VladimirF

Right. But it WILL cause access violation trying to copy the stuff that is not there, and will catch the exceprion.

---

You were right, i get access violation trying to do "puts(parsed_request->body+100000);".
So now i have to recheck all the program logic that fills the buffer...

Code:

---

#define _READ_CHUNK 512
static void _receive_request( void** buffer, int client_connected_socket )
{
        int buffer_current_size = 512;
        int buffer_current_offset = 0;
        int r = recv( client_connected_socket, (*buffer)+buffer_current_offset, _READ_CHUNK, 0 ); // read first chunk
        buffer_current_offset += _READ_CHUNK;
        while( r==_READ_CHUNK && r!=0 && r!=-1 )
        {
                // increase buffer size of a chunk and realloc
                        buffer_current_size += _READ_CHUNK;
                        *buffer = realloc(*buffer, buffer_current_size);
               
                // read next chunk
                r = recv( client_connected_socket, (*buffer)+buffer_current_offset, _READ_CHUNK, 0 );
                // increase offset for next read
                buffer_current_offset += _READ_CHUNK;
        }
}

static void _handle_http_request( int client_connected_socket )
{
        void* http_request_buf = calloc( 512, 1 );
        _receive_request( &http_request_buf, client_connected_socket );
        // ...
}

---

On Windows, the buffer fills up to 29184 bytes (much smaller than the data i send).

Quote:

---

Originally Posted by hi1

You were right, i get access violation...

---

Great, did I win something? :)

Quote:

---

Originally Posted by hi1

Code:

---

while( r==_READ_CHUNK && r!=0 && r!=-1 )

---

---

This condition is redundant: if r is 512, you know that it isn’t 0 or -1; if it isn’t 512 – the rest won’t even be tested.

Also, your allocation strategy is not very effective. You realloc (and copy) your buffer 200 times (for that 100K transfer), wasting time and fragmenting your heap. Few suggestions here:
- you can increase allocation unit to at least 4K;
- when you need more room – double existing space instead of simply adding another unit;
- and the winner is: can you send the total size required at the beginning of you transfer? Then you could first read 4 bytes, allocate entire buffer and be done with that.

Quote:

---

Originally Posted by hi1

On Windows, the buffer fills up to 29184 bytes (much smaller than the data i send).

---

Almost forgot: you need to add error checking. Your current situation is a BIG argument for that. At the moment, if you didn’t receive the requested 512 bytes, you simply assume that you are done.

So this turned out to be a problem with the socket API, and has nothing to do with fwrite.
I've tried to increase the receive socket buffer size with "setsockopt", but i still receive partial body.

Quote:

---

Originally Posted by VladimirF

Also, your allocation strategy is not very effective. You realloc (and copy) your buffer 200 times (for that 100K transfer), wasting time and fragmenting your heap.

---

I know that, but usually HTTP requests are small (only headers), so there is no need to realloc the buffer many times.
(btw, i've increased _CHUNCK_SIZE to 4096)

Quote:

---

Originally Posted by VladimirF

can you send the total size required at the beginning of you transfer? Then you could first read 4 bytes, allocate entire buffer and be done with that

---

I could, but i think this won't solve the problem.

Quote:

---

Originally Posted by hi1

So this turned out to be a problem with the socket API, and has nothing to do with fwrite.
I've tried to increase the socket buffer size with "setsockopt", but i still receive partial body.

---

That is one fix. The other fix is to make sure the program doesn't crash when something that the program relies on is not set up correctly.

There should be checks in the code for these possible scenarios you're describing.

Regards,

Paul McKenzie

Quote:

---

Originally Posted by VladimirF

Almost forgot: you need to add error checking. Your current situation is a BIG argument for that. At the moment, if you didn’t receive the requested 512 bytes, you simply assume that you are done.

---

You are right again. I can't rely on that test.
This is why i received partial body on Windows.
On Linux the flag "MSG_WAITALL" is available, but this does not seem a reliable solution.

So this is how i've changed the code now.
First, on Windows recv hanged so i've set a socket receive timeout:

Code:

---

                int socketRecvTimeout = 1000; // 1 sec
                setsockopt(server_socket_fd, SOL_SOCKET, SO_RCVTIMEO, &socketRecvTimeout, sizeof(int));

---

Second, i've modified the _receive_request function like this:

Code:

---

#define _READ_CHUNK 4096
static void _receive_request( void** buffer, int client_connected_socket )
{
        int buffer_current_size = 0;
        int buffer_current_offset = 0;
        int r = _READ_CHUNK_SIZE;
#if defined(WINDOWS) || defined(WIN32) || defined(_WIN32)
        while( r!=0 && r!=-1 )
#else
        while( r==_READ_CHUNK_SIZE )
#endif
        {
                buffer_current_size += _READ_CHUNK_SIZE;
                *buffer = realloc(*buffer, buffer_current_size);
                r = recv( client_connected_socket, (*buffer)+buffer_current_offset, _READ_CHUNK_SIZE, 0 );
                buffer_current_offset += r;
        }

---

Now everything seems to work nicely both on Windows and Linux.
Do you think this solution is reliable too?

If you have some better solutions for this "receive a chunk at time" thing, you're welcomed to post them!