strange exception from WCF web serivces when response size is big

[George2]

Hello everyone,

I am hosting a WCF service in IIS using basicHttpBinding. The WCF web services queries backend SQL Server 2008 by using ADO.Net and return a DataTable to client side of WCF service.

I find when the returned DataTable is big, there will be exception said http connection is closed by IIS. Any ideas what is wrong and how to set bigger response size? Another idea is whether there is any hard limitation for the serialization size of an object (I thought maybe the DataTable instance is too big to serialize?)

The error information returned by IIS is,

BTW: I am using VSTS2008 + C# + .Net 3.5. Any ideas what is wrong?

Exception message:

    An error occurred while receiving the HTTP response to http://labmachine1/service.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
    
    {"The underlying connection was closed: An unexpected error occurred on a receive."}

Here is my whole source code for server side, for the server I host in web.config, no change for default values. Since I host in IIS, I am using basicHttpBinding.

    public class StudentManagement : IStudentManagement
    {
        public DataTable Poll(int Id)
        {
            return MakeParentTable();
        }

        private DataTable MakeParentTable()
        {
            // Create a new DataTable.
            System.Data.DataTable table = new DataTable("ParentTable");
            // Declare variables for DataColumn and DataRow objects.
            DataColumn column;
            DataRow row;

            // Create new DataColumn, set DataType, 
            // ColumnName and add to DataTable.    
            column = new DataColumn();
            column.DataType = System.Type.GetType("System.Int32");
            column.ColumnName = "id";
            column.ReadOnly = true;
            column.Unique = true;
            // Add the Column to the DataColumnCollection.
            table.Columns.Add(column);

            // Create second column.
            column = new DataColumn();
            column.DataType = System.Type.GetType("System.String");
            column.ColumnName = "ParentItem";
            column.AutoIncrement = false;
            column.Caption = "ParentItem";
            column.ReadOnly = false;
            column.Unique = false;
            // Add the column to the table.
            table.Columns.Add(column);

            // Make the ID column the primary key column.
            DataColumn[] PrimaryKeyColumns = new DataColumn[1];
            PrimaryKeyColumns[0] = table.Columns["id"];
            table.PrimaryKey = PrimaryKeyColumns;

            // Create three new DataRow objects and add 
            // them to the DataTable
            for (int i = 0; i <= 1000000; i++)
            {
                row = table.NewRow();
                row["id"] = i;
                row["ParentItem"] = "ParentItem " + i;
                table.Rows.Add(row);
            }

            return table;
        }
    }

Client side code:

        static void Main(string[] args)
        {
            StudentIdentifier identifier = new StudentIdentifier();
            identifier.Id = 100;
            StudentManagementClient client = new StudentManagementClient();

            DataTable student = client.Poll(identifier);

            Console.WriteLine(student.Rows.Count);
        }

Client side web.config:

                <binding name="BasicHttpBinding_IStudentManagement" closeTimeout="00:01:00"
                    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
                    allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
                    maxBufferSize="1000000000" maxBufferPoolSize="1000000000" maxReceivedMessageSize="1000000000"
                    messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
                    useDefaultWebProxy="true">
                    <readerQuotas maxDepth="32" maxStringContentLength="1000000000" maxArrayLength="1000000000"
                        maxBytesPerRead="4096" maxNameTableCharCount="16384" />

thanks in advance,
George

[monalin]

Could you post a Stack Trace or InnerException? Would be a little easier to figure it out if you could...

Also, How large of a table are we talking?

[George2]

Thanks monalin,

From my code, the table size is, 100,001 rows of the table. The table is of two columns. You can see the schema of the table (DataTable) in my posted server side code.

Here is the inner exception and stack trace, any ideas what is wrong? If I change the # of rows to small value, like 10,001, it works.

{"The underlying connection was closed: An unexpected error occurred on a receive."}

"\r\nServer stack trace: \r\n at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)\r\n at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)\r\n at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)\r\n at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)\r\n at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)\r\n at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)\r\n at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)\r\n at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)\r\n\r\nException rethrown at [0]: \r\n at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)\r\n at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)\r\n at TestServiceClient1.ServiceReference1.IStudentManagement.Poll(StudentIdentifier Id)\r\n at TestServiceClient1.ServiceReference1.StudentManagementClient.Poll(StudentIdentifier Id) in C:\\Users\\Documents\\Visual Studio 2008\\Projects\\WCFDemoClassv1\\TestServiceClient1\\Service References\\ServiceReference1\\Reference.cs:line 96\r\n at TestServiceClient1.Program.Main(String[] args) in C:\\users\\Documents\\Visual Studio 2008\\Projects\\WCFDemoClassv1\\TestServiceClient1\\Program.cs:line 18\r\n at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)\r\n at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)\r\n at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()\r\n at System.Threading.ThreadHelper.ThreadStart_Context(Object state)\r\n at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n at System.Threading.ThreadHelper.ThreadStart()"

Could you post a Stack Trace or InnerException? Would be a little easier to figure it out if you could...

Also, How large of a table are we talking?

[monalin]

Its very possible you're server is timing out. I ran your table generation code and it took 10827ms which is 10.827 seconds.

in the web config you have

receiveTimeout="00:10:00"

Try changing this or changing the number of rows to something like 900,000 instead of 1,000,000 if its timing out it will most likely work with 900,000 since it will complete in under 10 seconds. Unless of course your computer is a little slower who knows... but it looks like it should just be a config thing.

[George2]

Thanks monalin,

1. Should I configure timeout at client side or server side?

2. If the root cause is timeout, why in the exception message it is not indicated?

Its very possible you're server is timing out. I ran your table generation code and it took 10827ms which is 10.827 seconds.

in the web config you have

receiveTimeout="00:10:00"

Try changing this or changing the number of rows to something like 900,000 instead of 1,000,000 if its timing out it will most likely work with 900,000 since it will complete in under 10 seconds. Unless of course your computer is a little slower who knows... but it looks like it should just be a config thing.

[monalin]

Thanks monalin,

1. Should I configure timeout at client side or server side?

2. If the root cause is timeout, why in the exception message it is not indicated?

The errors I get with WFC or .NET Remoting are annoying. I can't think of a time where it said it timed out its always giving you the same message. "Underlying Connection was closed..." Very useful if you ask me.

Set the timeout on the client side. As thats where the GUI will be and the length of time will affect end-user experience before it will affect the performance of your server.

[George2]

Thanks monalin!

1. Why I only need to set at client side? I think from client side, the receiveTimeout means sendTimeout at server side? So, I think we should set both?

2. Why I only need to care about receiveTimeout, how about the other 3 timeout parameters -- closeTimeout, openTimeout and sendTimeout? Why in my scenario the other 3 does not impact?

The errors I get with WFC or .NET Remoting are annoying. I can't think of a time where it said it timed out its always giving you the same message. "Underlying Connection was closed..." Very useful if you ask me.

Set the timeout on the client side. As thats where the GUI will be and the length of time will affect end-user experience before it will affect the performance of your server.

[monalin]

Think of each request as 4 different parts.

1. Open Connection (openTimeout)
2. Send Request (sendTimeout)
3. Receive Data (receiveTimeout)
4. Close Connection (closeTimeout)

Any of these steps can timeout. But you may not want to wait 10 seconds for it to open the connection. If it takes 10 seconds to open the connection you have a problem hah. However, receiving data could potentially take awhile thats why they different timers.

I could be wrong, so you may want to test it. But the sendTimeout on the server would only be the time that it takes to actually transfer the data. It doesn't care how long it takes to create the data, it just cares how long it takes to send. Just like the sendTimeout from the client doesn't timeout if it takes your application 5 minutes to prepare data before it sends a request.

However, the client is waiting for a response the whole time the server is processing the data. Thats why the timeout on the client side would be the one causing the issue (if it is a timeout issue)

[George2]

Thanks monalin,

1. I have tried when I set both send/receive timeout at client side and server side, it will work. If I only set client side, it will not work. Any ideas?

2. I have tried to increase row number to 10 times, i.e. 10,000,000 (10M rows), then no matter how big value I set for timeout, it always fail. Could you reproduce my issue? Any ideas for solutions?

Think of each request as 4 different parts.

1. Open Connection (openTimeout)
2. Send Request (sendTimeout)
3. Receive Data (receiveTimeout)
4. Close Connection (closeTimeout)

Any of these steps can timeout. But you may not want to wait 10 seconds for it to open the connection. If it takes 10 seconds to open the connection you have a problem hah. However, receiving data could potentially take awhile thats why they different timers.

I could be wrong, so you may want to test it. But the sendTimeout on the server would only be the time that it takes to actually transfer the data. It doesn't care how long it takes to create the data, it just cares how long it takes to send. Just like the sendTimeout from the client doesn't timeout if it takes your application 5 minutes to prepare data before it sends a request.

However, the client is waiting for a response the whole time the server is processing the data. Thats why the timeout on the client side would be the one causing the issue (if it is a timeout issue)

[monalin]

Thanks monalin,

1. I have tried when I set both send/receive timeout at client side and server side, it will work. If I only set client side, it will not work. Any ideas?

I guess that means you have to set both. Nothing wrong with that.

2. I have tried to increase row number to 10 times, i.e. 10,000,000 (10M rows), then no matter how big value I set for timeout, it always fail. Could you reproduce my issue? Any ideas for solutions?

If you're really planning on processing something that will take that long. I don't think you should use a persistent connection like you're doing. I wouldn't keep a connection open that long. Not sure if this is possible with WFC, i would have to do some research.

[George2]

Thanks monalin!

1. Let me know if you find any hard limitation by WCF. :-)
2. I have some confusion about connection lifetime. Suppose I create a client proxy instance, then using this specific instance to call various methods exposed by WCF service at server side. Then, each time I make a method call will make a new connection to be established? Or there will be a constant connection between client and server (life time of connection is from creation of client proxy instance, to disposing of client proxy instance)?

I guess that means you have to set both. Nothing wrong with that.



If you're really planning on processing something that will take that long. I don't think you should use a persistent connection like you're doing. I wouldn't keep a connection open that long. Not sure if this is possible with WFC, i would have to do some research.

[monalin]

Don't have a lot of experience with WCF to know all of its limitations. My assumption is that they are few and far between. When i say I don't think it can do it, that most likely means I don't know how lol.

To answer your other question. Check this link out. Scroll down to the part where it talks about Singleton Sevices.

http://msdn.microsoft.com/en-us/magazine/cc163590.aspx

[Arjay]

Most likely the problem is because the default message size in WPF is set pretty small.

Unfortunately, by default this error doesn't get passed back to the client.

In fairness to WPF, these defaults are set up to reduce the security footprint. Out of the box, you wouldn't want to open yourself up to attack by reporting all errors and allowing the sending of huge messages.

At any rate, on the server side, turn on passing client exceptions:

<serviceBehaviors>
  <behaviorname="MyServiceBehavior">
    <serviceMetadatahttpGetEnabled="true" />
    <serviceDebugincludeExceptionDetailInFaults="true" />
  </behavior>
</serviceBehaviors>

This should allow you to see that the default message size parameters are too small.

To increase the message size on the client..

<wsHttpBinding>
  <bindingname="IMyServiceBinding"closeTimeout="00:01:00"
openTimeout="00:01:00"receiveTimeout="00:10:00"sendTimeout="00:01:00"
bypassProxyOnLocal="false"transactionFlow="false"hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="55150000000"maxReceivedMessageSize="2147483647"
messageEncoding="Text"textEncoding="utf-8"useDefaultWebProxy="true"
allowCookies="false">
    <readerQuotasmaxDepth="150000000"maxStringContentLength="150000000"maxArrayLength="150000000"
maxBytesPerRead="150000000"maxNameTableCharCount="150000000" />
    <reliableSessionordered="true"inactivityTimeout="00:10:00"
enabled="false" />
    <securitymode="Message">
      <transportclientCredentialType="Windows"proxyCredentialType="None"
realm="" />
      <messageclientCredentialType="Windows"negotiateServiceCredential="true"
algorithmSuite="Default"establishSecurityContext="true" />
    </security>
  </binding>
</wsHttpBinding>

Note: the size values are cranked up in this example. You may want to adjust them down to more reasonable values.

Search google for "maxReceivedMessageSize large datasets" for more info.

[George2]

Hi monalin,

Why do you think Singleton relates to my issue? I read it and did not find any relationship. Any more description please? :-)

Don't have a lot of experience with WCF to know all of its limitations. My assumption is that they are few and far between. When i say I don't think it can do it, that most likely means I don't know how lol.

To answer your other question. Check this link out. Scroll down to the part where it talks about Singleton Sevices.

http://msdn.microsoft.com/en-us/magazine/cc163590.aspx

[George2]

Thanks Arjay, I have already set the value of serviceDebugincludeExceptionDetailInFaults to true, but still not detailed error message from server side. Any ideas? Could you reproduce my problem?

Most likely the problem is because the default message size in WPF is set pretty small.

Unfortunately, by default this error doesn't get passed back to the client.

In fairness to WPF, these defaults are set up to reduce the security footprint. Out of the box, you wouldn't want to open yourself up to attack by reporting all errors and allowing the sending of huge messages.

At any rate, on the server side, turn on passing client exceptions:

<serviceBehaviors>
  <behaviorname="MyServiceBehavior">
    <serviceMetadatahttpGetEnabled="true" />
    <serviceDebugincludeExceptionDetailInFaults="true" />
  </behavior>
</serviceBehaviors>

This should allow you to see that the default message size parameters are too small.

To increase the message size on the client..

<wsHttpBinding>
  <bindingname="IMyServiceBinding"closeTimeout="00:01:00"
openTimeout="00:01:00"receiveTimeout="00:10:00"sendTimeout="00:01:00"
bypassProxyOnLocal="false"transactionFlow="false"hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="55150000000"maxReceivedMessageSize="2147483647"
messageEncoding="Text"textEncoding="utf-8"useDefaultWebProxy="true"
allowCookies="false">
    <readerQuotasmaxDepth="150000000"maxStringContentLength="150000000"maxArrayLength="150000000"
maxBytesPerRead="150000000"maxNameTableCharCount="150000000" />
    <reliableSessionordered="true"inactivityTimeout="00:10:00"
enabled="false" />
    <securitymode="Message">
      <transportclientCredentialType="Windows"proxyCredentialType="None"
realm="" />
      <messageclientCredentialType="Windows"negotiateServiceCredential="true"
algorithmSuite="Default"establishSecurityContext="true" />
    </security>
  </binding>
</wsHttpBinding>

Note: the size values are cranked up in this example. You may want to adjust them down to more reasonable values.

Search google for "maxReceivedMessageSize large datasets" for more info.