[RESOLVED] WCF large collections

A while ago I posted a threadw about WCF client configuration (http://www.codeguru.com/forum/showthread.php?t=464674)

At that moment I needed to retrieve a list of 480 object from the WCF host. After editing the client's app.config, everything worked fine.

Now I have a different list (of the same type) which has 1800 object in it. Now I get this CommunicationException everytime again. When I reduce this list to 500 objects, it works fine, so somehow, the list is to big.

I changed the binding to very large numbers, but still I can't retrieve the list.
The binding look like next:

Code:

---

                <binding name="UserTcpBinding" closeTimeout="00:01:00" openTimeout="00:01:00"
                    receiveTimeout="00:10:00" sendTimeout="00:01:00" transactionFlow="false"
                    transferMode="Buffered" transactionProtocol="OleTransactions"
                    hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                    maxBufferPoolSize="50000000" maxBufferSize="50000000" maxConnections="10"
                    maxReceivedMessageSize="50000000">
                    <readerQuotas maxDepth="50000000" maxStringContentLength="50000000" maxArrayLength="50000000"
                        maxBytesPerRead="50000000" maxNameTableCharCount="50000000" />
                    <reliableSession ordered="true" inactivityTimeout="00:10:00"
                        enabled="false" />
                    <security mode="Transport">
                        <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                        <message clientCredentialType="Windows" />
                    </security>
                </binding>

---

Is there anyway to resolve this by modifying the app.config. Or do I need to return the list by a callback function in several times? (so first send back the first 500 object of the list, then the 2nd 500 and so on)

What kind of CommunicationException? What's its inner exception? Message?

- petter

To help debug, include exception details in the service.

Code:

---

<behavior name="myBehavior">
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
</behavior>

---

Arjay: I have includeExceptionDetailInFaults set to true

Wildfrog: it's a CommunicationObjectFaultedException
- InnerException: null
- Message: "The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
- Source: mscorlib
- Static members: Non-public Members: _COMPlusExceptionCode = -532459699

Try searching google for "COMPlusExceptionCode 0xE0434F4D" to see if that would give you any ideas.

I googled more then half an hour now, but I can not find a good answer.

The only thing I think it has to with it, is serialization/deserialization of the collection. But I don't know if that is correct.
And if it is correct, what is causing this and how to solve it?

I'm not sure if you did this already, but you need to increase the default on the service (as well as the client).

Quote:

---

<bindings>
<wsHttpBinding>
<binding name="wsHttpBindingConfig" maxReceivedMessageSize ="50000000" maxBufferPoolSize="50000000" >
<readerQuotas maxArrayLength="50000000" />
</binding>
</wsHttpBinding>
</bindings>

---

First I did this only at the client, but after increasing the default on the service I still get the same error;

Client app.config

Code:

---

                <binding name="UserTcpBinding" closeTimeout="00:01:00" openTimeout="00:01:00"
                    receiveTimeout="00:10:00" sendTimeout="00:01:00" transactionFlow="false"
                    transferMode="Buffered" transactionProtocol="OleTransactions"
                    hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                    maxBufferPoolSize="50000000" maxBufferSize="50000000" maxConnections="10"
                    maxReceivedMessageSize="50000000">
                    <readerQuotas maxDepth="32" maxStringContentLength="50000000"
                        maxArrayLength="50000000" maxBytesPerRead="50000000" maxNameTableCharCount="50000000" />
                    <reliableSession ordered="true" inactivityTimeout="00:10:00"
                        enabled="false" />
                    <security mode="Transport">
                        <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                        <message clientCredentialType="Windows" />
                    </security>
                </binding>

            <endpoint address="net.tcp://tcwork11:10001/UserService/service"
                binding="netTcpBinding" bindingConfiguration="UserTcpBinding"
                contract="srUserService.UserService" name="UserTcpBinding">
                <identity>
                    <userPrincipalName value="tcwork11\Danny" />
                </identity>
            </endpoint>

---

Service app.config

Code:

---

  <system.serviceModel>
    <bindings>
      <netTcpBinding>
        <binding name="MaxBinding" maxReceivedMessageSize="50000000" maxBufferPoolSize="50000000" maxBufferSize="50000000">
          <readerQuotas maxArrayLength="50000000" maxStringContentLength="50000000" maxBytesPerRead="50000000" maxNameTableCharCount="50000000"/>
        </binding>
      </netTcpBinding>
    </bindings>
    <behaviors>
      //some code here
    </behaviors>
    <services>
      <service behaviorConfiguration="UserBehavior" name="YieldManagerPlus.Service.Services.sUser.UserService">
        <endpoint address="service" binding="netTcpBinding" bindingConfiguration="MaxBinding" contract="YieldManagerPlus.Service.Services.sUser.IUserService" name="UserTcpBinding" />

    //some code here

---

Shouldn't behaviorConfiguration be set to MaxBinding instead of UserBehavior?

No, that gives a ConfigurationErrorsException: There is no service behavior named 'MaxBinding'. (because it's not an behavior but a binding)

Don't forget, that if a communication exception has occured, the channel remain in faulted state and you cannot use it any more, you have to create a new channel.

Yes I know that, but I still don't know why this exception is thrown.

Finally resolved.

Some gave me the advice of turning on WCF Tracing (http://msdn.microsoft.com/en-us/library/ms732023.aspx).

I traced the client, but nothing special to find. Just the same exception as already was thrown.

Also traced the service. When the service is running, it does not throw an execption, but the trace did log an exception:
There was an error while trying to serialize parameter YieldManagerPlus.Service.Services.sUser:GetAllAPIentityResult. The InnerException message was 'Maximum number of items that can be serialized or deserialized in an object graph is '65536'. Change the object graph or increase the MaxItemsInObjectGraph quota. '. Please see InnerException for more details.

I added into client app.config:

Code:

---

<behaviors>
  <endpointBehaviors>
    <behavior name="MaxBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
    </behavior>
  </endpointBehaviors>
</behaviors>

---

Now it works fine!


Only thing that I don't understand, why does the trace log that exception, but the exception is never thrown at the service application?

I heate repeating myself...but...

Making MAJOR increases to the configuration values for WCF communications is very likely to create subtle problems at some point in the future.

Services should (almost always) be configuraed to operate on messages that are within (or at least close) to these limits.

In many cases, the best pattern is to have the server perform multiple transfers using a "reverse channel". This provides:

1) Lower memory resource utilization for both the server AND the client.

2) Faster processing, as each "chunk" can be processed while the next one is still "in-flight".

Of the 20+ enterprise class WCF solutions I have deployed, I have followed the practice of using config files with the limits set to (approx) 80% of their default values. This allows me to implement the architecture so that the production systems can run with the default values.

Can you explain the term "reverse channel", please? Everything I googled out refered to telecommunications, but it says nothing about software usage the term.

I think TheCPUWizard means callback functions.

So the method (on host) 'public List<MyObject> GetMyBigCollection()' changes into a void.

Next, the big collection is returned to client using a callback function.

Quote:

---

Originally Posted by boudino

Can you explain the term "reverse channel", please? Everything I googled out refered to telecommunications, but it says nothing about software usage the term.

---

Quote:

---

Originally Posted by dannystommen

I think TheCPUWizard means callback functions. So the method (on host) 'public List<MyObject> GetMyBigCollection()' changes into a void. Next, the big collection is returned to client using a callback function.

---

WCF Callback messages/functions are a type of reverse channel (which is simply where the request/response flows in the opposite direction from a normal/forward channel). Depending on the transport selected for WCF, this may be a viable option.

The important part is that there is NEVER a "big collection" or a "big message".

Consider a object collection where each instance takes 2K to represent via serialized content. Also that up to 500 of these may exist in a collection (being moved over the wire). This amounts to 1M of data [A BIG Message].

Changing the seuqence to internally use messages with a maximum of 20 objects per message would limit the message size to 40K. This means that:

1) Client requests collection [whichwill have 500 items] (request may contain parameters)
2) Server sends 13 messages \ on reverse/callback channel.


Now lets look at performance: Assume the server takes 1mS to read an object from the database and serialize it, the network takes 2mS to transmit an object, and the client takes 1mS to deserialize and process. This are artifical numbers for easy illustration.

Done as a single message the time is 500mS+1000mS+500mS = 2 SECONDS. Done using the method outlined above, the time is 500mS + 40mS + 20mS = 560mS. The "single message: takes 3.57 times as long!!!

Now lets look at memory.

a) There never needs to be more than 20 instances in server memory [96% savings]

b) There is NEVER a message string big enough to utilize the LOH on the server [risk 100% eliminated]

c) There may be no need for more than 20 instances in client memory, if the objects can be processed "on the fly" [potential 96% savings]

With just a little work, this entire pattern can be implemented using generics, and then re-used in all cases where the developer has design control over both the client and the server.

But doesn't it mean to have statefull server and to implement some kind of paging?

Quote:

---

Originally Posted by boudino

But doesn't it mean to have statefull server and to implement some kind of paging?

---

Not at all!!!! EVERYTHING occurs within the context of the message..

Using Psuedo code... you would originally have something like this on the server:

Code:

---

WCFResponse Process(WCFRequest request)
{
  Collection col = GetDataFromDB();
  WCFResponse retVal = ....
  retVal.Data = col;
  return retVal;
}

---

This becomes:

Code:

---

WCFResponse Process(WCFRequest request)
{
  WCFLink link = new .... // Create the reverse/callback...
  Collection col;
  do
      Collection col = GetNextGroupFromDB();
      WCFREquest request = new ...
      request.Data = col;
      link.Process(request);
  while (col.Count > 0);
    WCFResponse retVal = ....
  return retVal;
}

---

The same type of implementation is done on the client (ie All of the callbacks occur and are acknowledged within the processing period of the main message.

An additional benefit that I did not list: The client can cleanly cancel the transmission (and all associated processing) at any boundary...

Thanks.

Thanks for explanation CPUWizard. It will consider changing it into this reverse channel technique.

But why does the .Net framework provides this functionality to send large object, if this is not a nice way to do this (memory risk, LOH).

Quote:

---

Originally Posted by dannystommen

Thanks for explanation CPUWizard. It will consider changing it into this reverse channel technique.

But why does the .Net framework provides this functionality to send large object, if this is not a nice way to do this (memory risk, LOH).

---

Why does it provide the ability to multiply 1000 by 1000 by writing an addition loop???

One additional question: Does it mean to communicate on duplex channel created using DuplexChannelFactory?

Quote:

---

Originally Posted by boudino

One additional question: Does it mean to communicate on duplex channel created using DuplexChannelFactory?

---

Once again, there are MANY possible implementations, one of them would be to use the DuplexChannelFactory, but that is NOT a requirement.