WCF service with callback and background threads

i posted this same query in msdn forums but i thought i'd also ask here

Hi everyone!

We have an existing WCF service using BasicHttpBinding (for backward compatibility with .NET 2.0 clients) that performs a potentially long database search operation and returns an array of objects to the caller. We previously addressed the problem with a search taking very long by breaking it into "pages", meaning only a subset of the search is performed at a time. It worked OK but it is far from optimal because the database backend would perform the entire complex SELECT query and just return a subset of the results for every page request.

Now that we've confirmed our users are capable of .NET 3.0, we are planning on transitioning to full WCF. We plan to use the wsDualHttpBinding so we can continue to host our service using IIS, then utilise callbacks so that the client will only trigger a search operation once, and the server/service will periodically call the client back, handing over a few hundred or thousand records from the result set at a time. The service operations and callbacks will be one-way calls only.

I've come up with an initial implementation so that the service, upon receiving the search request, will spin a background thread to perform the long database operation. This same background thread will be iterating through the datareader and will periodically call a callback operation to give partial results to the client.

Do you think this is a good idea, and will this be safe? During my tests I've noticed that sometimes the db will issue a weird SqlException saying a "severe error occurred." And there are times the search request would seemingly not be received at all by the IIS 6.0 web server. Is this probably due to processes or threads being terminated/recycled by the web server even while it's processing?

I'd like to find out your thoughts on this before I proceed with this implementation so I can correct these issues. Thanks for your inputs

argh! sorry for the duplication (i have no idea why it's happening, and why after deleting it and posting the edited text back, it still keeps on duplicating the paragraphs!)

Gosh - it does seem that it has been a while since I answered a WCF/Web Services post, hasnt it ?

Personally,I dont think this is a good way. A couple of things come to mind such as heavy, security overheads and beng able to punch through client-side firewall/NAT issues. Are your end-clients really "addressable" in every sense of the word - so that "the server/service will periodically call the client back, handing over a few hundred or thousand records from the result set at a time" ?

I doubt so. Even if they are, I am concerned about the scalability of your idea.

Keep the original thread since you are implementing a one way. Make sure that you are thread safe and that you mark the service behavior instancemode as Single or (the instance mode as session and the ConcurrencyMode as multiple). Alternatively, you should use an async service implementation. The difficulty with this is: you must use Session mode with Concurrency = multiple, create an async handler element (if you go this route, I have an enhanced (from the WCF samples) async handler base class) and hand the OperationsContext to the async completion routines. There are some golden nuggets here as a subitem that may be of some use to you http://blogs.msdn.com/heinrichg/default.aspx

I dont think I have the time to follow through with you on this. Understand that your initial thoughts may not be a good idea in the first place and follow aome of my tips for clearer prescriptive guidance.

Good Luck !

I know exchange 2007 web services does something similiar, where a client can subscribe to be notified of certain events, like new mail in a particular folder, and exchange 2007 will send the information to the client.

Only had time to complete the subscription, but didn't managed to get any data returned. Too long to remember did exchange 2007 use wcf or normal web services.

thanks for the input! (did I wake you up? ha ha just joking)

well yes as far as addressability is concerned, the servers and the clients are all on an internal network, so no problem there with security and addressability (for the callback). but i am indeed concerned about the scalability.

the problems seemed to go away when I kept the callback to sending only a hundred or so records at a time. then again i'm not sure how well it will hold up when it gets many simultaneous search requests from users.

the only reason why I want to go this route is because of the amount of time it takes for the db server to execute a complex query. if i go with a "paged" (e.g. get records 1 to 3000, then 3001 to 6000, and so on) approach i don't need to do callbacks but the query will have to be executed for every page request, which makes response time terribly slow.