HttpInvoker and streaming

[adepue]

I have a Swing client that needs to "upload" and "download" large files from the server. The files are larger than available memory, so they must be streamed. Currently, I've been using Spring's HttpInvoker (actually, a modified version of it) to transparently remote my service interfaces. Everything has been working well. Now, I know that HttpInvoker does not support streaming - but it sure would be nice if it could. So, I thought I would start this thread with the hope of brainstorming with others how this could be approached. Note, my goal is to keep with the Spring spirit and support streaming in a transparent and easy manner. Of course, I could just write a servlet or something, but then I wouldn't get my nifty transparent Spring goodness.
Ideally, it would be awesome to be able to do something like this:

public interface MyService
{
  public void storeContent(String id, InputStream content);
  public InputStream getContent(String id);
}

This is just an example, of course. Now, imagine if I could do something like this on the client side:

InputStream fileContent = new BufferedInputStream(new FileInputStream(file));
try {
  myService.storeContent(id, fileContent);
} finally {
  fileContent.close();
}

HttpInvoker would just magically somehow stream that InputStream across the wire to the server side where you could just read from that InputStream and do whatever you need with it.
Has anyone done anything like this? Think it would be difficult to implement? I'm thinking that HttpInvoker could be taught to notice when an InputStream is the last parameter to a method, and then handle that InputStream in some special manner. In the same way, it could notice when a method returns an InputStream.

- Andy

[adepue]

I've been digging into HttpInvoker source, and believe there may be a possible way to accomplish this. The main problem is that AbstractHttpInvokerRequestExecutor uses a ByteArrayOutputStream to buffer the entire invocation before transmitting. I may be able to implement my own HttoInvokerRequestExecutor (probably by extending CommonsHttpInvokerRequestExecutor) that checks to see if an InputStream is present in the invocation argument list. If not, it will perform a standard invocation using CommonsHttpInvokerRequestExecutor. If so, it will remove the InputStream from the argument list (probably by replacing it with some dummy marker object), setup the "PostMethod" to use CONTENT_LENGTH_CHUNKED, and then setup a special InputStream to represent the PostMethod body that is a composite of the serialized RemoteInvocation (which has been modified by removing the InputStream from the argument list) immediately followed by the InputStream parameter itself. In other words, when "read" is called on this special InputStream, it will first return bytes that compose the serialization of the modified RemoteInvocation, and once that is finished, it will then directly delegate reading to the InputStream parameter. On the server side, this special condition can be detected by first reading in the RemoteInvocation as is normally done, looking for the special marker argument/parameter in the RemoteInvocation, and, if present, pass in the remaining chunked Post body as the InputStream parameter to the service method... if that makes any sense. Anyway, I'm gonna try it and see what happens.

- Andy

[silly_me]

I too send large bytes back and forth using the httpInvoker protocol. What I do is have a serializable class that stores a byte array of the file contents.

But as a tangent, I am experiencing the httpInvoker protocol hanging after doing some large file transfers( ~4mb files). After transfering from a client to the server, all subsequent connections seem to hang. The connection doesn't seem to complete it's handshake?? Not sure but any debugging info would be helpful.

[adepue]

I currently have a "StreamSupportingHttpInvoker" that passes its unit tests - just working on doing some integration testing right now. Once I have something, I'll respond to this thread.

- Andy

[silly_me]

The hang that I was experiencing wasn't actually caused by the httpInvoker but due to exhhausting the connection pool. The datasource was blocking for connections.

httpInvoker was working like a champ..

[adepue]

I've finished my "StreamSupportingHttpInvoker" implementation and have posted the source along with unit tests to this Spring JIRA: http://opensource.atlassian.com/proj...rowse/SPR-1223

To quote from the issue:

I've extended Spring's HttpInvoker to provide support for InputStream parameters and return values for remote service methods. In other words, you can define a service method to take an InputStream as a parameter or return an InputStream as a return value (or both) and Spring will take care of ensuring the InputStream is streamed across the wire. Of course, the major reason for doing this is to handle large amounts of data without needing to load all the data into memory first, so care is taken to ensure that the InputStream is not fully buffered into memory.

See the JIRA issue for more detail.

- Andy

PS I'll post a message on the dev list in the hope that just maybe they will adopt this into Spring.

[brendandev]

Although I implemented this using the spring version of Commons.FileUpload and so some of it won't be relevant to the author's issue - I think the solution I used should work with adepue's StreamingHttpInvoker. I say this because what I implemented just maps a logical binary filey type from the client to a java.io.File object on the server side using streams. I also needed to store large files to the database since our app is clustered and any number of app machines will need to access the same file.

Anyway, my solution for streaming from the client to app server and from app server to the db (and vice versa) is documented at the end of http://forum.springframework.org/showthread.php?t=13663. In short, I used a CustomFileEditor to bind from the client to a java.io.File type on the server, and a custom usertype to map the java.io.File field to a binary value column in the DB.

I can provide either class upon request - they work well in our environment.

Brendan

[James Selvakumar]

Thanks adepue,

I've tried your code and it works like a charm for streaming the data back and forth.
I just had to refer "StreamSupportingHttpInvokerServiceExporter" instead of "HttpInvokerServiceExporter" and "StreamSupportingHttpInvokerProxyFactoryBean" instead of "HttpInvokerProxyFactoryBean" in my respective application context files.

Thanks again for sharing your knowledge.