Well, I understand your task.
So, let me make step back.
1. HttpRequestHandlingEndpointSupport already has ByteArrayHttpMessageConverter
2.
Code:
HttpMessageConverter#canWrite(Class<?> clazz, MediaType mediaType)
HttpMessageConverter#write(T t, MediaType contentType, HttpOutputMessage outputMessage)
These methods are used for determine appropriate Converter and put into HttpHeaders some content-type with writing payload into response outputStream.
3. For example. If you can determine content-type of your document in the fetchDocument just return Map<String, Object>
One key contains your byte[], another - is determined content-type.
4. And now it is simple:
HTML Code:
<chain input-channel="Channel">
<service-activator ref="Service" method="fetchDocument"/>
<header-enricher>
<header name="content-type" expression="payload[contentType]"/>
</header-enricher>
<transformer expression="payload[document]"/>
<chain>
Otherwise 'content-type' will be determined by request's 'accept-type' header.