what I would do is just inject an authentication manager into the <password/> token granter
What is the password token granter Dave? Is it this:
<oauth:authorization-server client-details-service-ref="clientDetails" token-services-ref="tokenServices"
user-approval-handler-ref="userApprovalHandler">
...
<oauthassword/>
...
</oauth:authorization-server>
How do I inject a an authentication manager into there to authenticate the access token request?

Another question for you: The oauth2 spec requires the username and password to be passed in as part of the access token request for the password grant. If these credentials are not verified by the oauth server when providng the token then what is the point of it? I can just pass in any username and password?