It seems that Spring Security is caching the login credentials, and I'm not sure how to prevent it. Here are the steps I take:
1. login with user/pass
2. change password to pass1
4. login with user/pass (this shouldn't work but still does)
6. logout with user/pass1 (this should work and does)
Notice that now the user can login with the old and new password. If I redeploy my application, then it is corrected - the old password doesn't work and the new one does.
I am using standalone ApacheDS for LDAP and Tomcat for my application.