I am pleased you used AffirmativeBased and it sorted out your main issue.
Your proposed ConditionalRoleVoter (or whatever you would like to call it) would work fine. So long as the ConditionalRoleVoter can identify the config attributes it is responsible for servicing, there shouldn't be a problem. You'll probably need to prefix them to assist. eg:
A more sophisticated approach would be to somehow do it at the AccessDecisionManager level. The issue is you'd need to ensure ConfigAttributes remain independent, as each ConfigAttribute would potentially be serviced by its own AccessDecisionVoter (or RunAsManager or AfterInvocationManager for that matter). An example approache might be:
/foo.html=CONDITIONAL_ROLES(FOO && BAR || MANAGER),SOME_OTHER_ATTRIBUTE