I found a workaround by defining my own host bean. Take a look in the metadata settings how I'm able to get the host name and the context path which do not contain port numbers.

Code:
<bean name="host" class="java.net.InetAddress" factory-method="getLocalHost"/>

<bean id="metadataGeneratorFilter" class="org.springframework.security.saml.metadata.MetadataGeneratorFilter">
  ...
  <property name="entityBaseURL" value="https://#{host.canonicalHostName}#{servletContext.contextPath}"/>
  <property name="entityId" value="https://#{host.canonicalHostName}#{servletContext.contextPath}"/>
  ...