Paged Search Results

[sigiswald.madou]

Hi everyone,

Recently I was trying to get paged search results working properly. As it turned out, I had to make 2 changes in the standard Spring LDAP source. The purpose of this post is to discuss the issue I faced and the way I fixed it. I must say I'm not an LDAP expert, so I hope to get some feedback about the solution. Perhaps there're some consequences I didn't notice. I'm using Spring LDAP 1.3.0.RELEASE.

This what the code looks like to retrieve all results.

private <T> List<T> pagedSearch(Name base, Filter filter,
    ParameterizedContextMapper<T> mapper) {
  List<T> entries = new ArrayList<T>();
  String encodedFilter = filter.encode();
  PagedResultsCookie cookie = null;
 
  do {
    PagedResult<T> pr = search(base, encodedFilter, mapper, cookie);
    entries.addAll(pr.getResultList());
    cookie = pr.getCookie();
  } while (cookie.getCookie() != null);
 
  return entries;
}
 
private <T> PagedResult<T> search(Name base, String filter,
    ParameterizedContextMapper<T> mapper, PagedResultsCookie cookie) {
  PagedResultsDirContextProcessor processor =
    new PagedResultsDirContextProcessor(PAGE_SIZE, cookie);
  List<T> entries =
    ldapTemplate.search(base, filter, SEARCH_CONTROLS, mapper, processor);
 
  return new PagedResult<T>(entries, processor.getCookie());
}

Unfortunately this didn't work. Every search resulted in
FATAL - No matching response control found for paged results - looking for 'class javax.naming.ldap.PagedResultsResponseControl
(and an additional NPE because the PagedResultsCookie cookie object was always null in the while loop.)

This FATAL error is defined in the postProcess method of AbstractFallbackRequestAndResponseControlDirContex tProcessor. As it turned out, ldapContext.getResponseControls() was always null. Then I figured out that in the preProcess method of AbstractRequestControlDirContextProcessor ldapContext.getRequestControls() initially always contained one request control: an instance of com.sun.jndi.ldap.ManageReferralControl. This in turn pointed me to the Context Request Controls chapter in Sun's JNDI tutorial. When I tried something similar for paged search, it worked fine. I noticed one major diffference: they basically use ldapContext.setResponseControls(new Control[] {new PagedResultsControl(...)}); and don't worry about about adding a new request control to the existing array.

That was the first fix I did:
AbstractRequestControlDirContextProcessor/preProcess

public void preProcess(DirContext ctx) throws NamingException {
  LdapContext ldapContext;
 
  if (ctx instanceof LdapContext) {
    ldapContext = (LdapContext) ctx;
  } else {
    throw new IllegalArgumentException(
      "Request Control operations require LDAPv3 - "
        + "Context must be of type LdapContext");
  }
 
  //traceControls(ldapContext.getRequestControls());
  ldapContext.setRequestControls(new Control[] {createRequestControl()});
  //traceControls(ldapContext.getRequestControls());
}
// this is the entire method

The 1st trace statement above prints 1 control: com.sun.jndi.ldap.ManageReferralControl.
The 2nd trace statement above prints 2 controls: javax.naming.ldap.PagedResultsControl + com.sun.jndi.ldap.ManageReferralControl

Then I faced another issue because I was using connection pooling (with org.springframework.ldap.pool.factory.MutablePooli ngContextSource). Apparently the contexts in the pool ended up in an "inconsistent" state and I received errors like
WARN - DirContext 'javax.naming.ldap.InitialLdapContext@e0cc23' failed validation with an exception.
javax.naming.OperationNotSupportedException: [LDAP: error code 53 - Base DSE differs from first page request]; remaining name ''
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.jav a:3114)
and
FATAL - No matching response control found for paged results - looking for 'class javax.naming.ldap.PagedResultsResponseControl
ERROR - [LDAP: error code 53 - Invalid query ref]; nested exception is javax.naming.OperationNotSupportedException: [LDAP: error code 53 - Invalid query ref]; remaining name '<some DN>'
org.springframework.ldap.OperationNotSupportedExce ption: [LDAP: error code 53 - Invalid query ref]; nested exception is javax.naming.OperationNotSupportedException: [LDAP: error code 53 - Invalid query ref]; remaining name '<some DN>'
at org.springframework.ldap.support.LdapUtils.convert LdapException(LdapUtils.java:199)
...
Caused by: javax.naming.OperationNotSupportedException: [LDAP: error code 53 - Invalid query ref]; remaining name '<some DN>'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.jav a:3114)

And this is the second fix I did:
AbstractFallbackRequestAndResponseControlDirContex tProcessor/postProcess

public void postProcess(DirContext ctx) throws NamingException {
  LdapContext ldapContext = (LdapContext) ctx;
  ldapContext.setRequestControls(null); // added this line
  Control[] responseControls = ldapContext.getResponseControls();
  ... etc. ...
}

The above changes allowed me to retrieve a large number of results using paged search and with connection pooling enabled. I didn't had to use a SingleContextSource or use LDAP transactions. Although I must say I never tested doing paged searches with different threads. So I can imagine LDAP transactions might be required after all. Or maybe it depends on the LDAP server that doesn't has the limitation of reusing the exact same connection between different search requests (we use a Siemens DirX LDAP server).

Kind regards,
Sigiswald

[jvrobert]

Thanks Sigiswald, I just ran into this same issue.

Unfortunately it doesn't look as if it's fixed yet. How do we go about getting someone to add this to 1.3.1 for a nightly build?

[sigiswald.madou]

Hi J,

Thanks for your reply. Perhaps I can submit a bug report? I didn't want to do this right away, because usually things are more complex than they look at first sight. There might be an impact when using other LDAP servers or when used together with a SortControlDirContextProcessor. I really don't know, but it could be the case. I was hoping to get some more feedback from the experts on this subject. But submitting a bug report anyway can only help; I'll try to do it tomorrow. FYI, right now I use the following extended class:

import javax.naming.NamingException;
import javax.naming.directory.DirContext;
import javax.naming.ldap.Control;
import javax.naming.ldap.LdapContext;
import org.springframework.ldap.control.PagedResultsCookie;
import org.springframework.ldap.control.PagedResultsDirContextProcessor;
 
public final class MyPagedResultsDirContextProcessor extends
  PagedResultsDirContextProcessor {
  public MyPagedResultsDirContextProcessor(int pageSize,
    PagedResultsCookie cookie) {
    super(pageSize, cookie);
  }
 
  @Override
  public void preProcess(DirContext ctx) throws NamingException {
    if (ctx instanceof LdapContext) {
      final LdapContext ldapContext = (LdapContext) ctx;
      ldapContext.setRequestControls(new Control[] { createRequestControl() });
    } else {
      throw new IllegalArgumentException("Request Control operations require "
        + "LDAPv3 - Context must be of type LdapContext");
    }
  }
 
  @Override
  public void postProcess(DirContext ctx) throws NamingException {
    final LdapContext ldapContext = (LdapContext) ctx;
    ldapContext.setRequestControls(null);
    super.postProcess(ldapContext);
  }
}

Kind regards,
/Sigiswald

[jvrobert]

Edit: It seems it does actually work out of the box in 1.3.1.CI-SNAPSHOT. I must have had some other problem because I tried it again with just the stock control and it works fine now. Maybe I didn't copy the new jar file over to my deployment area when I tested.

[sigiswald.madou]

Hi J,

Thanks for your reply. I just checked the latest source code and noted the issue is still there (just by looking at the code, not by actually testing my application). Strange your problem is fixed. But I'll actually test it on Monday. Promise!

BTW, did you use the "1.3.1.CI-SNAPSHOT" Maven dependency? Just to be sure we're looking at the exact same version...

Kind regards,
/Sigiswald

[sigiswald.madou]

Hi J,

I just tested with 1.3.1.CI-SNAPSHOT (actually spring-ldap-1.3.1.CI-20100220.201426-67-minimal.zip), but I got the exact same issue as before.

Kind regards,
/Sigiswald