PagedResultsRequestControl, getting [LDAP: error code 4 - Sizelimit Ex

[djeanprost]

Hello,

I'm trying to implement a ldap search to get *all* groups I have in my ldap server. As the number of group is bigger than my server search limit (500), I'm trying to use PagedResultsRequestControl.
Here is my code :

 List<String> groups = new ArrayList<String>();
        final int PAGE_SIZE = 100;
        PagedResultsRequestControl control = new PagedResultsRequestControl(PAGE_SIZE, null);

        do {
            List<String> groupsTemp = this.ldapTemplate.search(this.baseGroupSearch, this.groupSearchFilter, this.searchControls,
                    new ParameterizedContextMapper<String>() {

                        public String mapFromContext(Object arg0) {
                            DirContextAdapter dirContextAdapter = (DirContextAdapter) arg0;
                            return dirContextAdapter.getStringAttribute(CheckHomonymy.this.groupIdAttribute);
                        }
                    }, control);
            groups.addAll(groupsTemp);
            control = new PagedResultsRequestControl(PAGE_SIZE, control.getCookie());
        } while (control.getCookie().getCookie() != null);

Please note that I'm using a page size smaller than my server limit.

I'm getting this exception when I try to get the 5th page of object (5*100 objects fetched).

I don't understand why I have this as I'm using pages of 100 objects. I thought it would reset the limit after each call of ldapTemplate.search(...).

Where am I wrong ?
thank you.
dom

org.springframework.ldap.LimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; nested exception is javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'o=internet'
	at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:140)
	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:295)
	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:234)
	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:583)
	at org.springframework.ldap.core.simple.SimpleLdapTemplate.search(SimpleLdapTemplate.java:101)
	at com.dexia.sofaxis.checkhomonymy.CheckHomonymy.getGroups(CheckHomonymy.java:154)
	at com.dexia.sofaxis.checkhomonymy.CheckHomonymy.checkGroups(CheckHomonymy.java:110)
	at com.dexia.sofaxis.checkhomonymy.CheckHomonymy.run(CheckHomonymy.java:99)
	at com.dexia.sofaxis.common.batch.engine.Launcher.run(Launcher.java:42)
	at com.dexia.sofaxis.common.batch.engine.UnitTestLauncher.run(UnitTestLauncher.java:53)
	at com.dexia.sofaxis.checkhomonymy.CheckHomonymyTest.testRun(CheckHomonymyTest.java:22)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:38)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'o=internet'
	at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3037)
	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2931)
	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2737)
	at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1808)
	at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1731)
	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:321)
	at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:248)
	at org.springframework.ldap.core.LdapTemplate$4.executeSearch(LdapTemplate.java:230)
	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:272)
	... 31 more

[ulsa]

I believe this is the expected behavior. The Paged Results Control is described in RFC 2696. In chapter 6, Security Considerations, it says:

Servers implementations may enforce an overriding sizelimit, to
prevent the retrieval of large portions of a publically-accessible
directory.

[djeanprost]

I don't really understand this behaviour. I understand that there may be a server side limit to avoid fetching the whole ldap data during one ldap search.

As far as I understood how the PagedResultsRequestControl works, I thought that as I'm using a brand new ldapTemplate.search each time I iterate in my while statement, it would use another ldap search, meaning that it's a new search in the ldap server side point of view, and so the server side "counter" would be reset, and that I wouldn't meet the SizeLimitException.
Can you explain me how my example is different from making 5 times a ldapTemplate.search with each time a size limit of 100 ?

Can someone tell me how to retrieve the full group list in my ldap if the PagedResultsRequestControl strategy doesn't work ?
regards,
dom

[ulsa]

I don't really understand this behaviour. I understand that there may be a server side limit to avoid fetching the whole ldap data during one ldap search.

Apparently, the RFC authors believe that if there is a server size limit of 500, then there should be no way of retrieving more than 500, regardless of any clever use of paging. This is, as I indicated, more a security issue than a server resources issue. You must ask the LDAP experts why, but I'm sure there are use cases that will make sense.

Can you explain me how my example is different from making 5 times a ldapTemplate.search with each time a size limit of 100 ?

I guess you could say that a paged search is actually a full search with a limited result set, while manual searches are limited searches with full result sets.

Can someone tell me how to retrieve the full group list in my ldap if the PagedResultsRequestControl strategy doesn't work ?
regards,
dom

The Server Side Sorting Control is described in RFC 2891. In Chapter 4, Security Considerations, it says:

Implementors and administrators should be aware that allowing sorting
of results could enable the retrieval of a large number of records
from a given directory service, regardless of administrative limits
set on the maximum number of records to return.

A client that desired to pull all records out of a directory service
could use a combination of sorting and updating of search filters to
retrieve all records in a database in small result sets, thus
circumventing administrative limits.

So, perhaps using sorting combined with manual searches will work for you. Unless increasing the server side size limit is an option, of course.

[djeanprost]

Thank you for helping me.
I have to admit I don't really understand how my problem of size limit is related to the rfc you mention, which deals, as far as I know, to server side sorting. What I'm trying to do is getting the whole data, sorted or not, I don't care. There must be something I'm missing here.

[ulsa]

Let's assume there is a server size limit of say 500, and you want to retrieve all say 1200 of something. You can't simply say get me all "(cn=*)" because you will get a size limit exceeded exception. So you want to divide the search somehow. You can't use paged results, since under the hood the server still keeps track of how many you retrieve and will stop once you reach the limit, which in this case is 500. What can you do?

One option is to sort the result and use carefully crafted search filters to get everything anyway. You sort on "cn" and you try with a filter "(cn=a*)" and see where that gets you. If you hit the size limit, you must limit the search further. If you get a result, you can continue with "(cn=b*)".

The key issue here is that the result is sorted, because otherwise there is no way you can keep track of where to begin the next search.

[djeanprost]

Ok, I think I uderstood. I'll investigate this way.
thank you

[brianbeech]

I agree with djeanprost, this really is an unacceptable work around.

Say there is an LDAP search limit of 500 and I'm trying to get all the user's in my LDAP. Even if I do a loop and search filter by alphabet (ex. (cn=a*) then (cn=b*)) eventually even these searches will fail; given that the organization continues to grow. So code written today may break in the -not so distant- future. This poses a real problem in the gov't space, where release cycles are surrounded by time consuming paperwork (and i mean TIME CONSUMING).

Anyone have any updates on how to get around this or how to elegantly navigate these waters? Not having control of the LDAP, but having to read from it, may be more of a problem. Maybe if there is a way of counting the entire results without returning anything, then doing logic; or being able to tell the LDAP which part of the result set you desire.