Working with Large List Item (List Threshold) CAML Query does not work – GetFolderByServerRelativeURL

Again, let us discuss a strange issue. I have a document library which is going to have almost 1 million documents in it. The Library has few folder structure and the documents were spread across these folders. When we go to the Settings, the screen will be

Through the code, I was iterating the documents by getting the hold of the Folders. The sample code was like,

Now, the scenario was I need to iterate through the F1.1 and get all the files, process those files. For that, the steps which I have followed are,

1. Get the document Library

2. Get the Root Folder

3. Get the Sub Folders (Folder 1, 2, … x)

4. Get the Sub Folders again (F1.1, 1.2 ….. 1.x)

5. Then Iterate through the Files.

It was working fine, till the number of folders in the level F1.4999. When the threshold for the folder reaches 5000, the Subfolders count returned 0.

Please find the piece of code below. (Though it is the wrong code, I just wanted to publish, so that it can be a BAD Example)

 using (var clientContext = authMgr.GetSharePointOnlineAuthenticatedContextTenant(siteUrl, userName, password))
             {
                 
                 Web web = clientContext.Web;
                 clientContext.Load(web);
                 clientContext.Load(web.Lists);
                 clientContext.ExecuteQuery();
 
                 List list = web.Lists.GetByTitle(“MyDocLibrary”);
                 clientContext.Load(list);
                 clientContext.Load(list.RootFolder);
                 clientContext.Load(list.RootFolder.Folders);
                 clientContext.ExecuteQuery();
                  
                 foreach (Folder topFolder in list.RootFolder.Folders)
                 {
                      
                         clientContext.Load(topFolder);
                         clientContext.Load(topFolder.Folders);
                         topFolder.EnsureProperties(f => f.ServerRelativeUrl);
                         clientContext.ExecuteQuery();
 
                          
                          
                         foreach (var SubFolderName in lstTargetProjects)
                         {
                             System.Console.ForegroundColor = ConsoleColor.Yellow;
                             System.Console.WriteLine("Processing the Folder : " + SubFolderName);
                              
                             if (topFolder.FolderExists(SubFolderName))
                             {
 // Do My Manipulations here.
 }
 }
 }
 }
 

The code, topFolder.FolderExists(SubFolderName) was giving the result, until the count reaches 5000. Once, it crosses that, always it gives back FALSE.

I don’t have any clue why it is always returning FALSE.

After few though process, got the solution from Microsoft, instead of using the Folders, iterating the folders, use the GetFolderByServerRelativeURL from the WEB class.

 Folder subFolder = web.GetFolderByServerRelativeUrl("/sites/MySite/MyLibrary/MyFolder/" + MySubFolder);
                         clientContext.Load(subFolder);
                         clientContext.Load(subFolder, sf => sf.Files, sf => sf.Folders);
                         clientContext.ExecuteQuery();
 
 

Happy Coding,

Sathish Nadarajan.