1. Number of files and kilobytes used per file owner (who is using the same)
2. Number of files per file extension and kilobytes (useful to determine how many audio files, video files, Office documents, etc…)
3. Age of files grouped into several buckets (0 to 6 months, 6 to 12 months, 12 to 24 months, and over 24 months)

The first method, “CrawlDirectories”, accepts the root directory you want to start crawling, any directories you want to exclude, and if you want to retrieve file owner information for each file.

private void CrawlDirectories(string RootDirectory, bool getFileOwners)
{
    // get a directory object
    try
    {
        DirectoryInfo di = new DirectoryInfo(RootDirectory);

        // loop through directories and look at files
        try
        {
            DirectoryInfo[] myDi = di.GetDirectories();

            Parallel.ForEach(myDi, folder =>
            { 
                    //look at files
                    AddFiles(folder.GetFiles(), getFileOwners);

                    //see if subdirectories to crawl
                    CrawlDirectories(folder.FullName, excludeDirectory, getFileOwners); 
            });
        }

        catch (Exception ex)
        {
            myLog("Error getting subdirectories: " + di.FullName + "; " + ex.Message, true);
        }
    }
    catch (Exception ex)
    {
        myLog("Error getting directory Info: " + RootDirectory + "; error: " + ex.Message, true);
    }
}

Notice the For loop using the new “Parallel” class above (in red).  This defines a parallel loop.  The application will spin up multiple threads to crawl through the directories.  Within this loop, we are calling the “Addfiles” method which loop through the files within the current directory, and then it calls itself to recursively crawl any subdirectories within the current directory.  Monitoring my machine resources, I can see on average 70 threads spin up to crawl the directories.  All of these is handled automatically by the framework!

The next method, “AddFiles”, will use a parallel loop to report on the files within the directories.  When you are using arrays or other types of collections in a multi-threaded application, you must use thread-safe collections!  I have chosen to use “System.Collections.Concurrent.ConcurrentDictionary” to store the file data I’m collecting.  This is basically a key, value pair that is thread-safe and can be accessed by multiple threads concurrently.

private void AddFiles(FileInfo[] filesInDir, bool getFileOwners)
{
    string ownerName = "";
    string fileExtension = "";
    Parallel.ForEach(filesInDir, fi =>
    {
        try
        {
            //calculate # of files and bytes per file owner                  
            if (getFileOwners)
            {
                IdentityReference NTAccountName = fi.GetAccessControl(AccessControlSections.Owner).GetOwner(Type.GetType("System.Security.Principal.NTAccount"));
                ownerName = NTAccountName.Value.ToUpper();
                ownerBytesDC.AddOrUpdate(ownerName, fi.Length, (key, old) => old + fi.Length);
                ownerCountDC.AddOrUpdate(ownerName, 1, (key, old) => old + 1);
            }

            //calculate file extension # of files and bytes
            fileExtension = fi.Extension.Replace(".", "").Replace(",", "").ToUpper();
            extensionBytesDC.AddOrUpdate(fileExtension, fi.Length, (key, old) => old + fi.Length);
            extensionCountDC.AddOrUpdate(fileExtension, 1, (key, old) => old + 1);

            //calculate ages of files
            if (fi.LastAccessTime >= DateTime.Now.AddMonths(-6))
            {
                System.Threading.Interlocked.Add(ref TotalByte0to6mo, fi.Length);
                System.Threading.Interlocked.Add(ref NumberOfFiles0to6mo, 1);
            }
            else if (fi.LastAccessTime < DateTime.Now.AddMonths(-6) && fi.LastAccessTime >= DateTime.Now.AddMonths(-12))
            {
                System.Threading.Interlocked.Add(ref TotalByte6to12mo, fi.Length);
                System.Threading.Interlocked.Add(ref NumberOfFiles6to12mo, 1);
            }
            else if (fi.LastAccessTime < DateTime.Now.AddMonths(-12) && fi.LastAccessTime >= DateTime.Now.AddMonths(-24))
            {
                System.Threading.Interlocked.Add(ref TotalByte12to24mo, fi.Length);
                System.Threading.Interlocked.Add(ref NumberOfFilee12to24mo, 1);

            }
            else if (fi.LastAccessTime < DateTime.Now.AddMonths(-24))
            {
                System.Threading.Interlocked.Add(ref TotalByteOver24mo, fi.Length);
                System.Threading.Interlocked.Add(ref NumberOfFilesOver24mo, 1);
            }
        }
        catch (Exception ex)
        {
            myLog("Cannot report on file: " + fi.FullName + "; error: " + ex.Message, true);
        }
    });
}

We end up with several key, value pair collections:

1. ownerBytesDC and ownerCountDC.  These collections both have a key of the user name, and the value of the kilobytes total for that user and the count of files. 
2. extensionBytesDC and extensionCountDC.  Key is the file extension, with values of total bytes and file count.
3. The remaining data is stored in Int64 variables for the age of files in each age bucket.

ASP.NET 4.0 framework provides a great way to update a collection in thread-safe manner.  The collection has a method “AddOrUpdate()” that will add the key if it doesn’t exist in collection, or update it if it’s already there.

You may be wondering why we have the “getFileOwners” boolean parameter on the method.  Getting the file owner of a file requires a call to the “GetAccessControl” method – which is part of the file security methods within Windows.  This operation is expensive!  It slows down the crawl by several-fold.  Thus, if you don’t require this information, you can turn it off.

This application can crawl through about two million files and three terabytes of data per hour (tested on a two-proc VMware machine).  More robust hardware could speed this up.

Easy multi-threaded applications – finally!

1. Windows 7 64-Bit
2. Utilizing an external USB drive to host the guest VHD
3. 8GB RAM
4. Using a Dell laptop (tried on a D630 and Precision M4500)

To fix this issue, navigate to the following file and add these settings:

C:\ProgramData\VMware\VMware Workstation\config.ini AND C:\ProgramData\VMware\VMware Player

prefvmx.minVmMemPct = "100"
mainMem.useNamedFile = "FALSE"
mainMem.partialLazySave = "FALSE"
mainMem.partialLazyRestore = "FALSE"

You should notice immediate improvement in shutdown speed.

THE LEFTOVERS BECKONED -
THE DARK MEAT AND WHITE,
BUT I FOUGHT THE TEMPTATION
WITH ALL OF MY MIGHT.

TOSSING AND TURNING WITH ANTICIPATION,
THE THOUGHT OF A SNACK BECAME INFATUATION.
SO, I RACED TO THE KITCHEN, FLUNG OPEN THE DOOR,
AND GAZED AT THE FRIDGE, FULL OF GOODIES GALORE.
GOBBLED UP TURKEY AND BUTTERED POTATOES,
PICKLES AND CARROTS, BEANS AND TOMATOES.

I FELT MYSELF SWELLING SO PLUMP AND SO ROUND,
‘TIL ALL OF A SUDDEN, I ROSE OFF THE GROUND.
I CRASHED THROUGH THE CEILING, FLOATING INTO THE SKY,
WITH A MOUTHFUL OF PUDDING AND A HANDFUL OF PIE.
BUT, I MANAGED TO YELL AS I SOARED PAST THE TREES….
HAPPY EATING TO ALL – PASS THE CRANBERRIES, PLEASE.

MAY YOUR STUFFING BE TASTY,
MAY YOUR TURKEY BE PLUMP.
MAY YOUR POTATOES ‘N GRAVY HAVE NARY A LUMP.
MAY YOUR YAMS BE DELICIOUS.
MAY YOUR PIES TAKE THE PRIZE,
MAY YOUR THANKSGIVING DINNER STAY OFF OF YOUR THIGHS!!

HAPPY THANKSGIVING TO ALL

1. Use WIQ.  As stated above, this option is limited to basic sorting.  You can sort by a single field, or multiple fields.  Conditional logic or advanced grouping is not supported.
2. Develop a web service triggered by the Work Item Changed event.  By adding a field to the work item definition (perhaps an integer field labeled "SortOrder") you can evaluate the work item to determine what sort order it should be assigned.  In our case, we identified 5 sort levels, so the "SortOrder" field was assigned a number between 1 and 5.  Work Item queries were written to sort based on this field first, then by other fields to secondary sort the work items.

   The web service gets run after you save the work item.  The web service event fires, evaluates the work item, assigns a value to the "SortOrder" field, and then saves the work item.

   Sample code (this uses the TFSEvents framework available at Codeplex):

   public void Notify(string eventXml, string tfsIdentityXml)
       {
           WorkItemChangedEvent workItemChangedEvent = this.CreateInstance<WorkItemChangedEvent>(eventXml);
           TFSIdentity tfsIdentity = this.CreateInstance<TFSIdentity>(tfsIdentityXml);

           int WorkItemID = workItemChangedEvent.CoreFields.IntegerFields[0].NewValue;
           Microsoft.TeamFoundation.Client.TeamFoundationServer TFS = new TeamFoundationServer(tfsIdentity.Url);
           TFS.Authenticate();

           WorkItemStore WIS = TFS.GetService(typeof(WorkItemStore)) as WorkItemStore;
           WorkItem updatedWI = WIS.GetWorkItem(WorkItemID);
           string priority = "";
           string environment = "";

           if (updatedWI.Fields.Contains("CustomFields.SortOrder"))
           {
               priority = updatedWI.Fields["CustomFields.Priority"].Value.ToString();
               environment = updatedWI.Fields["CustomFields.FoundInEnvironment"].Value.ToString();

               if (priority == "1-Now")
               {
                   updatedWI.Fields["CustomFields.SortOrder"].Value = 1;

               }
               else if (priority == "2-ASAP")
               {
                   updatedWI.Fields["CustomFields.SortOrder"].Value = 2;
               }
               else if (updatedWI.Fields["CustomFields.CRID"].Value.ToString() != "" ||
                   (updatedWI.Fields["System.IterationID"].Value.ToString() != "154" &&
                   updatedWI.Fields["System.IterationID"].Value.ToString() != "314"))
               {
                   updatedWI.Fields["CustomFields.SortOrder"].Value = 3;
               }
               else if (environment == "2-PRE-PROD" || environment == "3-Test" || environment == "4-DEVL")
               {
                   updatedWI.Fields["CustomFields.SortOrder"].Value = 4;
               }
               else if (environment == "1-PROD")
               {
                   updatedWI.Fields["CustomFields.SortOrder"].Value = 5;
               }

               if (updatedWI.IsDirty)
               {
                   updatedWI.Save();
               }
           }
       }

   A big disadvantage to this method — the work item gets updated by the web service, thus if a user has the work item loaded on their screen, makes a change, and tried to save, it will fail and notify the user the work item has been updated by another user.  This scenario is very likely to happen since the user will have the work item on their screen during the initial save and may want to add additional data or update information on the work item.  They are forced to close and reopen the work item after each save.  For most organizations, this would be frustrating and probably disqualify this solution.

3. Develop a client-side custom control.  This option is similar to the web service, but the logic lives on the client.  Using a custom field such as "SortOrder" you can assign a sort value based on the work item information.  The logic runs before the work item is saved, so only one update/save operation is required.  The advantage to this method is you don’t have the update problem as you do with the web service.  The disadvantage is that this control needs to be deployed to all your developer/client machines or else the sort logic won’t run and be assigned.

Based on all these options, the best option if you require advanced sorting logic is to develop a client-side custom control.