It all started with a slow synchronisation. Further inspection yielded several elements to the solution that strayed from recommended practice. One aspect that was definitely playing a part in the slow synchronisation was several uniqueness checks against Active Directory Domain Services (AD DS) within the rules extension(s). The check was required – to call-out or not to call-out is an argument for another time – but some of the locations were wrong. My initial recommendation was to pull the uniqueness attributes, e.g. the sAMAccountName attribute, of all users into a Dictionary<String,String> within IMVSynchronization.Initialize and then use this local dictionary object for uniqueness checks (adding unique values as they are consumed). There were some other recommendations as well but I won’t bore you with those.
Jump forward two or three months and we’ve got a critical situation – a severity A Premier Support case. Over the course of a full synchronisation the FIM Synchronization Service exhausts available RAM and (gracefully, I might add) exits. The host in question is a VM; 8GB RAM, 60GB HDD, 12GB page file. Customer takes some traces and dumps and provides them to CSS and David quickly identifies that wldap32 is using all the RAM – 7GB in the data we’re looking at (while everything is still running). The data shows ~65,000 125KB allocations. Of 8.12GB of outstanding allocations 7.54GB from this stack alone! FIM is doing this. We prove it’s the rules extension and not FIM itself by running it in it’s own process and now mmsscrpt.exe (the process that runs rules extension code marked as “run out of process”) is using the RAM and not miiserver.exe (FIM Synchronization Service).
wldap32 is the Windows LDAP API. Active Directory Services Interfaces (ADSI) sits on top of this. And System.DirectoryServices (S.DS) sits on top of ADSI. What is the rules extension using? Right. We have a memory leak coming from S.DS.
If we whip out the bible the issue is obvious. Several of the classes within S.DS have to be explicitly disposed. Let’s look at some documented remarks:
Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed.
Furthermore, Joe and Ryan have a sidebar in the bible that discusses Close() and Dispose(). Let me quote some of the text:
Both Close and Dispose are intended to be used to clean up the underlying COM object. The primary difference between the two is that Dispose also suppresses .NET finalization, and Close does not. Supressing finalization means that the garbage collector will not bother to run the Finalize method on the object because we have signalled that we have already cleaned up the underlying resource that the finalizer needed to take care of. Objects that need to be finalized are automatically promoted one garbage collection generation, so the tend to hang around in memory longer, which is something we probably want to avoid if possible.
And:
In summary, use Dispose. It does everything that Close does, and it takes care of the finalization.
Better yet, use the built-in language features to ensure that objects are Disposed properly. In C# always use the using construct:
Later in the book they call out the SearchResultCollection class too:
This collection of SearchResult instances represents the result set as accumulated by the server for a specific query. This class internally holds references to unmanaged resources, and as such should always explicitly be disposed using the Dispose method, just like DirectoryEntry and DirectorySearcher.
The crux of the matter here is this: when using S.DS in a process that runs in a loop it is imperative that the objects are disposed. In the case of FIM the loop is synchronisation, i.e. calling the rules extension code over and over again (per managed object).
The customer quickly fixed the utility class that was using DirectorySearcher to return a SearchResultCollection to another method that was consuming it by properly calling Dispose in a couple of places and deployed the code. The result was different but the same. Different allocation sizes but still a leak. Bring in the heavy artillery and the ninjas identify the actual method, in the customer rules extension, that is making the call(s). A look at the revised code quickly identifies the issue. The S.DS.DE and S.DS.DS objects are being disposed but the SearchResultCollection isn’t – probably because of the way it is being used, i.e. the object is returned by a method. A second flaw is also identified –not the root of the problem but something worth fixing– all attributes are being returned. We provide some recommendations around rejigging the method and properly disposing and also favouring a using statement in many places.
The next result was…problem solved. While we see increased memory usage during synchronisation five minutes after synchronisation and it’s freed up (expected, as in-process rules extensions are unloaded after five minutes of inactivity).
Before I finish this post I’ll state some observations. Many of us know these values but I cannot see that they’re actually publically documented.
- A rules extension that runs in-process is unloaded after five minutes of inactivity. You should expect to see all memory used by your rules extension DLLs returned to the OS five minutes after synchronisation completes.
- A rules extension that runs out-of-process is unloaded after fifteen minutes of inactivity.
- Unmanaged memory and/or memory that is leaked is not returned to the host after IMVSynchronization.Terminate and/or IMASynchronization.Terminate is called. The leaked memory is only cleaned up when the hosting or parent process is recycled. For out-of-process rules extensions this means the memory is returned after fifteen minutes when FIM kills mmsscrpt.exe. For in-process rules extensions the memory won’t be returned until the FIM Synchronization Service (miiserver.exe) is recycled.
It was an interesting case and I learned some very valuable information. Hopefully you’ll find this interesting and helpful too.
p.s. Kudos to David, Steve and Rob Lane, without these clever folk we would not have got to the bottom of the problem!
