For about several months we are trying to figure out a bug with Adobe. Support is telling me they are unable to reproduce the issue as always. The problem is that the DataDirect Driver for Microsoft SQL that Adobe delivers with ColdFusion 10 and 11 product seems to have a major memory leak. You can analyse this very easily by running your system for a few hours and creating a heap dump on your CF servers and open the resulting .bin file with Eclipse Memory Analyzer (MAT).
- If you never made a heap dump yet, you need to install JDK on the ColdFusion server very first.
- Check out task manager or execute
Get-Process coldfusion | ft Id, ProcessName, Path -auto in powershell to find the PID of your coldfusion.exe process.
- Execute
"C:\Program Files\Java\jdk1.7.0_45\bin\jmap" -dump:format=b,file=C:\heapdump.bin [PID] where [PID] is the process id of the task.
- You get the heap dump file c:\heapdump.bin that you need to copy to a computer with Eclipse MAT installed. Make sure the MAT - MemoryAnalyzer.ini file has a Xmx value that is larger than your heap dump file in size. e.g.
-Xmx7g for 7GB.
- Now open Exclipse MAT and go to File > Open Heap Dump
- Select Leak Suspect Report and run it
You will find two memory leaks. Please note below leak analysis was done only after a 4-5 hours run time. If you wait for longer time that instances and memory usage will grow more extremely.
[4.591] instances of "macromedia.jdbc.sqlserverbase.dddg", loaded by "coldfusion.bootstrap.BootstrapClassLoader @ 0x6b5e2bea8" occupy 1.030.929.088 (59,42%) bytes. Keywords: macromedia.jdbc.sqlserverbase.dddg, coldfusion.bootstrap.BootstrapClassLoader @ 0x6b5e2bea8[2.781] instances of "macromedia.sqlserverutil.ddo", loaded by "coldfusion.bootstrap.BootstrapClassLoader @ 0x6b5e2bea8" occupy 189.392.360 (10,92%) bytes. Keywords: macromedia.sqlserverutil.ddo, coldfusion.bootstrap.BootstrapClassLoader @ 0x6b5e2bea8
So far so bad. You know there is a memory leak. Adobe provided us the driver of ColdFusion 11 version to give it a try and we also tried a non-public version that is post ColdFusion 11 RTM, but this will also not solve the memory leak. The only issues that disappeared are the macromedia.sqlserverutil.ddo instances that disappeared with the CF11 driver. Just a side note, the issue is not caused by FusionReactor as you see the same behavior without FusionReactor.
The JVM arguments increase maximum heap size and perm size to -Xmx4096m -XX:MaxPermSize=512m only and the arguments -XX:NewRatio=4 -XX:SurvivorRatio=8 -XX:+UseCompressedOops -Xss256k -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:CMSInitiatingOccupancyFraction=68 to workaround the issue documented at ColdFusion Threads Hanging On JDBC Pool CheckIn / CheckOut. We are running these settings for more then 6 months and they helped a lot.
We are currently investigating the replacement of the driver with Microsoft JDBC Driver 4.0 für SQL Server driver and it looks like this can be switched without any issues.
Heap with Adobe / DataDirect MsSQL Driver
Very high and bad memory usage compared to Microsoft driver. Memory usage will grow over longer period of time until system goes out of memory. CPU usage will also grow extremely over time to 50% or more where a normal load is 5-10% when the service is restarted. This is one extreme example:
This are more typical examples, but it always ends with the same result - out of memory - it only takes a much longer time:
Heap with Microsoft JDBC Driver 4.0 for SQL Server
Relatively low and stable memory usage compared to Adobe driver. Great result, isn't it?
The result looks excellent and we can only strongly recommend to replace the Adobe / DataDirect driver with the Microsoft JDBC driver. It is obvious that Microsoft know better how to write a driver for their database. See How to configure data sources with Microsoft JDBC Driver 4.0 for SQL Server if you need some help setting this up. We have not changed anything else in this step except replacing the driver!
History:
- 11/28/2015: Just a short update. We are running this configuration for about 1.5 years without any issues. CF has extremly stabilized.
- 08/25/2014: Decreased maximum heap size values from
-Xmx6144m back to -Xmx4096m as memory usage is back to normal now and we only increased it to keep the machine a bit longer running with the Adobe drivers.
- 08/18/2014: Documented the bug to public.
- 01/31/2014: Issue identified.