|
7/8 |
2005/2/15-16 [Computer/SW/Languages/C_Cplusplus, Computer/SW/Compilers] UID:36174 Activity:moderate |
2/15 Technical question: we have a memory leak in our C code, we think, but it's not the sort of memory leak where the memory's unreferenced. What we'd like to do is sort of a poor-man's profile, we want to know who calls our memory allocator "New"... Sorta like a stack trace. Using an actual profiler is sort of difficult 'cause it's a parallel application. Thanks, --peterm \_ Not sure about parallel application, but how about looking into ccured? It reads in code, analyzes it, then generates another C code with annotation that could be useful. Never used it myself though. \_ I've heard very good things about Purify but it is not free. Also take a look at other tools used for Mozilla development: http://www.mozilla.org/projects/xpcom/MemoryTools.html Please tell us which tools you ended up using and what you thought about it, thanks! \_ seconded. it's been years, but I recall purify being able to summarize how much memory was allocated by what code very much like op is asking, e.g. counts of how many times a particular calling context was used to hit malloc. don't recall whether there were limits to how many levels of caller it tracked. \_ What is your platform (OS, version, etc.)? What is your compiler (vendor, version, etc.)? \_ It's definitely a problem with our code. Three compilers, 3 platforms: gcc 3.2 linux, HP cc on 21264, xlc on AIX. \_ The Boehm Collector is free, and can be used to detect leaks. http://www.hpl.hp.com/personal/Hans_Boehm/gc \_ valgrind is free, Linux only \_ I thought you said Viagra |
7/8 |
|
www.mozilla.org/projects/xpcom/MemoryTools.html FAQs How to debug memory leaks/refcnt leaks Last update: September 5, 2000 What tools do we have? The mozilla team has developed a number of memory analysis tools to augme nt commercial tools like Purify. These can help us more quickly spot and fix memory leaks and memory bloat (our term for taking up too much memo ry, aka footprint). Here's a list of what we have at our disposal: * BloatView - This tool dumps out per-class statistics on the total num ber of refcounts and instances, as well as unreleased refcounts and un-deleted instances, and the amount of memory consumed by them. below) * Boehm GC Leak Detector - The Boehm garbage collector has be modified to serve as a leak detector. It's output can be post-processed by the "Leak Soup" tool to find the roots of leaked objects. This lets developers quickly focus on the key objects that need to be freed, rather than the whole graph of objects to which they refer. below) * Trace-Malloc - A tree of callsites, where paths from the root to leav es form stack traces that call malloc, calloc, realloc, and free, can by built in-process by building with --enable-trace-malloc and running with --trace-malloc filename. are called, log events identifying the function call and its callsite are emitted to filename. below) * Menu items to interactively control Purify - Interactively triggering a dump of all leaks or new leaks from within the running application is still one of the best ways to debug leaks in your subsystem. Menu items for this can be enabled for both viewer and apprunner. How to turn on refcnt/memory logging Assuming you have a build with refcnt logging enabled (we'll tell you how to do that next), here's what you have to do to use it. All of the foll owing environment variables can be set to any of these values: * 1 - log to stdout * 2 - log to stderr * filename - write log to a file The log environment variables are: XPCOM_MEM_BLOAT_LOG If this environment variable is set then xpcom will use the "bloat" trac kers. The bloat trackers gather data for the BloatView output that occu rs when the program exits, when about:bloat is loaded, or a call to nsT raceRefcnt::DumpStatistics is made. When an addref/release/ctor/dtor call is made, the data is logged and at tributed to the particular data type. By default enabling this environment variable will cause the BloatView s oftware to dump out the entire database of collected data. If all you w ant to see is that data for objects that leaked, set the environment va riable XPCOM_MEM_LEAK_LOG. XPCOM_MEM_LEAK_LOG This is basically a subset of XPCOM_MEM_BLOAT_LOG, and only shows classe s that had object that were leaked, instead of statistics for all class es. XPCOM_MEM_REFCNT_LOG Setting this environment variable enables refcount tracing. Only enable this for severe pain (unless you are using refcount tracing or leaky, see below). What this does is to enable logging (to stdout) o f each and every call to addref/release without discrimination to the t ypes involved. The output includes mapping the call-stacks at the time of the call to symbolic forms (on platforms that support this) and thus will be *very* *VERY* *VERY* slow. It is not as slow w hen using XPCOM_MEM_LOG_CLASSES and XPCOM_MEM_LOG_OBJECTS XPCOM_MEM_COMPTR_LOG This environment variable enables logging of additions and releases of o bjects into nsCOMPtrs. XPCOM_MEM_ALLOC_LOG For losing architectures (those that don't have stack-crawl software wri tten for them), xpcom supports logging at the *call site* to AddRef/Rel ease using the usual cpp __FILE__ and __LINE__ number macro expansion h ackery. This results in slower code, but at least you get *some* data a bout where the leaks might be occurring from. XPCOM_MEM_LEAKY_LOG For platforms that support leaky, xpcom will endeavor to find at run tim e the symbols "__log_addref" and "__log_release" and if found, instead of doing the slow painful stack crawls at program execution time instea d it will pass the buck to the leaky software. This will allow your pro gram to actually run in user friendly real time, but does require that your platform support leaky. In addition, the following variable may be set to a list of class names: XPCOM_MEM_LOG_CLASSES Instead of slowing to a useless, instead you can slow to a meer crawl by using this option. When enabled, the xpcom logging software will look for the XPCOM_MEM_LOG_CLASSES environment variable (for platforms that support getenv). The variable contains a comma separated list of names which will be used to compare against the type's of the objects being l ogged. Note that setting XPCOM_MEM_LOG_CLASSES will als o list the serial number of each object that leaked in the "bloat log" (that is, the file specified by the XPCOM_MEM_BLOAT_LOG variable). An o bject's serial number is simply a unique number, starting at one, that is assigned to the object when it is allocated. You may use an object's serial number with the following variable to furt her restrict the reference count tracing: XPCOM_MEM_LOG_OBJECTS Set this variable to a comma-separated list of object serial number or r anges of serial number, eg, 1,37-42,73,165. When this is set, along w ith XPCOM_MEM_LOG_CLASSES and XPCOM_MEM_REFCNT_LOG, a stack track will be generated for only the specific objects that you list. Note that this number does not reflect any memory held onto by the class, such as internal buffers, etc. Data points are taken whenever new and delete are called (not at regular intervals). Data points are taken whenever AddRef and Release are called (not at regular intervals). If they aren't, then you're not using the NS_IMPL_ADDREF and NS_IMPL_RELEASE (or NS_IMPL_ISUPPORTS which calls them) for xpcom objects, or MOZ_COUNT_CTOR and MOZ_COUNT_DTOR for non-xpcom objects. That means no one is looking at them, and we won't be able to tell if someone introduces a leak. below for how to fix this) * The Bytes Leaked for your classes should be zero! h), or perhaps you're just not freeing any of the instances you've allocated. This class might be a candidate for a non-xpcom object (but be very cautious about changing this if it implements any interfaces besides nsISupports). This indicat es a few objects of this class are allocated and then freed, as opposed to the opposite case where there's a big build-up of instances that are all freed together (perhaps at the end of the program). about: bloat in the location bar, or by using the menu items under the QA menu in debug builds. Note that you need to have the XPCOM_MEM_BLOAT_LOG or X PCOM_MEM_LEAK_LOG envirionment variable defined first. txt), and the total amount of memory allocated in the fourth log file. These columns provide an idea of how hard the memory allocator has to work, but they do not indicate the size of the working set. The second set of columns, Bytes allocated but not freed, shows the net m emory gain or loss by subtracting the amount of memory freed from the am ount allocated. The Show Objects and Show References buttons show the same statistics but counting objects or AddRef'd references rather than bytes. The bloat number is a metric determined by multiplying the total number of objects allocated of a given class by the class size. Note that although this isn't necessarily the amount of memory consumed at any given time, it does give an indication of how much memory we're consuming. The more memory in general, the worse the performance and footprint. T his happens when something that didn't leak before is now leaking. Bloat Statistics on Tinderbox Each build rectangle on Tinderbox will soon be capable of displaying the total leaks delta and bloat delta percentages from one build to the next . Warren checked in and the number of leaks went down by 3%. Sometimes bloat can go up because new features were added that just take up more memory (or if the set of test URLs were ch anged, and the activity is different from last time), but in general we' d like to see both of these numbers continue to go down. You can look at the end of the log (by clicking on the L) to see the bloat statistics a nd delta report for a breakdown of what actually happened. It is best to set t... |
www.hpl.hp.com/personal/Hans_Boehm/gc -> www.hpl.hp.com/personal/Hans_Boehm/gc/ Weiser conservative garbage collector can be used as a garbage collecting replacement for C malloc or C++ new. It al lows you to allocate memory basically as you normally would, without exp licitly deallocating memory that is no longer useful. The collector auto matically recycles memory when it determines that it can no longer be ot herwise accessed. The collector is also used by a number of programming language implementa tions that either use C as intermediate code, want to facilitate easier interoperation with C libraries, or just prefer the simple collector int erface. Later versions may contain additional features, platform support, or bug fixes, but are likely to be less well tested. Note that versions co ntaining the letters alpha are even less well tested than others, especi ally on non-HP platforms. IT IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRAN TY EXPRESSED OR IMPLIED. Empirically, this collector works with most unmodified C programs, simply by replacing malloc with GC_malloc calls, replacing realloc with GC_rea lloc calls, and removing free calls. Platforms The collector is not completely portable, but the distribution includes p orts to most standard PC and UNIX/Linux platforms. The collector should work on Linux, *BSD, recent Windows versions, MacOS X, HP/UX, Solaris, T ru64, Irix and a few other operating systems. Irix pthreads, Linux threads, Win32 threads, Solaris threads (old style a nd pthreads), HP/UX 11 pthreads, Tru64 pthreads, and MacOS X threads are supported in recent versions. Their collector takes advantage of multiple processors during a collection. Starting with collector vers ion 60alpha1 we also do this, though with more modest processor scalabi lity goals. finalization code to be invoked when an object is collected. It can tak e advantage of type information to locate pointers if such information i s provided, but it is usually used without such information. cord) package that provides for fast concatenation and substring operations on long s trings. A simple curses- and win32-based editor that represents the enti re file as a cord is included as a sample application. Performance of the nonincremental collector is typically competitive with malloc/free implementations. If the collector is used in a multithreaded environment an d configured for thread-local allocation, it may in some cases significa ntly outperform malloc/free allocation in time. We also expect that in many cases any additional overhead will be more th an compensated for by decreased copying etc. if programs are written and tuned for garbage collection. Boehm, H, "Dynamic Memory Allocation and Garbage Collection", Computers in Physics 9, 3, May/June 1995, pp. This is directed at an othe rwise sophisticated audience unfamiliar with memory allocation issues. T he algorithmic details differ from those in the implementation. There is a related letter to the editor and a minor correction in the next issue . "Mostly Parallel Garbage Collec tion", Proceedings of the ACM SIGPLAN '91 Conference on Programming Lang uage Design and Implementation, SIGPLAN Notices 26, 6 (June 1991), pp. "Space Efficient Conservative Garbage Collection", Proceed ings of the ACM SIGPLAN '93 Conference on Programming Language Design an d Implementation, SIGPLAN Notices 28, 6 (June 1993), pp. Boehm, H, "Reducing Garbage Collector Cache Misses", Proceedings of the 2000 International Symposium on Memory Management . Describes the prefetch strategy incorpor ated into the collector for some platforms. Explains why the sweep phase of a "mark-sweep" collector should not really be a distinct phase. M Serrano, H Boehm, "Understanding Memory Allocation of Scheme Programs ", Proceedings of the Fifth ACM SIGPLAN International Conference on Func tional Programming, 2000, Montreal, Canada, pp. Discusses the parallel coll ection algorithms, and presents some performance results. Boehm, H, "Bounding Space Usage of Conservative Garbage Collectors", Pro ceeedings of the 2002 ACM SIGPLAN-SIGACT Symposium on Principles of Prog ramming Languages, Jan. Includes a discussion of a collector facility to m uch more reliably test for the potential of unbounded heap growth. The following papers discuss language and compiler restrictions necessary to guaranteed safety of conservative garbage collection. We thank John Levine and JCLT for allowing us to make the second paper av ailable electronically, and providing PostScript for the final version. com is used for discussions, bug reports, and th e like. On-topic posts by nonsubscribers will usually also be accepted, but it may take some time to review them. |