Daybo Logic DPCRTLMM documentation

Daybo Logic's C memory management library V1.2.0
technical documentation

Contents
1. About the library
1.1 So who are Daybo Logic?
1.2 Contacting the author
1.3 FAQ
1.4 Compiling the sources
1.5 Configuration options explained

2.0 Enumerations and detail
2.1 How much does it cost?
2.2 What platforms is the library available for?

3.0 Concepts and structures used by the library
3.1 Getting started
3.2 Portability issues
3.3 8086 builds
3.4 Precompiled headers

4.0 History
4.1 Plans for next version

5.0 Function listings
5.1 Descriptions of all functions and their use

1. About the library
This could be used as an alternative to
memory tracking tools such as CodeGuard of MemorySleuth,
it's not as thorough as them however, it can't detect memory over
runs on pointer accesses or anything like that. It can be used
to detect failures to release memory (as long as it's allocated
by us) and attempts to release memory which we do not own.
It has an automatic garbage collection feature to free up all
unfreed memory too. It should be used by the entire program to
be most effective, it can provide statistics on memory left and
such like too. The best thing about the MM is that it provides
a central place for dynamic memory allocation in a program, so
we can trace all activity, an optional log mode is available too.
Swapping is not implemented but may be in a future version.
If the program is going to be released even though
it contains memory bugs, the traps can be turned off or passed to
a handler which will only execute the serious ones.

1.1 So who are Daybo Logic?
Daybo Logic started in 1997 as a software development company. It then became a general troubleshooting company, as a sole trader, David Duncan Ross Palmer and in 2006, it reverted to being purely a programming hobby of mine. I still release all my software, including source code on the Daybo Logic web-site at www.daybologic.co.uk.

1.2 Contacting the author
The author of this library is David Duncan Ross Palmer, a C/C++ programmer since 1997 and a BASIC programmer since 1989. Born 5th March 1982 and interested in computers since an early age. This product was a side effect of wanting to be able trap memory errors on users machines during beta testing phase, it was not originally designed to be released, it was originally too dependant on other Daybo Logic internal libraries. It was then offered for download for free and a fee for the advanced features. Eventually the whole source was open sourced under the GNU General Public License. IN 2006, the library was re-released under a BSD-style license. The author of the library can be conacted by visiting the following web-site: http://www.daybologic.co.uk/mailddrp/. I especially welcome bug reports, feature requests and technical questions.

1.3 FAQ - Frequently Asked Questions and common compile problems

Q - I am using ./configure --enable-threads --with-gnu-pth to link with the GNU Portable Threads library. As soon as my program initialises the DPCRTLMM library, the program crashes with a segmentation fault. Is this a bug?
A - No, GNU Portable Threads needs explicit initialisation before use. DPCRTLMM uses GNU Portable threads for an internal lock and initialises this lock when DPCRTLMM is initialised. It is your program's responsibility to initialise GNU Portable Threads before DPCRTLMM is initialised. You will need to call pth_init( ) at some point before dpcrtlmm_Startup( ) is called.

Q - Error dpcrtlmm.h (line number): Declaration missing ;
A - The following answer applies to 16-bit segmented programming only. This only happens if you selected to have explict far data for the 16-bit tiny, small or medium memory models. The answer is one of these solutions I hope, firstly check the language is set to non-ANSI compliance, if your compiler only supports ANSI you cannot support explicit far pointers so give up! The other problem is on line 112 (ish) of BUILD.H #define DPCRTLMM_FARDATA _ _ far, change the _ _ far bit to just far and try again. Look for what the 'far' keyword equivilant is in your language. If you still have problems contact me.

Q - No prototype for function 'farmalloc' or 'farcalloc' or 'farfree' or 'farrealloc'
A - The following answer applies to 16-bit segmented programming only. The explicit fars are non-ANSI, unfortunately the functions differ from one compiler to the next when one strays from ANSI. In BUILD.H there are a list of functions here, change them to point to your compiler's equivilant functions if you can discover what they are. If you don't have any you will have to give up and turn off this option, make sure you don't have strict ANSI C compliance on or anything first but if all else fails just compile for the compact/large or huge model.

Q - I want to compile for 16-bit x86 processors in small memory models. However, the pointers returned from DPCRTLMM use my segment address and not real segment address, When I try to success these pointers, the program crashes, or unexpected results happen.
A - The library only functions correctly with far data, run ./configure with --enable-fardata to enable explicit support. Unfortunately, this is not within the scope of ANSI C and you may find difficulty compiling. If your compiler cannot support explicit far pointers, you may be forced to compile for another memory model or not use DPCRTLMM at all.

Q - My program is crashing unexpectedly when I try to free blocks of memory (or earlier while using the memory manager)
A - The most common problem is forgetting to start the library, if this is not the case and you are compiling for a 16-bit processor it may be that the data is near. The easiest way to avoid this is to rebuild in a memory model which has far data. There is another way in this version of the library, Configure with --enable-fardata. If this does not solve the problem please send me all details possible.

1.4 Compiling the sources
The way in which the library is built can change from time to time, last minute notes may be added in a file called README if these do not work.
Compiling under UNIX:

Unpack the souce tarball with the following command:
tar -zxvf dpcrtlmm-1.2.0.tar.gz or
tar -jxvf dpcrtlmm-1.2.0.tar.bz2 if you are using the bzip2ed version.

cd dpcrtlmm-1.2.0/
./configure --help

This will display a large output of help text. It may help to pipe this though 'more'. Read this help carefully and when you have decided upon the configuration options you want to use, type:
./configure [add any options you want to use here]
Once the script has configured (assuming all went well), type:
make.
This builds the library.
You can now type make check if you want to build and run some example/test programs.
Finally, as root, type make install.
The DPCRTLMM library is now installed and ready for use by your system.

Compiling under Win32 (Borland C):
(write these instructions)

Compiling under Win32 (Microsoft C):
(write these instructions)

Compiling under Win32 (Open Watcom C):
(write these instructions)

Compiling under DOS (Turbo C):
First, you will need to have either a Win32 or UNIX-like box available and have Perl installed. Then run perl ./mkdosdist.pl
This will update all of the of the source files in a sub-directory called dos_dist/ and rename them to DOS compatible names.
Finally, it will process each of the files multiple times to rename any internal references to the old filenames (this may take some time).
Warning: Do not run the mkdosdist.pl script if you have build the sources under another OS because you will have your own
config.h which will over-write the DOS-crafted one which is stored as dos_dist/config.h.
You now have a useable source for building with Turbo C. I have also supplied project files which may or may not work for you.
I built them from the directory c:\dpcrtlmm. If it works, you might be able to get away with typing:
tc /b dpcrtlmm.prj
This will build dpcrtlmm.lib from the sources. You may need to mess around with memory models and the contents of config.h
to get things to work. I found that things generally worked only if built with a large memory model. This is generally what you
will need to do nowadays anyway, as people don't tend to explicitly use far or near pointers any more.

1.5 Configuration options explained
Since version 1.2.0, the old config program has been removed. It has been replaced with a GNU autoconf generated configure script. This is good news for the portability and maintenance of DPCRTMM on UNIX platforms but it does means that pure DOS and Win32 users must either run the script via Cygwin (Win32) only, or edit config.h manually and then build their own makefile for their compiler. However, I have decided that it is in the best interest of the project because, although we do have DOS and Win32 users, the majority of the library's users use a form of UNIX and UNIX is my development platform for the library.
./configure --help displays some help but if you're still stuck, these explanations should reveal some more detail and history.

--enable-win32
The configure script can't tell if you are running it under Cygwin and indeed, you may still want to build for POSIX. Enabling this switch turns on specific Win32 code. If you are building with --enable-threads, it will use the Win32 API for mutual exclusion, rather than the pthread (POSIX threads) API.

--enable-debug
This option defines DEBUG, it means extra debugging code is added to DPCRTLMM for when I am debugging it, it does not add any extra information for debugging your programs, only to my code. This could cause the library to abort on assert( ) traps and such. If you want to help submit problems with DPCRTLMM, make sure it's built with --enable-debug. This will also add symbolic debug information to DPCRTLMM where possible.

--enable-threads
Enables multithread safety. Note: This does not inply support, DPCRTLMM can't handle threads portably for logging purposes etc. This option makes sure that DPCRTLMM is mutually exclusive, that is, if two threads attempt to call the library at the same time, one will succeed and one will have to wait until the first finishes. This stops internal corruption of DPCRTLMM's data structures but adds overhead, don't use it if you don't use pre-emptive threads. This is supported only on the following platforms: POSIX, Win32.

--with-gnu-pth
This will enable you to build against the GNU Portable Threads library, rather than the POSIX threads API. This is useful if your program uses the GNU Portable Threads rather than POSIX threads. You will get a warning if you specifiy --enable-win32 and --with-gnu-pth because at time of writing, The GNU Portable Threads are only supported on UNIX-like platforms. Your program is resonsible for initialising the GNU Portable Threads library before DPCRTLMM is started.

--enable-stderr
Error traffic will be printed onto stderr, a stream which can be redirected separately from the normal standard output stream.

--with-hook-chain-size:nnnn
The default hook chain size is 32. In DPCRTLMM before it was Open Sourced advanced debug hooks were only allowed in paid-for versions. to get rid of them set this to zero. Otherwise a debug matrix is used, a matrix of chains of hooks. Callers may install hooks into the matrix, up to nnnn hook per hooktype. For example, if monitoring allocation requests, only 32 (by default) hooks may be installed for the allocation type of hook. If one does not plan to use this feature, set this to zero to avoid space wastage. Being a matrix, a high value will occupy a large amount of space quickly. The total size used can be calculated by the hook chainsize, multiplied by the size of a hook pointer, multiplied by the number of debuggable hook types (eg. allocation is one specific type of debug hook).

--with-safety-list-size:nnnn
The default is 512. When creating new block arrays, each address of a block array is stored in an internal "safety list". It's a fixed size array determined here. It may not be zero but may be in the future. I recommend a new block array per module, so if the program contains 512 modules this is fine, otherwise adjust accordingly to save space or provide more space.

--enable-fardata
For tiny, small and compact memory models in 16-bit segmented systems ONLY. This makes sure pointers are not truncated between callers in modules using near pointers and the DPCRTLMM library in other modules using near pointers. This ensures the DPCRTLMM library explictly uses far pointers and the host program's code knows it. Far pointers are non-ANSI, therefore, if this doesn't work, editing build.h might be necessary, see the FAQ. If you enable this option on 32-bit platforms, you will just break the build.

--enable-hdrstop
This will produce #pragma hdrstop and allow pre-defined header stop-points for pre-processors which support it. It is harmless and will just produce a warning for every file if it is not supported.

--with-no-null-array
When I created the new NULL (default) array so that calls could be made to dpcrtlmm_Alloc( ) and other functions without specifying a block array, I realised that existing code might be broken if it accidently failed to created a new block array and then used it. Therefore I have always kept up an effort to make sure that the new NULL array could be ignored and cause the same errors as before. This really is for legacy support only.

--enable-log
A file called DPCRTLMM.LOG wil be created and record the activity of every library call. This is an alternative to installing debug hooks. Do not forget to disable this feature before releasing your program! DPCRTLMM.LOG can grow very quickly!

2.0 Enumerations and detail
Blah

2.1 How much does it cost?
Nothing, but I do accept donations via the web-site. If you have found this software useful, please donate. If you can't afford to donate (which I wholly understand), please send me an email letting me know what you are using the library with. I really would be interested to know! I get quite a bit of email but I do enjoy reading email from folk who are using my software.

2.2 What platforms is the library available for
The library has been tested on the following platforms: Win16, Win32, DOS (16-bit and 32-bit builds), GNU/Linux and FreeBSD.
I aim to make sure this library can be built anywhere and if you have any specfic problems building it, please let me know.

3.0 Concepts and structures used by the library
The primary memory managment structure is called a block descriptor, it contains basic information about a block of memory allocated by the program, it is not normally directly accessed by a program, it is generally only used by the library, it is possible to access it but I don't know why a program would need to because all information may be obtained using the library interface.

These block descriptors contain this simple information: The block's base address, size in bytes and a number of flags with special meanings.

The other structure you will encounter is the block descriptor array. I recommend that every C file maintains it's own array. When a block is allocated it's decriptor is always inserted into one of these arrays and nearly every library call will want to know the address of one of these arrays. This only causes a problem if the program passes a descriptor to another module the program which tries to get size information on the block using the library functions while passing it's own array, then the program will be terminated because the library will assume that the address is invalid. I recommend using NULL as a placeholder for arrays, for blocks which are designed to be shared. Normal sharing is fine, as long as the library is not used on that block from another module. Note: If you have configured the library --with-no-null-arrays, it will not be possible to use NULL in this manner.

The final structure is stored internally and the program should never need to access, this structure is called the safety list. It is a large fixed size array of all the addresses of the block descriptor arrays used throughout the program. The interface to this list is kept internal to the library, if you want to access it you could always add it to DPCRTLMM.H, although there should be no reason to do this. The safety list's size can be adjusted at configuration time via the --with-safety-list-size option.

So what is this safety list for if I can't access it under normal circumstances? Well, it's used to trap any call to the library which specifies an invalid pointer to a block descriptor array. It is also processed when the library is shut down to ensure all blocks have been released, it is therefore is the primary structure for producing a leak report.

3.1 Getting started
If you already have a program to convert to use the library I suggest this. Download at least version 1.1.4 and define USING_DPCRTLMM before including dpcrtlmm.h in each module. This makes malloc( ), calloc( ), realloc( ) and free( ), call DPCRTLMM instead. Also, since the library should be started and shutdown correctly, I recommend code similar to example3 as a typical transparent usage program.

If you are writing a new program use the library from the start, add a block descriptor array structure to the main module at file level, then in main call the library startup routine, create the array then allocate blocks using it. When main() is about to exit free all blocks, destroy the array with the dedicated function and shutdown the library. These destroy and shutdown routines are the main ones which check if anything was not freed from the array.

3.2 Portablilty issues
This library will slow the program down but not make it any less portable as such. However if you later want to speed the program up and remove the library read the section Getting Started, the first paragraph describes a hack, designing the program in the first place to use this hack is the only way to easily remove the library. Another approach might be to write a dummy version of the library, only doing the bare minimum, I intend to write one of these at some point soon.

By default the library source is portable, it can be setup to use some non-ANSI features:
far pointers, mutexes and critical sections. Most instructions for changes are in build.h, please use the configure script where appropriate.

3.3 8086 builds
When building the library for the 8086-80286 (16-bit versions of Intel's processor) the library should be built for one of these memory models: Compact, Large or Huge. Trying to build the library for smaller memory models results in near data. If such a build is made I have discovered that the hosting program will be assuming the wrong data segment when the pointers are returned from the library. There is a way to get around this if you really want to use a smaller memory model, build with --enable-fardata. This is non-ANSI and you're on your own if your compiler can't support the '__far' keyword.

3.4 Precompiled headers
My compiler (Borland C++) supports this pragma:
#pragma hdrstop
However using it on other compilers might cause a warning, to avoid annoyance to people, it is disabled by default. To enable it, re-configure with ./configure --enable-hdrstop.

4.0 History
2nd May 2000 - Overlord:
Removed dependancies that made this library rely on other Daybo Logic
libraries, this will mean the library can be used by Colelus without
disrupting the portability of the Colelus source tree. This step
involves stripping out all references to STDS\DLSTRLIB.H a very annoying
thing to do because all code which is Daybo Logic shared code (and therefore
not part of Colelus) should include this header. However if this library
source is to be made available to the public too, this dependancy must go.
The other dependacy which follows in a similar fasion is the use of the
generic list object as the list of block arrays, the so-called safety list.
To avoid this I will design a safety list object type which is only used by
us and indeed only makes sense to us! This new object definition is in
SAFELIST.H and the code to run is SAFELIST.C

3rd May 2000 - Overlord:
Totally got rid of SAFELIST.C/.H and decided to start again. The
declaration of the safety list object is in INTDATA.C and the size is
determined in the configuration file BUILD.H
The new SAFELIST.C will contain simple functions to save having to write
the list processing several times, code to count the number of items,
code to add a new one at the first blank entry and remove items.
There is no need for a linked list or shift up code because it's a fixed
size determined at compile time therefore anything which is not a valid
entry will just be NULL.

4th May 2000 - Overlord:
Compiled in the new SAFELIST.C and tried to build the library muliple times
until all references to other Daybo Logic stuff were stripped out. BOOL had
to change to unsigned int, however the user of the library may still treat
these values as BOOL/TRUE/FALSE even though they are used as unsigned ints
inside the library. The user can only do this if they have their own
compatible versions of BOOL/TRUE/FALSE which most developers have by now, or
can easily make them if they do now have them.
Also changed defintion of _safeList and all people who used it had to be
altered to be aware of the new definition.
Wrote a new page on the web site offering it for sale.

5h May 2000 - Overlord:
Wrote documentation for library and released it on the Daybo Logic web site
http://www.daybologic.co.uk

10th May 2000 - Overlord:
There was a bug in the new safety list which stopped the whole library
working, a single missing '!' (logical not). Republishing the library.

14th June 2000 - Overlord:
In 16-bit memory models on x86 processors, if the data segment is near the pointers do not mean anything to the hosting program when given to it, the near pointers are for us with DPCRTLMM's own assumed data segment, therefore explicit far data is needed, added a macro to build.h for doing so and severely modified this documentation.

18th July 2000 - Overlord:
OK this isn't a technical note, it's just that today I set myself the deadline of August 1st for the version 1.0 release.

19th July 2000 - Overlord:
Coverted all the filenames to lower case and all the #includes to lower case for compatibility with UNIX systems.
Tried and failed miserably to create a working makefile and the Borland C++ IDE auto-generated one came up with warnings about some of my symbols not being in the library, works ok from the IDE and I don't ever get warnings for it so I gave up on this for now. The library is saimple enough to build I feel.
    Merged BBATRAP and BBPTRAP, well, put them in the same module with another function call _VerifyPtrs() which calls both, this stops having to do both calls in some functions which looks untidy. Both functions were always called at once anyway, the new module is called vptrap.c
    In realloc.c the string "Realloc()" was used twice, optimized out second instance.
    Noticed quite a few modules called the old BBA trap and did not have a pointer to verify also, so I changed _VerifyPtrs() slightly so argument three could be NULL and the second trap call would be skipped. Well mustn't grumble, it still feels cleaner this way and I got rid of a whole module in effect!
Free() used the string "Free()" twice, made it share a variable, only has an effect on non-optimizing compilers which don't support duplicate strings merged.
    Removed all #pragma hdrstop PCH support unless DPCRTLMM_HDRSTOP is defined, to save annoying warnings to people who's compiler does not understand the pragma
Added support for NULL as the default built-in block array, define DPCRTLMM_NONULL_BLOCKDESCARRAY in build.h to restore old behaviour, old behaviour was to say that NULL was not a valid block desc array and crash the program.
    Added a hack table for shorter function names, this was suggested by Jeremy and will also shorten the structure names. Developers already using the library can totally ignore this.

20th July 2000 - Overlord:
In adding the support for default block arrays I had to split up the trap routine called TrapUnFreedArrays() in strtstop.c into a new routine also TrapUnFreedBlocks(). I realized that not all data which is unfreed (garbage) is just arrays now due to the default array which is never destroyed but can contain blocks. I renamed the define DPCRTLMM_TRAP_UNFREED_ARRAYS to DPCRTLMM_TRAP_UNFREED_DATA. If you used this (which I doubt) at this stage just rename the macro in your code.

21st July 2000 - Overlord:
Corrected an assertion failure due to the new NULL block array. In the betas, when DPCRTLMM_LOG was not defined the messages were not output to a log file but the messages were still taking space in the resulting program, got rid of most of them, some are used as traps too so I couldn't get rid of them. To try to stop users making the mistake of including build.h in their own programs I made build.h look for a macro I define in my source only.
Loads of warnings about variables that were never used when I tried to define DPCRTLMM_LOG for an experiment, filtered them all out, all the #ifdef DPCRTLMM_LOG #endif /*DPCRTLMM_LOG*/ stuff everywhere looks a right mess though.

24th July 2000 - Overlord:
Limited block descriptor arrays to 64 entries in standard builds

26th July 2000 - Overlord:
Added a new module getstats.c, countains these functions: dpcrtlmm_GetBlockCount( ) & dpcrtlmm_GetStats( ), GetStats( ) is very simple and it uses _blockCount which I added to intdata.c. Alloc()/Free() don't update the counter yet
Limited library to 50 blocks for standard builds, first set up a define in build.h so I could modify it easily and then put all the statistics (except flag counting) into internal vars (intdata.c). Flag counting is done by an internal routine called CountFlagsInUse( ), I put that in getstats.c
    Added dpcrtlmm_AreTrapsEnabled( )
    There is now an extern in town called dpcrtlmm__EnableTraps, it used to be an internal variable called dpcrtlmm_int__enableTraps, now the modifiers dpcrtlmm_EnableTraps( ), dpcrtlmm_DisableTraps( ) & dpcrtlmm_AreTrapsEnabled( ) are just macros to use it.
    Making this new extern meant the simple module enbltrap.c was redundant, deleted it.
    Limited standard version of the library to 50 blocks, not sure it's turned off in payed for versions, better test that or I could be in trouble!

27th July 2000 - Overlord:
Been doing other stuff mainly, we ran out of coffee, arghhh! Fixed a link problem and Mike agreed to test the library.
Got Alloc()/Free() to update statistics, got code to work to nag user and terminate the program if more than DPCRTLMM_STDBLOCKLIMIT blocks are used in the program, oh and no worries it doesn't nag registered users.

28th July 2000 - Overlord:
Renamed getstats.c stats.c, added support for the default array into CountFlagsInUse( ). It seems Alloc( ) is correctly allocating blocks in the default array because of the NULL array resolver macro, must be Free( ) that is not so aware. I have noticed that GrowBlockArray( ) is relying on the run-time library realloc( ) to accept NULL and pass it on to alloc( ), I'm not sure that this is always supported, I may have to change it. Had to make some changes to isbad.c both IsBad...( ) calls did not resolve array pointers and dpclrtmm_IsBadArrayPtr( ) was so somple it was just asking the safety list. Some functions may try dereferencing NULL pointers without resolving them first so I'll have to find all this instances of this behaviour, all functions which are exposed in the library so the user can call them will probally be affected.
Whoops! vptrap.c is a bit odd, looking for bad blocks is simple, it called IsBadBlockPtr( ), seems logical, and yet the trap on a bad array pointer (the trap above) decides to call the safety list itself, hmm, it might be more advisable to call IsBadArrayPtr( ) of course, changing code again....

29th July 2000 - Overlord:
The trap unfreed arrays routine was always seeing one less block then it should, traced the problem down to a loop which looked at PArr->Count but that count was being changed by dpcrtlmm_Free( ) which the routine was calling as automatic garbage collection on every unreleased block it found. I changed it to while ( PArr->Count ) and took off element 0

30th July 2000 - Overlord:
The change to the loop has (or perhaps not but I suspect it's this) has caused the machine to halt running as a 16-bit program under DOS. I have been trying to debug it all day but it's very hard to see what goes wrong, might have to re-write this loop in the next release, it's too close to release day now, I'll have to say that the library is currently limited to flat builds ie. 32-bit.
Wrote dpcrtlmm_Ver( ), which uses DPCRTLMM_VER as in build.h

31st July 2000 - Overlord:
Hmm, this morning the program no longer halts, if anybody out there decides to read this revision history I'd really like to know what was going on! Not a lot of programming done today, too hot. Only 7 hours until it's midnight in the USA so I should have the demos built by then.

1.0 - 1st August 2000 - Overlord:
Altered the way the version number is reported, it now puts data into a small structure, S_DPCRTLMM_VERSION, removed the define from build.h and made the structure in dpcrtlmm.h I don't think the debug hooks are properly set up I noticed dpcrtlmm_CreateBlockArray( ) at no time lead to the hook chain being processed. Must fix this in all functions I suspect.
Added dpcrtlmm_IsDefaultBlockArray( )
Whoops saw dpcrtlmm_Realloc( ) forgot to resolve NULL arrays, fixed.
Invalid page faults were being caused at the end, changed raise(SIGABRT) to abort( ), then found this wasn't the real cause, the real cause was the library was designed for the static run time library and then example was built for the DLL version. I'll leave that as abort( ) anyway.

1.1 - 3nd August 2000 - Overlord:
Changes between 1.0 and 1.1 are reasonably simple and will only affect people who have used the hooking system properly, I had discovered that Misc0 and Misc1 are unsigned int and are used as pointers sometimes, his works ok in flat model but when segmented the segment is lost, so now they are unsigned longs, the trap DPCRTLMM_HOOK_TRAPENABLEMENT was removed because it never worked, this is because the variable in question is externed so I can't monitor it. I could not support DPCRRLMM_HOOK_INSTHOOK and REMHOOK either, inst hook was too mashed, it even calls itself so hooking would be made very difficult indeed.

24th August 2000 - Overlord:
Created a Makefile for Borland C++

29th August 2000 - Overlord:
The GNU C compiler does not support line continuation characters ('\'), I believed my source was all ANSI C because I use the -A switch on the Borland compiler. To get rid of line continuations I made changes to dpcrtlmm.h and safelist.h. Of course this does not effect the use of the library and it certainly is not worth releasing a new version for. Other porting issues still exist in dpccap.cpp (my C++ encapsulation layer), it seems that GNU are right up to date with ANSI/ISO C++ and have removed except.h, the new version is simply called except, I have not changes these yet as xalloc is still used in this module, it works with warnings on Borland C++. Today I added a clean section to the makefile. Attention programmers! There was a typo 'dpcinstalltrapcallback' in the lazy lazy section of dpcrtlmm.h was actually typed 'dpcinstallrrapcallback', sorry about that!

5th September 2000 - Overlord:
I had two different versions of the isbad.c functions, in the C file they were constant and in dpcrtlmm.h they were not, caused me two warnings on Visual C++ so I corrected the copy on dpcrtlmm.h, so the formal version is constant, people.

November 2000 - Overlord:
A variety of changes, strtstop.c was renamed husk.c but then removed all together and put the startup and shutdown functions from husk.c into dpcrtlmm.c. dpcrtlmm.c contained a #pragma use which caused a warning, I got rid of that and got rid of all copyright notices from dpcrtlmm.c which would end up in the library. Changed all files to GPL license. Virtually rewrote dpccap.cpp, it's now completely ANSI/ISO C++. I'm not so sure of my '\' line continutation problems in September, whatever now, it works. realloc() issues resolved! I found out recently that realloc(NULL, 1024); for example won't neccersarily allocate memory so I've added a run-time warning printed to stderr if somebody tries this with DPCRTLMM. Also if somebody tried to use realloc() as a free-er.

11th December 2000 - Overlord:
Fixed a bug which I saw by reading the source, statistics for block flags were not being returned to the caller from the function dpcrtlmm_GetStats( ). Also got rid of an unneccersary structure. Is was this, forgetting to copy from the structure back to the stats structure which was the bug.
Fixed bug in C++ example (example2.cpp), it an instance of Overlord::TDPCRTLMM_MemManager, which is illegal and probally a mistake many users will make. The real version is global and called MemManager, storage reference dpccap.cpp.
Another bug found: Installed trap handler (which displays the stats again as a test) in example2.cpp, does not work in example1.c. It's unclear as to what differs at the moment.
Managed to get rid of two unneccersary integers in dpcrtlmm.c. Functions can use _libStarted (an internal variable). This would mean the library could be restarted from the same session and save a little memory.

12th December 2000 - Overlord:
Fixed yesterdays bug with the C++ example not running the trap handler after it had installed it when a trap was fired from the library. Turned out that the library was started after the trap handler was installed. My fault. oh yeah, all bugs in DPCRTLMM are my fault, damn.
Seems not even the GNU C compiler's -ansi -pedantic -Wall mode spots all warnings, on Micro$oft Visual C++ it spotted the missing const in parameter 1 of the trap handler. This only effected the examples not the library. Be clear about my meaning of resetting the library in a session by starting it up and shutting it down. I'm not sure everything would be cleared, indeed at the moment it probally wouldn't be and I will fix this later. Also I wouldn't advocate such a thing.

7th June 2001 - Overlord:
From now on read the ChangeLog, which will list all patches since the last release. Important postings (like the above but only if they are of consequence) now go to the mailing list instead.

1st August 2001 - Overlord:
Release DPCRTLMM 1.1.5b as source.

22nd February 2006 - DDRP:
Release DPCRTLMM 1.2.0 as source under new BSD-style license.

4.1 Plans for next version
The internal debug hook executive seems to use a parameter which it could get from the debug info structure passed to it, the hook type is not used and is a waste. This does not affect callers of the library, since the debug hook executive cannot be called from outside.

Swapping to disk should be implemented with an LRU algortihm, based on the flag in block descriptor (unswappable, bit 1) and the block descriptor may have to be extended with more fields to support this.

The library is a bit slow due to an excessive amount of calls to _VerifyPtrs( ), which an internal routine called to check the user's pointers are valid, however it's also called when functions call other functions, even though they already know the pointers are safe, to avoid this I may make an inner shell, where the user calls safety wrappers around the real library and all functions call the inner shell functions and not the entrypoints the users sees.

5.0 Function listings
No documentation is supplied or can be found that I have written for the hidden functions which I believe it is not worth library users trying to access. The file DPCRTLMM.H contains most structures that will be of any interest too. If you have the library source you can read all the other headers to discover the hidden interface if you are that nosey.

dpcrtlmm_InstallDebugHook( )
dpcrtlmm_GetDebugHookChainCount( )
dpcrtlmm_GetDebugHookMatrixCount( )
dpcrtlmm_UninstallDebugHook( )
dpcrtlmm_Alloc( )
dpcrtlmm_Free( )
dpcrtlmm_CreateBlockArray( )
dpcrtlmm_DestroyBlockArray( )
dpcrtlmm_IsDefaultBlockArray( )
dpcrtlmm_Startup( )
dpcrtlmm_Shutdown( )
dpcrtlmm_IsStarted( )
dpcrtlmm_GetBlockSize( )
dpcrtlmm_IsBadBlockPtr( )
dpcrtlmm_IsBadArrayPtr( )
dpcrtlmm_Realloc( )
dpcrtlmm_Calloc( )
dpcrtlmm_InstallTrapCallback( )
dpcrtlmm_RemoveTrapCallback( )
dpcrtlmm_GetTrapCallbackInfo( )
dpcrtlmm_ModifyDescriptorFlags( )
dpcrtlmm_SetBlockLockingFlag( )
dpcrtlmm_IsBlockLocked( )
dpcrtlmm_LockBlock( ) & dpcrtlmm_UnlockBlock( )
dpcrtlmm_ToggleBlockLockingStatus( )
dpcrtlmm_EnableTraps( )
dpcrtlmm_DisableTraps( )
dpcrtlmm_AreTrapsEnabled( )
dpcrtlmm_GetStats( )
dpcrtlmm_GetBlockCount( )
dpcrtlmm_Ver( )
dpcrtlmm_Dump( )

External variables
dpcrtlmm__EnableTraps

5.1 Descriptions of all functions and their use

unsigned int dpcrtlmm_InstallDebugHook(const unsigned short HookType, unsigned int(*NewHookProc)(PS_DPCRTLMM_DEBUGHOOKINFO));
Installs a new debug hook into the internal debug hook chain, see enumerations for a complete list of hook types. The debug hook handler will accept a pointer to a structure full of information which will be of use for debugging, some of this information will be debug hook type dependant. If the hook is installed correctly the function returns 1 (true), if the hook is of a bad type or cannot be installed the return value is 0 (false). Your user defined hook routine should take one of these actions, return 0U if you do not wish for other hooks which follow you in the chain to be notified and called also, return 1U if I may call following hooks.
Do not call the library from your hook routine because this may deadlock the library under threaded builds where mutual exclusion locks do not support recursion.

unsigned int dpcrtlmm_GetDebugHookChainCount(const unsigned int HookType);
Returns the number of hooks installed for this hook type, it is not valid to use DPCRTLMM_HOOK_ALL, for this functionalliy use dpcrtlmm_GetDebugHookMatrixCount( )

unsigned int dpcrtlmm_GetDebugHookMatrixCount(void);
Counts the number of hooks in the entire debug hook matrix, that is the total number of hooks for all types, it is very unlikely that a program will have a practical use for this, I just thought I'd add it really, plus it has a cool name!

unsigned int dpcrtlmm_UninstallDebugHook(const unsigned short HookType, unsigned int(*HookProc2Remove)(PS_DPCRTLMM_DEBUGHOOKINFO));
Remove the said hook, type of hook should be specified in order to find it, although the same hook handler can be installed for many hooks, so DPCRTLMM_HOOK_ALL is allowed.
It's not allowed to uninstall hooks by index, the address of the hook handler must be specified. On success 1 (true) is returned otherwise 0 (false) is returned.

void* dpcrtlmm_Alloc(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const size_t NewBlockSize);
This function is the main allocation function, the equivillant to malloc( ) in the C Run time library. It's prototype seems more complex only because of the block descriptor array. Block descriptor arrays are for organisation purposes, use dpcrtlmm_CreateBlockArray( ) to get one before allocating any blocks of memory or specify NULL to use the array provided by DPCRTLMM (the NULL array is not available if the library is configured --with-no-null-array). The size in bytes of the desired block should be specified. NULL is returned if the block cannot be allocated, check the log to make sure this is what caused the error, any non-zero return is a pointer to the newly allocated block of memory. Debug hooks installed and watching activities with this event will not see allocation requests unless they are valid, I'll fix this by the next version.

void dpcrtlmm_Free(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, void* Ptr);
This function is the main deallocation function, the equivilant to free( ) in the C Run time library. The block descriptor array in which the block was allocated must be specified, even if it is NULL, which represents the default internal library array, if the block did not belong to the array you specify a trap will be executed. If a debug hook is installed on this type of event, remember that it must never dereference the PRelDesc field, the descriptor will have already been released.

PS_DPCRTLMM_BLOCKDESCARRAY dpcrtlmm_CreateBlockArray(void);
Creates a new block descriptor array. A block descriptor array is returned and should be kept. Then during allocations one can use the returned pointer in the first parameter. Don't forget to free all blocks from an array and destroy it with dpcrtlmm_DestroyBlockArray( ) before the returned pointer goes out of scope. I recommend each module, or each major section of a large project has it's own block descriptor array. I also recommend that the main module of the program and any shared memory should use the "NULL" block array, when dpcrtlmm_Alloc( ) asks for a pointer to a block array just supply NULL to use it. Hooking this event is allowed, hooks will be executed whether this function succeeds or fails, check the Success indicator of the debug hook info your hook is passed.

void dpcrtlmm_DestroyBlockArray( PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray );
Destroys the said block descriptor array. NULL is not allowed, the reason is NULL for a block descriptor array represents an internal default array which the user is not allowed to destroy. If the pointer supplied is not in the internal safety list as a valid block array allocated by dpcrtlmm_CreateBlockArray( ), a trap will be executed. There is no return value. Hooks will normally see both success and failure, check the Success entry of the debug hook info.

unsigned int dpcrtlmm_IsDefaultBlockArray( PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray );
In debug hooks and the like, the user may be returned a pointer which does not corrospond to any of the pointers which were allocated, in this case it may be the resolved NULL pointer to the default array because the array is not directly available for comparison use this function to determine if that is the case, it will also report 1U (TRUE) for NULL since this is the unresolved user version. Hooking this function is not supported, if you want it, call me.

void dpcrtlmm_Startup(void);
Before one calls any other functions in this library, the library must be started, this allows various initialization tasks, there are currently no tests applied to ensure the library is started before any other functions are called, it's too time consuming as every function would have to call the test, the behaviour of the program calling the library before it is started is undefined and could end in disaster.

void dpcrtlmm_Shutdown(void);
If one ends the program without shutting down the library the whole point of this library becomes hidden, calling this function does not just allow internal library cleanup but will activate the search for unfreed memory and is the point at which all the leaks will be revealed. I can't make sure the program writer calls this unfortunately, I would recommend setting it up with atexit( ) as soon as the program to starts just incase the program ends from somewhere other than main( ). Personally I keep a routine called GlobalShutdown( ) or whatever in the main module of my programs which lists all global libraries to shutdown and global memory to cleanup etc. and then just register that one function with atexit( ), thus overcoming the 32 entry limit in atext( ). Also my programs contain GlobalStartup( ) which, as it's first function installs GlobalShutdown( ) with atext( ) and then inits all the libraries, I then call GlobalStartup( ) as the first thing to do in main( ) and then I don't have to worry about it.

unsigned int dpcrtlmm_IsStarted(void);
Version: 1.1.6
This function returns nonzero if DPCRTLMM has been started. It is intended to be used by libraries which rely on DPCRTLMM for debugging memory but need to know whether the program which uses it already uses DPCRTLMM in oder to know when to start or stop the library or to leave well alone.

size_t dpcrtlmm_GetBlockSize( PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, void* BlockPtr);
Returns the size of a block which one allocated earlier. The address of the allocated block must be specified also. The size will be returned.

unsigned int dpcrtlmm_IsBadBlockPtr( PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const void* BlockPtr);
Normally calling a function in this library with invalid details will cause a trap, this function is designed to test the validity of the block pointer within the block descriptor array. Calling this function with an invalid array will still cause a trap but if BlockPtr is not valid for a valid block array (supplied) the function returns 1U (true), meaning "yes, this block ptr is bad". If the block is valid 0U (false) is returned. To avoid a trap and test an array pointer call dpcrtlmm_IsBadArrayPtr( )

unsigned int dpcrtlmm_IsBadArrayPtr( PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray );
Verifies the validity of a block descriptor array without causing a trap, the return value is 1 (true) if the array pointer is bad, 0 (false) for a good array pointer. NULL is accepted as a valid array pointer, so is the resolved version but it's unlikely you'll see that unless your code is a hook.

void* dpcrtlmm_Realloc(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, void* OldBlockPtr, const size_t NewSize);
Reallocates a block, equivilant to C runtime library call realloc( ), this is used for resizing a block of memory. The block array must be specified, it can be NULL to specify the default array. The pointer to the block of memory to resize should be specified next, followed by the desired new size. Attempting to resize to 0 is equivillant to dpcrtlmm_Free( ) I rely on your run-time library supporting this bahaviour however, in the next version I want to make sure it will work whether your run-time library supports it or not. The block of memory might be moved by the operating system's memory manager the return value should be used and replaces OldBlockPtr which should now be considered invalid. Don't replace your old pointer with the new one until you are sure it is non-zero, if it is NULL the block could not be resized, most probally because there is not enough continuous free memory. Note that NULL is also returned when a block is freed (providing your run-time library supports it). If a debug hook is installed it will currently only see the event if the reallocation is successful, therefore the hook routine(s) will only see Success = 1.

void* dpcrtlmm_Calloc(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const unsigned int N, const size_t NewBlockSize);
Equivilant to C run time library calloc( ) purely a wrapper around alloc( N * NewBlockSize ) everything else is the same. See dpcrtlmm_Alloc( ) for more information

void dpcrtlmm_InstallTrapCallback( void(*UserCallbackProc)(const unsigned int TrapID, const char* TrapMessage), const unsigned int AsHook );
When a trap is fired, the default behaviour of the program is to put the message on stderr and call abort, this behaviour can be changed to call (for example) the Daybo Logic fatal trap handler (which puts up a blue screen), or something like that. The trap handler takes a constant pointer to a string (const char*), it should not return it should always terminate the program. Perhaps memory errors could be ignored by not doing so but behaviour of the program afterwards might be undefined. If the specified function is installed as a hook it is called when a trap is fired but if it returns the default library handler is called anyway. It just gets a preview. Do not try to remove a handler by passing NULL to this function, use dpcrtlmm_RemoveTrapCallback( )

void dpcrtlmm_RemoveTrapCallback( void(*CurrentCallbackProc)(const unsigned int TrapID, const char* TrapDesc) );
Removes a trap handler which was installed by dpcrtlmm_InstallTrapHandler( ), must pass address so we know you know and are not making a dreadful mistake, there is no return value.

signed char dpcrtlmm_GetTrapCallbackInfo(void);
Sometimes the user of the library has installed a handler for traps instead of the default library handler, the function returns very simple information about it. There are three possible values, the other values won't be used. The three possible values are
-1 for no user trap handler
0 for a handler which is installed as a handler
1 for a handler which acts as a hook.
The important thing about a hook is that it gets preview of the trap but the default one is called afterwards and therefore the hook is expected to return. A handler should terminate the program, if it returns the default handler will not be called, the user handler has replaced it. So if a handler returns the program may continue after a trap which could lead to unpredictable results.

unsigned char dpcrtlmm_ModifyDescriptorFlags(const PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const void* Ptr, const unsigned char* PNewFlags);
This function implements direct flag access (not reconmended except for advanced programmers who have the library source so they know the descriptor flag meanings), casual programmers should use the specialist locking functions or other functions which change the flags but don't actually explictly state that is what they are doing, this flag modifying function is usually used by the library internally and not by the users of the library, I have only made it available to the users because of the possibility of unforeseen circumstances arising, it's not very likely that users will need to use this function.

Pass pointer to new flags for the block, old flags are returned, to get the flags without modifying them, pass NULL as the
pointer PNewFlags. It is feasible that the function could return after a trap if traps are off or a user handler is called for the trap, which returned. It is not possible to tell, if anything is suspected on running the program, at redesign your code so that it does not use a user trap (just comment out the line installing the handler, or don't turn off trapping).

void dpcrtlmm_SetBlockLockingFlag(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const void* Ptr, const unsigned int NewStatus);
Locks or unlocks a block of memory. When a block of memory is locked, it means it cannot be freed or resized, if a trap is fired as a result of the call, the function ModifyDescriptorFlags( ) is mentioned as the location of the trap, so don't get confused!
NewStatus is a boolean, nonzero = true, zero = false.

unsigned int dpcrtlmm_IsBlockLocked(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const void* Ptr);
Simply returns the boolean status 1 if the block specified has been locked or 0 if it has not. When a block is locked, it cannot be freed or resized.

#define dpcrtlmm_LockBlock(pArr, pBlock) dpcrtlmm_SetBlockLockingFlag(pArr, pBlock, (1U));
#define dpcrtlmm_UnlockBlock(pArr, pBlock) dpcrtlmm_SetBlockLockingFlag(pArr, pBlock, (0U));
Both of these functions are simple macros to lock and unlock blocks of memory see dpcrtlmm_SetBlockLockingFlag( )

void dpcrtlmm_ToggleBlockLockingStatus(PS_DPCRTLMM_BLOCKDESCARRAY PBlockArray, const void* Ptr);
If the said block is locked, it will be unlocked and vice versa, not sure if anyone will bother to use this function really

void dpcrtlmm_EnableTraps(void);
See dpcrtlmm_DisableTraps( ), normally only that will be used in a program for the reasons described. The only reason one might want to use this function is for temporary things say like, disable traps, perform actions, re-enable traps. Implemented as a macro, see dpcrtlmm__EnableTraps

void dpcrtlmm_DisableTraps(void);
Normally when the library starts traps are enabled, however let's say that the program is massive, a deadline is coming and it must be released. Although there are a few leaks that can't be tracked down it's stable and users would have nothing to complain about. They won't be too happy if DPCRTLMM keeps outputting all the mistakes at the end. Removing the library may not be feasible if the program is designed intimately around it so the answer is to disable the traps, just call this function once after starting the library to avoid traps. Implemented as a macro, see dpcrtlmm__EnableTraps

unsigned char dpcrtlmm_AreTrapsEnabled(void);
Returns whether traps are enabled or not, 1U means enabled 0U if not. Implemented as a macro, see dpcrtlmm__EnableTraps

void dpcrtlmm_GetStats(PS_DPCRTLMM_STATS PReadStats);
Returns memory statistics for the library, such as, number of allocated blocks, amount of memory used (charge) and peaks, pass a pointer to the structure to read the data, forgetting to pass the pointer will do absolutely nothing. Most of the information can be returned directly from internal counts, most statistics are modified at the time they are changed, for example allocating a block would increment the count of blocks and the amount you added would be added to the total allocation charge. However, because I made it possible to modify the flags of a descriptor I cannot tell how many locks etc. there are, for these I had to implement a nested loop to check the flags of all descriptors, shouldn't reduce performance too much but remember that the function should not be called too often.

unsigned long dpcrtlmm_GetBlockCount(void);
Returns the number of blocks allocated, don't worry I'm not completely mad, the count is stored internally it is not worked out by this function, I don't go mad, looping through all the arrays looking for blocks!

PS_DPCRTLMM_VERSION dpcrtlmm_Ver(PS_DPCRTLMM_VERSION PVerStruct);
Returns library version info, the caller supplies the structure and I return them the same structure, passing NULL does nothing.

void dpcrtlmm_Dump(FILE* Target);
Dumps a listing of all allocated blocks of memory which are currently managed by DPCRTLMM. Target may be any stream which allows writing, for example stdout, stderr or a writable file. The output may be used for knowing what state memory is in at any time without having to wait until a trap for a leak dump.

extern unsigned char dpcrtlmm__EnableTraps;
This can be used to disable/enable traps, while it is off no trap will be executed and if the program makes a mistake it could crash, if can either be modified directly or with the macros dpcrtlmm_EnableTraps( ), dpcrtlmm_DisableTraps( ) and dpcrtlmm_AreTrapsEnabled( ). Generally one will want to switch it off when releasing a program, simple start the program with a call to dpcrtlmm_Startup( ) and then set this variable to 0U, the two valid states are 0U = no traps, 1U = traps will be used. 1U is the default. If you have the absoluetely version (source) this variable is store in dpcrtlmm.c

For the lastest news check the web page and update your documentation & library regularly, remember updates cost nothing. If you have an equiry of a more technical nature you can contact me directly at http://www.daybologic.co.uk/mailddrp/