File System Memory Usage

This topic explains the mechanisms involved in internal memory management. This explanation is intended only to help in system configuration and tuning, and to understand the various performance counters explained in D3 User’s Guide.

A D3 table is composed of various Windows files, as detailed in Hashed Table Structure. These various Windows files are mapped in the addressing space of the file system server by pages of 64 KB, or less if the total file size is less than 64 KB. Memory mapping of files makes access to Windows files very efficient because it eliminates the double buffering associated with classic I/O. However, most Windows systems do not allow mapping more than 2 GB of memory, while some allow 3 GB of memory (64 bit versions of Windows allow 4 GB). Since a D3 application can access very large amounts of data, it would be impossible to map all files simultaneously; therefore, some level of memory management is required.

The server maintains a table of all the pages currently mapped into memory. This table allows mapping approximately 1.5 GB of memory at any given time. The maximum size of the mapped files can be limited by the New Disk Size configuration setting (see D3 Settings). When a new page is needed, the system unmaps the least used page and maps the requested page. This gives the system an infinite virtual space. Note that the file that is partially unmapped remains opened. Therefore, when the unmapped page needs to be brought back in, there is no performance cost in reopening it. Also, the unmapped page is not necessarily removed from memory; it remains in the Windows file cache and is readily available if requested.

NOTE	There is no memory management of the data itself. All D3 data is located in the Windows file cache, thus allowing smooth coexistence with other applications.

The 64 KB page size is a compromise between direct access and sequential access. One map operation brings 16 frames (for a 4 KB frame size table) in the virtual addressing space, but not necessarily in physical memory, which helps in sequential operations.