Overcoming the Windows 2GB Caching Limit

by Mike Richardson

For years Windows developers have struggled to exceed the 2GB per-process memory limit, especially when attempting to architect large-scale caching systems. The .NET CLR and Java’s VM also suffer from this limitation when running on 32-bit systems. Common solutions, such as implementing popular COTS solutions like TimesTen or NCache, do not solve the problem either. In 32-bit systems, it is simply impossible to store a very large amount of data in a process. However, there are a number of options available, including a nice addition to the .NET 2.0 Remoting API, which could make designing such a system more feasible. First, I will illustrate the background of the problem followed by various solutions. The goal of this post is to present an architectural pattern to combat the 2GB limit in Windows and other runtime environments.

Typically, a Windows process running in the 2003 operating system environment can access up to 2GB of address space. This memory is split between actual physical memory and virtual memory. Basically, the more processes that are running on the system, the more memory will be committed to reach the full 2GB address space.

When memory consumption approaches the 2GB limit, the paging process increases and performance begins to degrade. In order to improve performance and memory utilization, Windows memory managers use a concept known as PAE (Physical Addressing Extension) on Intel chips, which basically reduces the need to swap the memory of the paging file. The client program is not aware of the actual memory size. Rather, all the management and allocation of the memory addressed by PAE is handled independently of the program accessing the memory. In order to enable extended memory support and utilize PAE, the /3GB switch must be enabled in the boot.ini file, which is illustrated below:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(2)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /PAE

Even though PAE is enabled, the operating system is still based on 32-bit linear addressing. However, multiple processes can benefit from the increased memory because they are less likely to encounter physical memory restrictions and begin paging. Additionally, Windows applications can be modified to use the AWE API in order to allocate memory outside of the applications process space, essentially bypassing the 2GB constraint.

AWE is a set of application programming interfaces (APIs) to the memory manager functions that enables programs to address more memory than the 4GB that is available through standard 32-bit addressing. AWE enables programs to reserve physical memory as non-paged memory and then to dynamically map portions of the non-paged memory to the program's working set of memory. This process enables memory-intensive programs, such as large database systems, to reserve large amounts of physical memory for data without having to be paged in and out of a paging file for usage. Instead, the data is swapped in and out of the working set and reserved memory is in excess of the 4GB range. Additionally, the range of memory in excess of 4GB is exposed to the memory manager and the AWE functions by PAE. Without PAE, AWE cannot reserve memory in excess of 4GB.

Now let’s state the problem in terms of developing a large, highly scalable caching solution…

The purpose of caching in a Windows application is two make infrequently changing data readily available to an application. The actual cached data is usually co-located on the application server in order to increase performance. In addition, by enabling the application tier components to access data directly without having to make a database connection increases reliability and eliminates a single point of failure. Enabling caching on the application tier essentially creates a two-tier distributed application. Once data is retrieved from the database or other source, it can be cached so that excessive calls can be avoided. Most popular caching API’s (including .NET) also support sliding expiration of data items, callback functions to respond to underlying changes in the data store, and provide cache “controllers” for implementing common LRU and MRU algorithms.

If your application must cache data in a process larger than 2GB, it can be accessed in .NET by using unmanaged code and the AWE API. However, one major issue still exists and must be dealt with – performance. For example, if you use PAE and AWE to cache a table that is approximately 6GB in size, then accessing that data in any caching system will obviously degrade performance. The paging of memory will no doubt reduce performance and the disk seek times to locate the necessary data can also be quite long. Note that when I say you must cache more than 2GB of data, I am assuming that the data itself is already optimized. For example, I am assuming that you are using efficient lookup structures in your cache like StringDictionary and not just a bunch of ArrayList objects, or that you have already performed string interning to decrease the overall memory footprint.

The next step to effectively reduce the size of the data is to mimic the functionality of popular caching systems like NCache. For example, you can then split the data up into logical buckets, provided that you create some form of controller which “understands” where to locate the appropriate data. The trickery comes in when the need arises to access data between buckets. In .NET, the only method available to aggregate data between processes is to use .NET Remoting. If you are using the .NET 1.1 API, you may be out of luck if you are seeking the ultimate in performance; even with the binary formatter the fastest access method would be TCP/IP, which is not quite fast enough. The sheer number of CPU cycles needed to cross process boundaries would grind the caching system to a halt. Fortunately, the .NET 2.0 Remoting API introduced a new Remoting channel for IPC known as a named pipe.

By implementing IPC using a named pipe in addition to a “smart” cache controller and some LRU and MRU algorithms, it is possible to achieve high performance in systems which need to access more than 2GB of memory.

Of course there is one last point that I forgot to mention. If your project has a liberal budget, you can solve all your caching problems simply by buying a nice, high-end 64-bit machine.

3 Comments

simon_hibbs
2006-03-13 04:32:05
Enabling /3GB in the executable
>In order to enable extended memory support and
>utilize PAE, the /3GB switch must be enabled in
>the boot.ini file, which is illustrated below:


You also need to flip a bit in the executable to take advantage of this. This gives the application process acces to 3 GB of memory, reducing OS space in that address space to 1 GB. This can cause resourec shortfalls and a lack of file handles in the OS in some situations so you need to do some thinking, and testing before going this route.


Some MS pages refer to a utility called imagecfg.exe you can sue to switch this on in the executable, but I can't find this in any of the resources MS points you at (resource kits, SDKs etc). these is a utility in Visual Studio though called editbin.exe that can do the job.


Simon Hibbs

michaelrichardson1
2006-03-13 06:39:27
Enabling /3GB in the executable
Simon,


Thanks for the clarification. Also, let me state my main motivation behind the post, because I don't think it was very clear (now that I know people actually read my posts). I would NOT recommend using the AWE API from .NET. Although it ultimately depends on your performance requirements, I am simply making the assumption that if you are worried about caching then you are probably concerned with performance as well. The obvious exception is if you are developing for a 64-bit server, in which case performance is quite good.


If you need to cache large amounts of data, I would suggest a distributed approach as I described in my post. It will give you the ability to configure the amount of memory "buckets" in the application, more control over usage algorithms, etc. It would not be unlike Google's search architecture.


It is going to be interesting to see how all the caching vendors respond to the 64-bit movement. Although they usually include a bunch of extra features like tying to data stores and flushing based on time, their main claim to fame is managing data.


One more quick point - another solution that I did not mention (if you have the money of course) is to use solid state disks for your cache. Access times would be much faster and you could probably even implement some type of ISAM like solution.


Thanks,
- Mike

Richard Taylor
2006-03-16 23:07:12
A couple of comments. Firstly, we have had a lot of trouble with memory mapped files and running out of page table entries, particularly on older versions of Windows.


Secondly, I would advise anyone who wants to access large memory to go 64-bits. A new system you buy now will run 64-bits, and the time and trouble you save from going this route will save you much more than the hardware cost.