One of the primary reasons for deploying a Web proxy server is to reduce bandwidth utilization to the Internet. Microsoft Forefront Threat Management Gateway (TMG) 2010 provides very granular control over cache file management as well as cache content.
This article will help you understand how to make Forefront TMG 2010 caching much more useful and help you analyze the cache behavior of Web services to help you configure the Forefront TMG 2010 cache and monitor its relative effectiveness.
How Caching Works
Forefront TMG 2010 can be configured to maintain a cache of Web objects and fulfill Web requests from its cache. If a request cannot be fulfilled from the cache, Forefront TMG 2010 initiates a new request to the Web server on behalf of the client. After the Web server responds
to the request, Forefront TMG 2010 caches the response and forwards the response to the end user.
By default, caching is not enabled on Forefront TMG 2010 because no disk space is allocated for caching. When caching is enabled, an administrator can define cache rules that determine how content from specified sites is stored and retrieved from Forefront TMG 2010’s cache. Cache rules apply to destinations and do not factor in the source of the request. A destination can be specified as a network entity, domain name sets, and URL sets. For any network, if the TMG Client support is enabled or if Forefront TMG 2010 is configured as the
default gateway for internal computers (SecureNET clients), Forefront TMG 2010 may cache requests for them as well.
If a request is allowed by an access or publishing rule, Forefront TMG 2010 analyzes its cache configuration and cached objects to determine whether a request should be served from the cache or retrieved from the Web server. If the object is not present in the cache, Forefront TMG 2010 checks the Web Chaining rules to determine whether the request needs to be forwarded directly to the requested Web server, to another upstream proxy server, or to an alternate destination.
If the request is present in the cache, Forefront TMG 2010 performs the following steps:
1. Forefront TMG 2010 checks whether the object is valid. If the object is valid, Forefront TMG 2010 retrieves the object from the cache and returns it to the user. Forefront TMG 2010 determines whether the object is valid by performing the following checks:
– The Time to Live (TTL) specified in the source has not expired.
– The TTL configured in the content download job has not expired.
– The TTL configured for the object has not expired.
2. If the object is invalid, Forefront TMG 2010 checks the Web Chaining rules.
3. If a Web Chaining rule matches the request, Forefront TMG 2010 performs the action specified by the Web Chaining rule; for example, route the requested directly to a specified Web server, an upstream proxy, an alternate specified server.
4. If the Web Chaining rule is configured to route the request to a Web server, Forefront TMG 2010 determines whether the Web server is accessible.
5. If the Web server is not accessible, Forefront TMG 2010 determines whether the cache was configured to return expired objects. If the cache was configured to allow Forefront TMG 2010 to return an expired object as long as a specific maximum expiration time hasn’t passed, the object is returned from the cache to the end user.
6. If the Web server is available, Forefront TMG 2010 determines whether the object may be cached depending on whether the cache rule is set to cache the response. If it is, Forefront TMG 2010 caches the object and returns the object to the end user.
Forefront TMG 2010 can store objects on the local hard disk, and for faster access can store most of the frequently requested objects on both the disk and the RAM. Cached pages can be stored immediately in memory (RAM) to be accessed by end users requesting the Web content. A lazy-writer or buffered-writer approach is used to write pages to the disk. By default, 10 percent of physical memory is allocated for RAM caching. This approach results in faster availability of content. All additional data is stored on the disk. As objects are cached,
Forefront TMG 2010 adds them to the cache content file on the disk. If the cache content file is too full to hold new objects, Forefront TMG 2010 removes older objects from the cache based on age of the object, size of the object, and how frequently the object is accessed.
The recommendation and properties of the cache file are as follows:
– A formatted NTFS partition must be used and the cache drive must be local. When the cache drive is configured, a cache-content file, Dir1.cdat, is created in the location <drive>:urlcache according to your selections while enabling caching.
– The maximum size for a cache file on a single partition is 64 GB.
– To avoid write conflicts with other Windows activity, you should place the cache file on a physical disk separate from the operating system, Forefront TMG 2010 installation, and page files.
– The size and the location of the cache can be configured on a per-TMG-computer basis.
– Having a large cache is recommended because objects are dropped from the cache when the maximum size if exceeded. However, if the cache is too large (greater than 60 GB), file read-write delays may impact service shutdown and startup.
An administrator can take the following actions for the content to be stored in the cache:
– Specify the type of objects that can be cached.
– Specify how long the objects should remain in cache for objects that do not have a timestamp.
– Specify objects that do not return an OK response (HTTP 200 status code) to be
– Specify URLs larger than the maximum size limit of the RAM cache not to be stored in the RAM. Because objects cached in memory are retrieved faster than those on the disk, it is important to decide what needs to be cached in the RAM versus what needs to be cached on the disk to prevent excessively large objects from filling the RAM cache. The default maximum size limit is 12,800 bytes.
Forefront TMG 2010 provides caching in two different scenarios. Forefront TMG 2010 caches objects when an internal user makes a Web request to the Internet. This type of caching is known as forward caching. In a Web publishing scenario, Forefront TMG 2010 provides cached content from the internal Web server published by Forefront TMG 2010 to the external users. This type of caching is known as reverse caching.
Forward caching happens when a user located on a network protected by Forefront TMG 2010 requests Web content through Forefront TMG 2010. Forefront TMG 2010 makes a decision on how to serve that content depending on its availability in the cache.
If the content is available in cache, the process is as follows:
1. The user requests a Web page on the Internet.
2. Forefront TMG 2010 intercepts this request and determines whether the content is
available in cache (RAM cache or disk-based cache).
3. If the content it available, it is returned to the user in accordance with the cache
The settings determine whether only valid objects can be returned to the
4. The content is then moved to the in-memory cache (RAM cache) as per the cache
After a period of time, if the content is no longer being requested regularly
(is no longer “popular”), Forefront TMG 2010 copies this content from RAM to
based cache and flushes it from memory. If another user requests the content
stored in disk-based cache, Forefront TMG 2010 returns it to the RAM cache.
If the content is not available in cache, the process is as follows:
1. The user requests a Web page through Forefront TMG 2010.
2. Forefront TMG 2010 intercepts this request and determines whether the content is available in cache (RAM cache or disk-based cache).
3. If the request is not present in cache or has expired, Forefront TMG 2010 forwards the request to the Web server.
4. The Web server returns the requested information and, based on the cache settings, Forefront TMG 2010 caches the object in its RAM cache where frequently requested content can be stored for fast retrieval.
5. The content is returned to the end user.
6. After a period of time, or if the content is no longer frequently requested, Forefront TMG 2010 copies the content from RAM to disk cache and flushes it from the memory, leaving the only available copy on disk. It is only returned to RAM cache if another user requests the content.
Reverse caching happens when Internet users request content from a Web server published by Forefront TMG 2010. The process is as follows:
1. The Internet user requests a Web server published by Forefront TMG 2010.
2. Forefront TMG 2010 intercepts the request and determines whether the content is present in cache. If the content is not available in cache or has expired, Forefront TMG 2010 forwards the request to the published Web server.
3. After the published Web server responds, Forefront TMG 2010, as per its cache settings, stores the content in RAM cache where frequently requested content is stored for fast retrieval.
4. Forefront TMG 2010 returns the content to the Internet user who requested it.
5. After a period of time, if the content is no longer frequently requested, Forefront TMG 2010 copies this content from RAM to disk-based cache and flushes it from the memory.
Caching behavior in Forefront TMG 2010 can be made granular and more configurable by creating specific cache rules. Each cache rule allows for specific types of content to be processed in different ways, depending on your needs.
By default, when caching is enabled, a default cache rule is created that caches objects based on its default settings. You can create additional caching rules based on your specific caching needs and requirements. Each rule created can contain the following customizations:
– Specify how content retrieved by the rule is returned to the end user. Objects that are valid or expired can be returned and requests can be routed to the Web server if no cached object is found, or they can be dropped entirely.
– Specify type of content to be cached. By default, objects are only stored in the cache when source or request headers in the HTTP request indicate caching. You can define content that should not be cached or content that should be cached even if the source or request headers do not indicate caching.
– Specify a maximum size for the objects that the rule caches.
– Specify whether SSL responses should be cached. This does not apply to forward caching for sites exempted from HTTPS inspection.
– Enable caching of HTTP objects.
– Specify how long objects should remain in cache. Unless the source specifies an expiration time, HTTP objects remain in cache according to the TTL setting in the rule. The TTL is based on a percentage of time that has passed since the object was created or modified.
– Enable caching of FTP objects and specify a TTL to indicate how long FTP objects should remain in the cache.
You can see some of these settings on the Advanced tab when creating a cache rule (see figure).
Caching Compressed Content
HTTP compression reduces file size by using algorithms to eliminate redundant data. Most common Web-related file types can safely be compressed. HTTP compression uses the industry standard GZIP and Deflate algorithms. These algorithms compress static files, and
optionally perform on-demand compression of dynamically generated responses before sending them over the network. These same algorithms are again used to decompress the
static files and dynamic responses on an HTTP 1.1–supported client. A client that is configured to use HTTP 1.1 may request compressed content from a Web server. Web servers indicate in their responses whether they support compression.
HTTP compression in Forefront TMG 2010 is a global HTTP policy setting. It applies to all HTTP traffic that passes through Forefront TMG 2010 to or from a specified network or network object, rather than to traffic handled by a specific rule. HTTP compression is provided
by two Web filters:
– Compression Filter This filter is responsible for compression and decompression of HTTP requests and responses. This filter has a high priority, and is high in the ordered list of Web filters. This is because the filter is responsible for decompression. If you choose to enable inspection of compressed content, decompression must take place
before any other Web filters inspect the content.
– Caching Compressed Content Filter This filter is responsible for caching of compressed content and serving a request from the compressed content in the cache.
This filter has the lowest priority, and is low in the ordered list of Web filters, because caching can take place after all other filters have handled the content. Caching and compression together provide a more efficient way to serve compressed requests. Content is cached on Forefront TMG 2010 in one of the following ways:
– Compressed Content is requested in compressed format and cached in compressed format.
– Uncompressed Content is requested in uncompressed format and cached in uncompressed format.
– Uncompressed and incompressible If an end user requests compressed content and it arrives at the cache uncompressed, it is stored in the cache as incompressible.
The next time the request for the same compressed content is received, Forefront TMG 2010 will recognize that the content is incompressible and serves it from the cache uncompressed rather than from the Internet. Content that is inspected is also stored as uncompressed.
Compression settings applied to cache content are persistent. Therefore, if you want compression configuration changes to be reflected in the cached content, the cache on Forefront TMG 2010 needs to be cleared (this is discussed later in this chapter). Another
important thing to remember is that when inspecting incoming compressed content,
Forefront TMG 2010 Web filters must have access to the decompressed content. When compression is enabled, content is cached in compressed form. The Caching Compressed Content Web filter is responsible for decompressing the content so that the other Web filters can inspect the decompressed version of the cache content. When Forefront TMG 2010 receives a request for the cached content, the Compression Web filter recompresses the content before sending it to the end user. This in turn increases the amount of time it takes to transfer the content from Forefront TMG 2010 to the user.