Understanding Web Proxy Configuration

The “System Proxy”

Unlike on other systems like Mac OSX, Windows doesn’t really have the concept of a “system” proxy. Mostapplications respect the WinINET proxy setting, but a few do not.

• Firefox respects the WinINET setting only when configured to “Use System Proxy Settings” in its Network configuration.
• WinHTTP-based applications may or may not respect the WinINET setting (see below).
• Applications built on the .NET Framework typically will adopt the system proxy only when they start (and do not detect changes at runtime). Also, the proxy settings can be overridden by the app.exe.config ormachine.config files.

Historically, most applications were built on WinINET and thus respected its proxy settings directly, but today more and more applications are built on WinHTTP (or a descendent architecture like BITS) and System.NET and thus may require additional configuration.

Settings are Per-User

WinINET proxy settings are typically per-user rather than per-machine. This means that individual (even non-admin) users can set their own proxy settings without impacting the proxy settings of other user-accounts. WinINET can be configured to apply proxy changes on a machine-wide basis by creating a registry DWORD namedHKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\ProxySettingsPerUser with value 0. Subsequently, changes to the proxy settings can only be made by elevated applications running with Administrative permissions.

Configuring the Proxy

The WinINET proxy setting can be configured in Internet Explorer by clicking Tools > Internet Options > Connections. On the connections tab, select a Connection and click the Settings button. For most users, the proxy is configured by pressing the LAN Settings button at the bottom of the dialog:

Settings Precedence: Part 1: While the proxy settings for the Dialup/VPN/RAS connection and the LAN settingsbutton can be configured independently, internally WinINET will use the settings from any active/connected dialup/VPN/RAS Connection. Otherwise, the proxy setting specified in LAN settings is used. Many users incorrectly expect WinINET to first determine which network interface is used to establish a given connection before selecting a proxy; it doesn’t work like this.

Proxy Configurations

When you configure your proxy settings, you’re presented with the following dialog box:

Settings Precedence: Part 2: The settings in this dialog box are presented in the order of their precedence. First, the Automatically detect settings option is consulted, next the automatic configuration script option is evaluated. If neither of those options is enabled or neither specifies a proxy to use, only then are the fixed proxy server settings consulted.

Automatically Detect Settings

When Automatically Detect Settings is enabled, Internet Explorer performs a process called Web Proxy Auto-Detection (WPAD). This process consists of two phases:

1.       Issuing a DHCP INFORM query, asking DHCP to provide the URL of a proxy script

2.       Searching for a server in the current domain prefixed with WPAD (e.g. wpad.corp.contoso.com)

These operations may or may not be performed in parallel, depending on the network stack. Some browsers (e.g. Firefox) only undertake DNS detection and skip DHCP detection. For performance reasons, some clients may cache the result of the detection on a per-network basis ("SmartWPAD"), skipping the WPAD process in future sessions when connected to networks known not to be using WPAD.

If the client is able to automatically detect a proxy script's URL, it will then attempt to download and use that proxy configuration script.

Automatic Configuration Script

The user may also directly specify the URL of a proxy configuration script using the second checkbox in the dialog. The URL field below points directly at the target script (e.g. http://proxy.contoso.com/proxy.pac).

Proxy configuration scripts, whether discovered via WPAD, or manually specified by the user, are JavaScript files that expose at least one function FindProxyForURL(url, host). This function is called by the browser each time that it needs to decide where to send a given request. It returns a string which is either:

1.       "DIRECT" indicating that the request should bypass the proxy

2.       "PROXY PrimaryProxy:8080; BackupProxy:81" indicating that the request should be forwarded to the proxy PrimaryProxy on port 8080, unless that server is unreachable in which case the request should be sent to BackupProxy on port 81.

WinINET also offers support for another function FindProxyForURLEx, an extension which supports IPv6-aware proxy scripts.

NOTE: Proxy Scripts Impact the Zone

Warning: One sometimes surprising aspect of proxy scripts is that they impact the Internet Explorer Security Zone determination. To summarize my long MSDN article and blog post, by default, if a proxy script is in use and returnsDIRECT, the target site will be mapped to the Local Intranet Zone.

NOTE: File://-based Proxy Scripts Deprecated

In IE11, the WinINET team has made some changes to attempt to improve interoperability of proxy scripts.

Internet Explorer 10 and other WinINET-based clients would use the specified proxy script. However, WindowsUpdate and any other application built on a WinHTTP-based network stack would silently ignore the specified proxy script because WinHTTP has never supported the use of the file:// protocol as the origin of a proxy configuration script. Note: In November 2012, the .NET Framework was changed to use WinHTTP for PAC processing, which means that, by default, .NET Applications will not support file://-based proxy scripts any longer either.

In Internet Explorer 11, the WinINET team has disabled WinINET’s support for file:// based scripts to promote interoperability across network stacks. Corporations are advised to instead host their proxy configuration scripts on a HTTP or HTTPS server. As a temporary workaround, this block can be removed by setting the following registry key:

Key:   HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type:  REG_DWORD
Data:  1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix your WinHTTP-based applications.

NOTE: File://-based Proxy Scripts Require “Unhealthy” Syntax

While you shouldn’t use a file://-based proxy script, if you do so, IE requires that you use the legacy “unhealthy syntax” for file URIs in the proxy configuration dialog.

As you can see, the “Healthy” syntax is incorrectly parsed as a relative URL.

The legacy “unhealthy” syntax works properly:

But, to reiterate, don’t use file:// to host your proxy script.

NOTE: IE11 Deprecates Non-Silent Auth when downloading Proxy Script

When downloading a proxy configuration script, it’s possible that the server will demand credentials from the client application. A problem arises if the server requests credentials using HTTP BASIC or HTTP DIGEST authentication, as these authentication methods require that the user respond to a credential prompt dialog box. (In contrast, NTLM and Negotiate authenticate silently).

Many applications cannot handle showing prompts as a part of this workflow and will fail silently. To promote interoperability, IE11 also blocks the use of Proxy Configuration scripts that require authentication. As a temporaryworkaround, this block can be removed by setting the following registry key:

Key:   HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type:  REG_DWORD
Data:  1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix applications which cannot show UI prompts.

Generally speaking, you should attempt to migrate away from BASIC/DIGEST authentication both for access to the proxy script and for access to use the proxy itself. As I’ve noted previously, Manual Proxy-Authentication breaks many applications.

Fixed Proxy Configuration

Beyond automatic proxy detection, the user may manually configure “fixed” proxies. By default, the simple UI is shown:

…but clicking the Advanced button shows more options:

You can either specify a different proxy server/port for each protocol, or you may use the default option which applies the same proxy server to all protocols.

The Exceptions box allows you to specify what hostnames are configured to bypass the proxy. Beyond hostnames, it supports two special tokens:

• <local> – Matches “plain” hostnames (with no embedded dots)
• <-loopback> – Unmatches localhost, 127.0.0.1 and other “loopback” addresses.
• WinINET defaults to automatically bypassing the proxy for loopback addresses; this token disables that behavior.
• This token impacts WinINET and current versions of WinHTTP; the .NET Framework simply ignores it.

Note that Internet Explorer only supports SOCKSv4.

Setting the Proxy Programmatically 

Fiddler and some other applications attempt to programmatically adjust the WinINET proxy setting. Rather than attempting to “poke” the registry directly, the proper way to update the proxy setting is to use the InternetSetOption API.

Using the API is straightforward, whether you’re using C/C++ or .NET.

IE10+ Regression on System Shutdown

As a part of Internet Explorer 10, proxy configuration was centralized in an existing system service (still labeled “WinHTTP Web Proxy Auto-Discovery Service”) so that proxy configuration code no longer needs to run in every process using WinINET (e.g. each IE tab is in its own process).

Unfortunately, as a part of this update, a regression was inadvertently introduced. Code that attempts to change the WinINET proxy setting during a system shutdown (like Fiddler’s shutdown code, for instance) will find that the APIs return success, but upon restart you will see that the proxy settings change was discarded. No fix for this issue (also present in IE11) is yet available. A possible workaround for this problem would be to create an out-of-proc COM server and run a method (DllHostKeepAlive) that doesn’t return until the client calls a shutdown method (CancelDllHostKeepAlive). This would open a DLLHost that can be used to adjust the proxy settings during shutdown.

CONNECT Bug: https://connect.microsoft.com/IE/feedback/details/838086/internet-explorer-10-11-wininet-api-drops-proxy-change-events-during-system-shutdown

WinHTTP Considerations

Thanks to Eric Loewenthal, owner of WinHTTP's proxy code, for providing the expertise for this section

The WinHTTP stack was designed for use in services and other contexts where WinINET would be inappropriate (e.g. because WinINET may show UI or not scale well for unattended use). WinHTTP offers a limited subset of proxy support-- generally speaking, the calling code is expected to set the proxy configuration settings directly on the session or request objects. The user may attempt to configure the "default" proxy for WinHTTP using theproxycfg.exe (XP) or netsh.exe (Vista+) tools, but these tools only support fixed proxy settings (not autodetection or PAC script URLs) and WinHTTP-based applications may or may not use those settings.

The configuration tools were primarily intended or WinHTTP apps that are running as System (without a user to impersonate). 

Well-written WinHTTP apps will:

1.       If not running in a user's process, impersonate a user

2.       Call WinHttpGetIEProxyConfigForCurrentUser

3.       Plug the returned settings into WinHttpGetProxyForUrl

Sample code for this technique can be downloaded from MSDN. 

Note: WinHTTP will not process a proxy script if the Content-Type response header is not "application/x-ns-proxy-autoconfig" or the URL's file extension is not one of .js, .pac, or .dat. Also, remember that WinHTTP will not use proxy script files from file:// URLs, on any version of Windows.

Source: http://blogs.msdn.com/b/ieinternals/archive/2013/10/11/web-proxy-configuration-and-ie11-changes.aspx