Removing OpenSSL

Reducing Load Balancers exposure to OpenSSL Heartbleed

 

Overview

In this article, you will learn about reducing load balancers exposure to OpenSSL Heartbleed and what steps you must take to reduce those risks. The first step is to patch your load balancer. There are a number of leading vendors including Cisco, F5, Juniper, Citrix, Barracuda that have issued advisories. These advisories should be strictly followed. The next step is to closely evaluate whether your load balancers are prone to future catastrophic flaws. Advocate that your enterprise should stop using OpenSSL-based load balancers, now and in the future. Purify your architecture through dedicated SSL off-loaders built ground-up for security processing. Inquire why load balancers allowed your company to remain exposed to OpenSSL Heartbleed for 2 years without any advisories and patches.

As IT departments recover from the initial shock waves of Heartbleed bug and quickly apply OpenSSL patches to affected components of their infrastructure, there is bound to be a long term assessment on the robustness of critical components in the IT architecture. Responsible security practitioners and network architects will not let this one pass by with a “patch and forget” approach.

 

Why do we use load balancers with SSL?

In the mid-1990s, load balancers became popular for distributing network traffic across a farm of servers or routers/firewalls. By the late 1990s, Layer 7 load balancers were introduced to the market as HTTP (web) traffic began to dominate the Internet. These Layer 7 load balancers were initially blind to SSL traffic since they could not peer into encrypted traffic. Initially, to solve this problem, Layer 7 load balancers were deployed with SSL off-loaders.

SSL off-loaders were dedicated network appliances whose sole function was to terminate SSL traffic and pass unencrypted data straight to the farm of servers or to pass it through the Layer 7 load balancers. These Layer 7 load balancers would make intelligent load balancing decisions based on cookies, URIs, session IDs etc; attributes that would appear as part of the HTTP header. The sole function of SSL off-loaders was to accelerate SSL traffic by terminating SSL traffic at the edge at very high rates; nothing else.

As load balancers became a commodity, they started moving up the stack and consuming more application specific functionality. They cheapest way for load balancer to integrate SSL functionality was to take a popular open source SSL stack, OpenSSL, and bolt it onto their C-based load balancer code. This path provided customers with a unified solution and the load balancer vendors with a quick time to market, but in the process, exposed enterprises to significant vulnerabilities. A security-first mindset was not followed.

 

Typical load balancer deployment before Heartbleed

A load balancer is a reverse proxy gateway that distributes web traffic across a number of servers. It serves two primary functions:

  • Improve the overall throughput performance of applications by reducing the load on servers tied with the applications.
  • Provide high reliability and availability of servers and applications.

Load-Balancer-without-Forum-Sentry

A typical deployment scenario is shown above. A client browser or application makes an SSL request that is terminated at the load balancer. Once a URL, cookie, or session-based decision is made by the load balancer, the traffic is then load-balanced between web server #1 and web server #2 by establishing SSL connections to both servers. As shown in the figure above, the Heartbleed vulnerability may exist in the web servers, or load balancers. The load balancers also have a long term architectural issue: they are developed in C.

As soon as the Heartbleed OpenSSL vulnerability was revealed, the first scramble was to patch all web servers. This has been a relatively quick task since patching a farm of web servers with a new version of OpenSSL library can be scripted, automated or done manually.

However, this step only ensures that sensitive information from the web servers are not compromised. With the load balancer left un-patched, sensitive information from both web server #1 and web server #2, as shown in the figure above, are now available for a Heartbleed attack on the load balancer.

 

Current load balancer risks with OpenSSL

Majority of the load balancers rely on OpenSSL library for SSL processing. For example, see:

  1. Cisco product list
  2. F5 Product list including Big-IP
  3. Radware AppDirector
  4. Barracuda
  5. Citrix Netscaler

It is interesting to note that a couple of the load balancing vendors have declared victory over this vulnerability and flat out stated that they are not exposed given that they hadn’t bothered to stay current with the OpenSSL libraries. The older libraries have vulnerabilities of their own including BEAST and CRIME that required OpenSSL upgrades. A quick investigation of OpenSSL vulnerabilities shows no less than 8 buffer overflow flaws over the last 10 years. It is almost shameful now that the ones that did not patch are claiming that they are more secure than those that did patch. As if negligence is a badge of honor. The ones that did patch claim their superior security architecture that remained vulnerable for over 2 years without notice. It brings to light the lack of focus on security in leading vendors selling load balancer with SSL and other security features.

All artifact that may now have been protected by rapidly patching web servers still remain exposed when they are managed by a central load balancer. Following are the potential risks when performing SSL protocol processing in a load balancer using OpenSSL:

The OpenSSL has broad richness in terms of SSL protocol processing features but has been hampered by memory management related bugs. OpenSSL is written in C language, known for not being a safe language when it comes to handling memory functions. In turn, it has exposed a weakness in the architecture of a load balancer that is one of the most mission critical pieces of an IT infrastructure.

 

Recommend deployment architecture after Heartbleed

Once the dust has settled down, web server, load balancers and other IT components using Heartbleed have been patched, the application security architecture of an enterprise has to be closely evaluated.

Load-Balancer-with-Forum-Sentry

The figure above shows the recommend deployment change. This is the deployment topology that all Forum Systems customer follow and not a single one has had to worry about patching networking component in their infrastructure. Load balancers are specialists at load balancing, but their inherent C-based architecture with OpenSSL will always remain exposed. An application gateway, such as Forum Sentry should be deployed behind a load balancer for SSL and other content-level security processing.

 

Advantages of removing SSL from load balancers

Here are some of the primary advantages of removing SSL from load balancers and offloading security processing to application security gateways such as Forum Sentry:

  • Forum Sentry does not use OpenSSL for SSL termination, initiation, or key generation.
  • The load balancer can focus on load balancing without worrying about security functions
  • Since Forum Sentry SSL and security processing is not C-based, it not prone to buffer overflow
 

Conclusion

Heartbleed unequivocally proves that load balancers are not designed with a security-first mindset. They are developed using C programming language that leverages OpenSSL and will continue to remain prone to such flaws. Companies that avoided Heartbleed had a sound architecture where every component performed a function that it was designed to do well. Firewalls did IP-level filtering, load balancer managed traffic, and application security appliances processed SSL termination. Companies that bought into the all-in-one appliance vision ended up risking all their corporate data. It’s time to stop asking your load balancers to do your SSL processing.