Facebook Twitter Linked In Google+ Vimeo YouTube Flickr     My Account     icon_home_white_small.png  Public Home     Connected Community Lobby     Contact Us

Public Sample of the
Online Ministries Community Blog

Apr 09
2014

The Heartbleed Vulnerability: Did You Get Whacked; If Not By the Bug, the Hype?

Posted by Bill Anderton

This week, starting April 7th, the press has been widely discussing the Heartbleed vulnerability (more formally known as CVE-2014-0160) that exists in certain versions of the OpenSSL package used by some online services and hosting providers for their SSL and TLS services.

When the vulnerability was announced in our computer security channels, as we do with all newly-discovered vulnerabilities, we formally assessed our risk to CVE-2014-0160.

Fortunately, this gave us a slight head-start on Heartbleed before the news hit the mass-market-oriented popular press.

The TransformingTheChurch.org websites and our servers do NOT have this vulnerability! We do not use OpenSSL for our SSL and TLS services. Below is the Netcraft report for https://www.transformingthechurch.org.

/files/heartbleed_report.png

You may be susceptible to Heartbleed at other sites and other devices you use, so we recommend you take the vulnerability seriously. Not only might you be vulnerable through websites you visit, but the websites you manage might also be vulnerable to CVE-2014-0160. If you are a webmaster, you have a little bit of work to do!

Lots of Hype and Some Bad Reporting

Heartbleed has been described as "one of the most serious security problems to ever affect the modern web" and "on a scale of 1 to 10 in importance, it's an 11!" and "two-thirds of all of the servers on the global Internet are vulnerable!"

In other words, "The sky is falling, the sky is falling!"

Yes, this is still an important bug, but some of the reporting has been over the top.

There has been a lot of reporting on Heartbleed and, unfortunately, some of it has been bad reporting if not downright inaccurate. Most of the electronic media, particularly the cable news channels, aren't particularly strong when reporting deep technology issues; obviously, no geeks on their reporting staff! This is also true of many other new outlets too, unfortunately. Many were simply ill-equipped to judge the facts, report this story in its correct context and give its correct weight. Also, on Monday, there wasn't a lot of hard data to report, and wild speculation and conjecture got reported as fact.

Now, after a couple of days of working in the trenches and a bit of time to survey the potential damage, I thought this was a good opportunity to set the record straight with what we know now.

What is Heartbleed?

In addition to the HTTP web protocol for the transmission of web content between software clients (web browsers) and servers, there is a secure version called HTTPS that encrypts all communications between clients and servers. This is how web-form data submitted from web pages and credit card transaction are made secure from eavesdropping, interception and exposure of things like credit card numbers, PINS, passwords, etc. With everything encrypted, even if someone intercepts the payload, it does the eavesdropper no good because they can't practically decrypt the payload if it is strongly public/private-key encrypted (in our case, a 256-bit key.)

At TransformingTheChurch.org, as a standard best practice, we use secure transmission for ALL pages by using the HTTPS protocol, so it was important for us to look into any reported vulnerability. And, yes, we were very happy we weren’t impacted and a major salute to our software development team!

The "HTTPS" prefix that you see in your browser’s address field (usually accompanied with a lock icon) indicates that the encrypted communications protocol is being used for pages/transaction between your web browser and the web server you are communicating with at the time. The HTTPS uses the "secure sockets layer" (SSL) protocols.

The SSL protocol and its sister, TLS, are provided by a software code module typically either internally developed, licensed from a specialist software developer or through the use of open-source software.

Heartbleed is a vulnerability of a specific open-source software library called OpenSSL. People who licensed other software libraries are not vulnerable to Heartbleed.

To be precise, the vulnerability is not in the SSL/TLS protocols themselves but their implementation in the software library of one open-source development project (more about this later.)

In fact, not only is the vulnerability limited to just the OpenSSL software, it is further restricted to two relatively releases of OpenSSL; specifically, versions 1.0.1 and 1.0.2-beta.

Unfortunately, and the reason for the hubbub in the press, OpenSSL is widely used in many web services and products, AND this vulnerability hasn't been previously detected for 28 months!. Because it is open source software, it doesn’t require the payment of software license fees; it is free. Many developers use OpenSSL to reduce the costs of their services and equipment. OpenSSL is often used on Linux operating-system servers using the Apache web server software (also open source software.) The Linux operating system is used to run a lot of websites, particularly in the value-based hosting arena.

It has been reported that perhaps two-thirds of all web servers, both large and small, use some version of OpenSSL, but this is probably an overstatement. The cited number comes from a gross statistic of servers CAPABLE of running OpenSSL.

In reality, I don’t think the vulnerability is quite so broad. Also, since the vulnerability was restricted to only one production release that is about 28 months old and its follow-on beta release of its newer version and not all providers immediately update core services such as SSL/TLS, those running older versions of OpenSLL are not vulnerable to the bug.

Also, OpenSSL is just one implementation of SSL/TLS; there are others. For example, Microsoft products don’t use OpenSSL in their software, so they aren’t vulnerable, including their Internet Information Server (IIS) web server. Even on Unix and Linux-based systems, not every system uses OpenSSL (but many do, see more below.)

To repeat, only systems using OpenSSL v 1.0.1 are susceptible to the Heartbleed vulnerability.

UpdateNetcraft has done some survey/audit work that indicates the number of websites vulnerable to Heartbleed is about 500,000 sites or about 17.5% of sites using SSL certificates. This is not the absolute final authority on the count, but it looks like their technique for coming up with this number is valid, so this number is probably somewhere in the ballpark.

Indeed, OpenSSL is widely used and the scope and scale of the problem will be large and many systems did/do run the vulnerable version 1.0.1 version so the vulnerability is wide-spread, has to be taken seriously and eliminated. This will cause some headaches and some engineering overtime as systems are patched, SSL certificates re-keyed and passwords reset but it is not the end of the world. Systems that employ OpenSSL v1.0.1 simple have to upgrade to v1.0.1g, install new SSL certificates and rest their passwords after the upgrade to make sure any compromised passwords get reset.

Yes, this will require a LOT of work but it is doable.

Most diligent providers have already fixed the vulnerabilities in the systems they manage. Those of us in the field, particularly those who use cryptologic services, have both formal information channels as well as “good-old-boy” networks where news and information get passed globally very quickly; we try to stay abreast of all know vulnerabilities. We use this information to audit our systems and remove any vulnerability as soon as they are discovered. In other words, we pay attention to what is happening and take positive proactive action at the earliest possible date. Most serious providers fall in this category and do what we do, if not even more aggressively.

In our case, our servers did not use OpenSLL per se and therefore our Content Management System was never vulnerable (which means our clients and their transactions and passwords were safe.) We did not have to do anything but confirm this through a quick code audit and testing. As a simple best-practice precaution, we also re-keyed all of our private crypto keys and certificates and re-set all of our admin passwords. This was done only as a better-safe-than-sorry precaution.

However, we host in two very large cloud-based data-center infrastructures (Rackspace.com and Softlayer) and both providers had to similarly audit their infrastructures. In both cases, Rackspace.com and Softlayer had to do some remediation work; OpenSSL is used in VMware (server virtualization software) and in at least one class of Cisco switch employed. These vulnerabilities wouldn’t directly impact our security or that of our users but it could potentially cause Rackspace.com and Softlayer issues. Some of the risks were pretty small because some of the vulnerable systems are notoriously hard to reach in network attacks because of layered security designs. Also, even if their private keys were compromised, it would not compromise our private keys which are separate and unique to us. Although, in theory, it could cause some service interruption if any unpatched systems were attacked.

Both Rackspace.com and Softlayer quickly worked with their suppliers for fixes and patches.

By the way, all of this is simply part of the normal operations management that webmasters and hosting providers do every day to keep our systems secure.

Unfortunately, there will still be some successful exploitation of this vulnerability for some time in the future because some providers are less than diligent, don’t pay attention and/or are lazy. Some of the “over-the-top and the-sky-if-falling reporting may actually be beneficial if it causes more of these types of providers to jump in and fix their systems.

It has been suggested that it might take a couple of years to completely eradicate Heartbleed becuase some people simply won't do the work required. This may be the typical hyperbole early in the game but maybe not; the vulnerability is pervasive. It is too soon to know for sure but the industry will be keeping a close eye on this.

The vulnerability is caused by a missing bounds check in the handling of the TLS heartbeat extension. Without this bounds check, someone wishing to exploit the vulnerability purposely crafts a very long input string and, without a bounds check that make sure the query string is within its limits, the query fails, the system goes into debug mode and dumps back to the person attempting the exploit 64k of memory data. The specific type of buffer error is "CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer." Inside the 64k dumped memory could be important things like passwords and private keys that could be extracted. The exploit could be run multiple times to “bleed” information from memory in 64k chunks, potentially revealing a lot of information in a dedicated exploit.

The missing bounds check could result in a classic overflow or data leak (a.k.a "bleed") and was related to a new "heartbeat" featured added in v1.0.1; hence, the reason for the name "Heartbleed" for the bug.

If you are interested in more details about Heartbleed, see http://heartbleed.com

What Should Webmasters Do?

As a webmaster, if you haven’t yet done so, you should check if your web server and/or other important infrastructure are vulnerable. This may require you checking with the Help Desk of your provider. Most “serious” providers have already released formal statements about any vulnerability to Heartbleed and any remediation needed.

You can also do a quick and dirty test. Filippo Valsorda, an Italian consultant specializing in cryptography and security, has quickly built a test at https://filippo.io/Heartbleed/. Simply enter the URL of you website and the test will check at least to a certain level.

Please note that you shouldn't ONLY rely on the test to confirm you vulnerability. Do indeed also check with your provider to confirm.

For example, you may pass the test now but only because your provider fixed the vulnerability recently. If so, it is possible that passwords and private keys might have already been stolen. Therefore, AFTER you have confirmed that your systems have been patched, revoke you old SSL certificates and re-key them and change your passwords. This will render any previous stolen private keys and passwords useless.

I strongly recommend, as a best practice, to regularly change passwords AND use only strong passwords. Examples of best practices for strong passwords include:

  • At least 10 characters in length
  • At least 1 capital letter
  • At least 2 numbers
  • At least 1 non-alphanumeric character (e.g. !, @, #, $, %)
  • Don’t use whole words (of any language) or names in passwords

A Challenge for Small Church Websites

At TransformingTheChurch.org, we are blessed to have a great deal of senior technical expertise on staff including all of the disciplines necessary (software development, system engineering, network engineering, technical architecture and webmastering) to deal with these types of threats quickly and easily. Also, we have close working relationships with state-of-the-art vendors to take pride in the security of their systems. Working together, we were able to assess any vulnerability to our systems and users within hours of the Heartbleed announcement. It was quickly determined that while not a direct thread to us or our users because we didn't use OpenSSL in our Content Management Software or operating systems, we were in a complex situation because of the broad scope of the thread. We decided that out of an abundance of caution, we would re-key our security certificates and change all of our passwords anyway.

However, we work with a large number of churches that do not have these type of technical experts available to them; perhaps 98% of all church websites are produced without in-depth technical support. A serious thread, often further magnified by over-the-top reporting, has to cause a great deal of concern among churches, particularly small churches.

First of all, don't panic. While this threat is serious, it can be easily dealt with.

If you are the webmaster of a church website, whither you do HTTPS transaction or not, check the website of your hosting provider for any announcements regarding their vulnerability to Heartbleed. By now, almost all have notices posted along with recommendations. Follow their guidance. If there is any question on your part, contact the Help Desk of your hosting provider. Once you have confirmed that their systems have been patched, follow my recommendations above.

If you have a SSL certificate, have it revoked and re-keyed. Certainly change all of your passwords and use strong passwords for your new ones.

That's it! It is no more difficult than that. As simple as these steps are, still be sure to do them. After that, your church website will be safe from Heartbleed.

Your hosting provider will have done (or should have done) all of the heavy lifting for you. Therefore, it will be simple on your part if you are using a good provider. If not, it is time to make a change in providers.

Please note that you may not currently use HTTPS; if you don't, you don't likely have an SSL certificate. If you don't, your haven't been exposed per se. Many small church websites indeed don't use the HTTPS security protocol because you aren't collecting data from users or doing credit-card transactions online. However, OpenSSL could still be used in your web server software awaiting a time that you do need HTTPS and indeed likely is! A SSL/TLS library is part of the standard configuration of all web servers whither you plan on using it or not. If your service provider doesn't patch the vulnerability of any unused-but-installed library anyway, the risk will continue to lurk in your server. Good providers will patch all of their software for all of their user. However, it is worth getting a positive confirmation this was indeed done. If you aren't using HTTPS, you won't likely have a SSL certification issued because they are typically only install when needed. If you don't have a cert you want need to revoke and replace it.

I would still reset passwords as a precaution.

In truth, you will likely have more exposure in your personal online services such as Facebook, e-mail and your e-banking services than you will in your church website. Check the status of these providers and change your passwords as soon as it is confirmed that their vulnerabilities have been patch.

In the comments section of this posting, I will post list of the status of various popular online services as I hear interesting information.

Why/How Did This Happen?

The short answer is because somebody screwed up. And here, I’m talking about only a couple of people; not a big team or a large corporation, but literally a single person and his mistake was overlooked by one other ordinary person; both are unpaid volunteer programmers on the open-source project.

OpenSLL has only four people on its core development team and the OpenSSL code library has 429,699 lines of source code, about 73% of it in the C language. In other words, not a trivial project to audit properly. The theory is that because this is open source and all of the source code is open for inspection by anyone (even you!), that multiple sets of eyes can find any flaws inadvertently made by the software coders. At least that’s the theory.

Programmer Robin Seggelmann has voluntarily admitted that he accidentally bungled the implementation of a keep-alive feature called the TLS Heartbeat Extension for OpenSSL. This feature was committed to the library's source code on New Year's Eve in 2011. Essentially, he forgot to check the size of a received message, allowing a classic overflow failure that create the possibility that miscreants can get 64k snapshots of the inner workings of the attacked software and extract sensitive data flowing through system memory.

This type of overflow hack has been well known for decades and not doing a bounds check just a simple rookie-type mistake and Mr. Seggelmann is definitely not a rookie! However, it is a simple programming error that can be done by anyone, experienced or not. In software development, source code is reviewed and tested before release. The theory of open-source projects is that LOTS of people will review and test the source code because it is open and freely available to anyone; therefore, with more people looking and reviewing means such mistakes will be quickly caught and corrected.

Again, that was the theory; but it resulted in an epic fail.

What has absolutely shocked the online industry is that nobody caught the error for 28 months after the library was released even though the OpenSSL library is so widely used, literally by millions of people. Only two people, Seggelmann's and OpenSSL core developer Dr Stephen Henson, ever reviewed the new code before it was released and both missed the error. Since the release of v1.0.1, all of the developers using OpenSSL and integrating it in their code also never noticed! There are lots of my colleagues scratching the heads this week wondering how such a significant bug slipped through without formal auditing for all this time.

The bug was just discovered and patched, Click here to see the behind-the-scenes story about its discovery. Also, click here for another background story.

Scope and Scale of the Problem

This was a tiny error but it had a major impact. Some are:

  • Various flavors of the Linux operating system potentially shipped a broken OpenSSL code, including Debian Wheezy, Ubuntu 12.04.4 LTS, OpenBSD 5.3, FreeBSD 10.0, and OpenSUSE 12.2
  • Google's Android 4.1.1 is vulnerable, which affects a large number of mobile phones.
  • Google also had to patch its Cloud SQL service and Google Search Appliances, plus its web services: Search, Gmail, YouTube, Wallet, Play, Apps, and App Engine.
  • Amazon also had to patch its cloud services.
  • Websites Facebook, If This Then That, Tumblr, Yahoo! and Yahoo! Mail all had to update their servers
  • 16 networking products by Cisco had the vulnerability.
  • Some Juniper networking gear is affected.
  • VMware has found 27 products that are potentially exposed, including its flagship ESXi 5.5 and vCenter Server 5.5.
  • Citrix is worried about nine products, including important parts of the XenApp application streaming stack.

I'm sure more vendors will "fess up" as time goes on.

Also, some other large powerhouse players have yet to announce clearly their vulnerabilities so the big part of this horror show may continue for weeks or months.

If you do secure transactions with any of these systems or at any of these places, change your passwords right away AFTER you confirm the website has been patched. If you operate server-based systems, revoke your old SSL certificates and renew them.

Those system not impacted include:

  • Microsoft Windows and the Azure cloud simply because Redmond uses its own SSL/TLS suite.
  • Apple's OS X and iOS software, and its websites, were not vulnerable

Summary

The discovery of the vulnerability was confirmed quickly and action was planned. OpenSSL developed a patch that was ready before the announcement of the bug was made. Information about Heartbleed was put out to providers quickly and efficiently. Many providers reacted quickly, within hours or days, to remediate the problem. A large portion of the problem was fixed before the general public learned about the problem.

For many, all that remains is to reset their passwords.

There will be some time before the Heartbleed problem goes completely away because of non-vigilante operators, but this eventually take care of itself as the providers fix things or their customers leave them for better more-agile providers.

There will be a significant positive effect of this problem.

Because of the scope and scale of this issue, there are a lot of people in the software industry that are reconsidering their dependency on open-source software and blindly using it. They are finally a lot of people starting to talk about doing formal audits of critical third-party libraries that are widely used and integrating security testing into their build cycles where they have the best chance at mitigating the risk so threats don’t penetrate quite so deeply.

This discussion has been needed for some time and Heartbleed may cause it to finally begin in earnest.

Category: (04-14) April 2014   Tag:


This is only the blog's abstract. To read the full text and participate in all of the interactive features of the community, please register. It's FREE!

Click Here To Register Into This FREE Community


 

Back