As the Damages Decline: A comparative look at the CSI/FBI survey loss numbers

Robert Richardson

Our June NetSec conference in Scottsdale, Ariz. provided an opportunity to grant attendees a preview of this year’s CSI/FBI Computer Crime and Security Survey results. The formal release of the survey report (downloadable at no charge from GoCSI.com) will be made on July 12. But the survey report sticks to the numbers and offers little by way of interpretation. At the NetSec preview, I was able to share a bit more by way of comparison to other reports and I’d like to do a bit of that in these pages as well.

The survey found respondents once again telling us that they’d lost less money on average than in the previous year. The average loss per respondent in this year’s survey was down nearly 18 percent from last year’s ($203,606 in the 2005 report, against $167,713 this year).

This is the fifth straight year that average losses have dropped; and the drops have been nothing short of startling in some years. Given a general security climate in which the vulnerability count is climbing and the reports of computer crime get more and more alarming, one must wonder how it can be that our respondents keep reporting lower and lower numbers.

The skew possibility

One thing worth bearing in mind: all survey respondents are CSI members and are, presumably, more security-savvy than might be found at average organizations throughout the country. (One suspects that many organizations have little or no active interest in security.) So one possibility is that CSI members get better results because they try harder and know more about successful implementation of security measures.

For the first time in the survey’s history, it’s possible to make a relatively “apples-to-apples” comparison of the CSI/FBI survey to a survey of organizations that don’t skew towards security-savvy respondents. This past year, FBI special agent Bruce Verduyn conducted a survey of businesses in major metro areas in four states. Several of his questions were nearly identical to CSI/FBI survey questions (Verduyn was aware of the survey, but his survey was a one-time endeavor not connected with our annual survey) and included questions about financial losses.

An initial glance at a question that asked what organizations did when they experienced a computer security incident seems to uphold the “savvy skew” theory. One response available in both the CSI/FBI and the Verduyn surveys is “reported the incident to law enforcement.” Verduyn’s “generalist” respondents said they took this action 9.1 percent of the time. Respondents to the CSI/FBI survey clocked in at 25 percent. There’s clearly a difference in readiness to involve law enforcement.

But the difference doesn’t necessarily hold when it comes to the question that counts: how much money is lost. If the “security savvy” theory holds, then you’d expect CSI/FBI respondents to have lower loss numbers than “the great unwashed.” In point of fact, Verduyn’s average loss per respondent is roughly $24,000, as compared to the roughly $168,000 reported in this year’s CSI/FBI survey. The unsavvy would appear to be losing a lot less money.

Now this isn’t quite a fair comparison. The CSI/FBI survey skews toward respondents being in larger organizations. Approximately half of respondents work in organizations with more than 1,500 employees (and 9 percent work at organizations with more than 50,000 employees). But the world at large has many, many more small organizations, and it’s this natural skew toward smaller organizations that is reflected in Verduyn’s survey.

I tried to get a cut of Verduyn’s data that would reflect the average of larger organizations, but Verduyn has since been transferred to another office, the data set has effectively been orphaned, and another look at the data was not forthcoming. It is Verduyn’s belief, however, that the average number for larger organizations was probably fairly close to the CSI/FBI average.

In other words, “the great unwashed” don’t think they’re losing more money than CSI members do. Nor do they think they’re losing all that much money in a given year, on average. Apparently there’s no skew for savvy when it comes to loss numbers, which is perplexing.

In fact, there have been years when average losses in other reports, had they been reported as distinct numbers (some surveys have been a bit cagey on the subject), would have shown a general pool of IT professionals reporting that they’d lost less money on average than the losses reported in the CSI/FBI survey. In 2004, for example, InformationWeek’s research group surveyed U.S. IT professionals with regard to cybercrime losses. They didn’t explicitly report an average loss amount, but did say that “few report being especially hard-hit financially.” Indeed, their report notes, “half of sites report losing under $100,000.”

The InformationWeek survey posed its question about cyberlosses by asking respondents to check off which of several possible ranges of losses they fell into. If one takes the median for each of these several ranges, plus the high end of the lowest range and the low end of the highest range (because the high losses are far less common and the range is very large), then multiplies each figure by the number of respondents in each range, dividing by the overall number of respondents yields an average loss turns out to be approximately $95,000. Depending on your estimate of the security savvy of InformationWeek readers (who are in IT, to be sure, but not necessarily specialized in security), you might expect this number to be higher than the corresponding CSI/FBI number for that year—but the average loss number for the 2004 survey report was approximately $526,000.

If one looks at economically advanced countries elsewhere in the world, the loss numbers are substantially lower than those in the United States, but that’s perhaps to be expected because the Internet and the e-commerce that runs on it have been so heavily dominated by U.S. firms. Take Japan, for example. Katsuya Uchida, an associate professor at the Institute of Information Security in Yokahama, has conducted a survey modeled on the CSI/FBI survey for several years. His most recent results show an average loss per respondent of 5,334,000 yen; about $46,000.

The takeaway here is simply that the seemingly low average losses in the CSI/FBI survey aren’t anomalous (unless one wanted to argue that they were too high). They are roughly in line with numbers in other surveys that have attempted to get a handle on financial losses.

Other low numbers

In thinking about this question of how the losses could be dropping, I decided to go back and take a look at other sources of information about cybercrime.

The vulnerability counts, though they may be rising, aren’t particularly relevant. They only indicate the possibility of exploit (and some of the vulnerabilities these days, frankly, seem fairly insignificant and unlikely to be used for crimes).

Not many other surveys ask directly about financial losses in the way that the CSI/FBI survey consistently has done (the survey is the oldest of its kind that we’re aware of, now in its 11th year). But another approach is to try and find out whether the number of attacks is dropping. Of course, increases (or decreases) of attacks may not directly correlate to increases (or decreases) in losses, but it would be interesting and suggestive to learn that overall attacks had dropped at the same time that losses dropped.

A survey that measures attacks by examining logs from more than 20,000 sensors worldwide is Symantec’s semi-annual Internet Security Threat Report. Here, the news is somewhat inconclusive. These reports used to track what Symantec termed “severe events.” This was a count, in other words, of things that actually looked like attacks, rather than port scans and the like. From the latter half of 2001 through the first half of 2003, the severe attack count dropped decisively. The number of organizations showing severe attacks as a percentage dropped from 43 to 11 percent during the period. Thereafter, Symantec switched gears and didn’t report this “severe event” category anymore.

Instead, reports from 2003 forward count average daily attacks, without breaking out the “severe” category. Here, the news has been mixed. Attacks dropped from 15.3 to 10.6 through the first half of 2004, jumped to 57 for two reports, then dropped in the most recent half-year report (for the latter half of 2005) to 39.

What Symantec’s reports have never shown, however, is a cybercrime situation that is spiraling out of control. There are plenty of things to be concerned about, but there are also periods when the “bad” numbers drop—in fact there are more such periods than periods when they rise. There’s a great deal more information in these reports, by the way. They’re well worth spending some time with.

Are the numbers believable?

Since providing preview numbers of the survey, I’ve been sent a preview of a Gartner “First Take,” an initial analysis for the benefit of Gartner clients looking for quick, “initial take” positions and recommendations. Gartner focuses on the average loss number and takes the position “that security administrators should view the findings of all such surveys with extreme skepticism.” This is, of course, perfectly good advice of the sort that it costs absolutely nothing to give.

It’s true that respondents are estimating their losses, equally true as well that there is no standardized approach to making these estimates. So the estimates may, as a whole, be wide of the mark.

At the same time, it is roughly the same group of security professionals responding to the CSI/FBI survey each year. Their demographics remain consistent year over year. They may make estimates that would prove inaccurate in an ideal world where such things could be accurately measured, but it is my belief that there is some overall consistency in the way they make their estimates. And for five straight years they have reported that they lost less money to computer crime. I think there is a consistency here that is worth crediting.

Or to put it another way, the evidence we have gathered from some five hundred or so security professionals each year for the past 11 years leads me to believe that their losses to cybercrime each year are emphatically not spiraling out of control. And this is not a trivial observation to be able to make—we’re constantly barraged with unsubstantiated claims that security professionals as a group have somehow thrown the barn doors open wide and are powerless to avoid losing the horses.

This, it’s worth adding, is accomplished with little money, relative to overall IT budgets. More than one-half of respondents said they made do with 2 percent or less of the IT budget.

What about disasters?

On the other hand, this year seems to have seen an onslaught of news stories reporting one data breach after another. If you have a look at the chronological list of data breaches maintained by the Privacy Rights Clearinghouse, you’ll find dozens of breaches where personal data was lost; 18 such occurrences for this past June alone.

Some of these losses are numerically spectacular, perhaps the most outrageous being the loss of some 28 million records of American veterans. This is a terrible occurrence, one that shouldn’t be condoned, but two points are worth making. First, the vast majority of the records stolen in these sorts of incidents don’t wind up being used for identity fraud. A report by Javelin Strategy claims that over 90 percent of access to unauthorized records subsequently used for fraud is obtained through traditional, non-electronic means.

More importantly, these breaches are rare in the scheme of things. There are upwards of 24 million businesses in the U.S., according to the Census Bureau. Some 60 percent of these are very small, with revenues of less than $25,000 per year. But if we discount these presumably part-time businesses, and conservatively say there are 10 million real, ongoing concerns, then the odds of any given business suffering a media-covered breach in any given year are something on the order of .001 percent (and we haven’t added in the large number of governmental and non-profit organizations that should also figure in here).

Nor are the real costs of such breaches to the organizations themselves by any means clear. After the ChoicePoint debacle in February 2005, there was speculation about whether the company would survive. The company’s game plan, internal sources said, included changing the company name as a way to try and sidestep the permanent mark on their brand. Meanwhile, however, ChoicePoint is still alive and well. According to the company’s financial releases, revenue from its business services operations in 2005 were “essentially flat.” Further, the company’s overall revenue was up 15 percent for that year. This year, ChoicePoint reports record revenues for the first quarter, though business services remain flat. There’s seemingly a price in lost growth, in other words, but not a crippling one. In terms of market capitalization, ChoicePoint had fully recovered by the start of this year (though it, like lots of other stocks, has recently taken a beating).

This is, of course, not to say that there aren’t expenses associated with data privacy breaches. The simple act of compliance with disclosure laws—the mailing of a letter explaining the breach to each affected customer—has a cost, as does offering a year or two of free credit monitoring. But the $240 that the Internet Crime Complaint Center says an average instance of online credit card fraud costs never gets charged back to the company that allowed the breach.

And however you reckon the costs of “the big one,” they aren’t in all likelihood ever going to figure into the numbers reported by CSI/FBI survey. The odds of one of any given 600 respondents (there were 616 respondents overall this year) having such an occurrence are simply too small for this to be reasonably expected. It’s probably also not likely that the poor security manager who’d just been “featured” on the nightly news would answer the loss question. It’s an anonymous survey, after all, but the $25 million entry under “theft of proprietary data” might still be a giveaway.

The survey numbers should be viewed more as an indication of what happens in the normal day-to-day grind, in years when “the big one” hasn’t happened.

On the whole, then, it seems reasonable to think that what CSI members have been telling us—even if their estimations of loss aren’t entirely accurate—provides a clear indication that, at least within the traditional enterprise intranet, cybercrime losses have been falling.

Meanwhile, of course, there are always new threats on the horizon. New and successful attacks have the potential to be devastating and ferociously expensive, so it’s not as though we can declare victory, sit back and watch the LEDs on the firewall blink. But “traditional” methods seem to be having good effect against run-of-the-mill threats.

There is plenty of other news in the survey, I hasten to add. Focus on the falling average loss numbers shouldn’t blind us to the bulk of the survey, which looks at what gets spent on security, what kinds of security technologies are deployed, and what measures infosec professionals take to ensure their security measures are working. The survey, as always, can be downloaded for free from GoCSI.com. Members additionally receive a free hardcopy edition.

The remarks here reflect Robert’s opinions, not necessarily those of his three survey co-authors. Robert Richardson is CSI's editorial director.

Download Survey PDF registration required