[ISN] Router Flaw Is a Ticking Bomb

[InfoSec News]

http://www.wired.com/news/privacy/0,1848,68365,00.html

By Kim Zetter
Aug. 01, 2005

LAS VEGAS -- Security researcher Mike Lynn roiled the Black Hat 
conference Wednesday when he resigned from his job at Internet 
Security Systems to deliver a talk about a serious vulnerability in 
Cisco IOS, the operating system powering its routers, defying efforts 
by the router manufacturer and his former employer to block the 
presentation.

In the aftermath, Lynn reached a legal settlement with Cisco and ISS 
in which he agreed to erase his research material on the 
vulnerability, to keep secret the details of the attack, and to 
refrain from distributing copies of his presentation, among other 
concessions.

Now facing an FBI investigation -- and sudden celebrity status in the 
tech world -- Lynn discusses the events leading up to this week's 
disclosure, and what he thinks it means for the security of the 
internet in an exclusive interview with Wired News.


Wired News: Can you tell me how all of this started? You were asked by 
your employer, ISS, to reverse-engineer the Cisco operating system, 
weren't you? 

Michael Lynn: I was very specifically told.... It was January 26th and 
Cisco had just announced a totally different vulnerability than the 
one I demonstrated. They'd announced a vulnerability for something 
called "Multiple Crafted IPv6 Packets Cause Router Reload" (as they 
worded it in their patch message). But that's a very vague term. It 
just says, "Hey, something is wrong in IP6 with the router reload" ... 
but it didn't say you could be in control of it. 

ISS wanted to get protection in their products (against this problem) 
so that their customers wouldn't be as affected by it. So they called 
up Cisco to try to get some more details for it ... and Cisco wouldn't 
give (the information) to them. So (ISS managers) came to me and said, 
"Can you reverse-engineer ... can you disassemble IOS ... to find out 
what their vulnerability is?" 


WN: So this was a different vulnerability from the one you 
demonstrated at the conference this week? 

Lynn: Yes, but (Cisco) had (also) found the vulnerability that I 
demonstrated on stage about two weeks before I (found it). 


WN: Then what happened? 

Lynn: So on January 27th, ISS comes out with their response to this 
vulnerability -- the advice to their customers based on my 
analysis.... I stayed up all night basically (to research it). 

I realized in looking at this (that the program) is actually way worse 
than Cisco said.... So (our guy) calls up ... Cisco and says, "OK, we 
aren't 100 percent sure that we found the same bug that you're talking 
about, but it's important we find out because the one we found has 
much, much greater impact. You said there's (the possibility) of a 
denial-of-service attack. But the one we found is fully exploitable." 

Cisco said, "You guys are lying. It is impossible to execute shell 
code on Cisco IOS." At that point (ISS) management was annoyed.... 
They were like, "Mike, your new research project is Cisco IOS. Go find 
out how to exploit bugs on Cisco IOS so we can prove these people 
wrong." 


WN: In your speech you said you worked on the reverse engineering with 
cooperation from Cisco. 

Lynn: We did, in fact. The cooperation came later. They didn't start 
that way, and they were not happy to begin with.... They didn't 
cooperate in the actual reverse engineering itself. They cooperated in 
the research effort, I would say, in finding vulnerabilities and 
confirming (them). 


WN: They didn't stop you. 

Lynn: They didn't stop us, and at this point there was some 
back-and-forth communication. (Lynn spent the next month researching 
the program.) 


WN: After you came to them with the serious flaw and said, "This is 
the bug we found...." 

Lynn: They said, "We don't believe you." And (ISS managers) said ... 
"come down to Atlanta and we'll show you." And that's never happened, 
by the way, at ISS. They've never brought somebody, let alone a 
competitor, into the office just to show them (something).... Mike 
Caudill, (Cisco's) customer advocate, came out. And they also sent out 
an engineer ... who described himself as an IOS architect.... I was 
told he helped design parts of the source code.... And his jaw hit the 
ground. He was very impressed, he was just (saying), "Wow, that's 
cool." That was June 14th. 


WN: Cisco saw your Black Hat presentation long before they decided to 
pull it. When did they see it? 

Lynn: Probably June 14th, the day that they came out (to Atlanta). We 
told them about the vulnerabilities well before (that). 


WN: So at what point did they get nervous about the talk? 

Lynn: When they saw the listing of the presentation on the Black Hat 
site is when they actually called us back and said, "Wait, you guys 
were serious?" And we said, "Yes, we were serious." Incidentally, it 
was ISS who submitted (the talk) for Black Hat. I was told (by ISS), 
"Hey, you want to go to Black Hat? We'd like you to do it." 


WN: So ISS knew the seriousness of the bug. 

Lynn: Yes, they did. In fact, at one point ... they apparently didn't 
get it, and they actually wanted to distribute the full working 
exploit very widely inside the company.... I was told ... "Give this 
to all the sales engineers and to all the pen testers." 


WN: Why would they want you to do that? 

Lynn: Well, because it bruises Cisco, remember? Mind you, this was 
something that Cisco hadn't gone public with yet and that's not useful 
to pen testers because what do they advise their customers to do (to 
protect themselves if no information about the vulnerability has been 
released yet)?

I told them, "You do realize if you do that, it's going to leak?" And 
(one of the ISS guys) says, "That's Cisco's problem." And then 
(another ISS guy) turns to me and says that they need to understand 
this could be their Witty worm. I was like, Whoa, what meeting did I 
walk into? 

(The Witty worm was a particularly aggressive and destructive code 
released by someone last year that targeted computer systems running a 
security program made by Internet Security Systems and even more 
specifically targeted military bases using the software. It infected 
more than 12,000 servers and computer systems in about an hour. 
Because of the worm's speed in spreading and its creators' apparent 
knowledge of who ISS' customers were, some security experts speculated 
that someone working for or connected to ISS might have been 
responsible for writing and releasing it.) 

At that point, I told them all no, and they fought it and I resigned 
right there on the spot. And this was about a month ago. 

I thought they were handling this in a non-ethical manner. Because it 
was just way too fast and loose with who can see this.... I mean, I 
don't even want people to see it now. (ISS talked him out of the 
resignation by agreeing to give him control over who could see or have 
the exploit.) 

So we start moving forward with the talk and we're working with Cisco, 
and Cisco seems OK with it. 


WN: They had already released information about what you found before 
your speech, right? 

Lynn: Yes, and the fix. The fix was about six months before the 
message. 


WN: So they already knew how serious the problem was. 

Lynn: If they didn't know, they should have. 


WN: But they didn't indicate to their customers how serious it was. 

Lynn: No, they did not. 


WN: And Cisco saw your Black Hat presentation long before they decided 
to pull it, right? 

Lynn: Probably June 14, the day that they came out (to Atlanta). 

(Then) it was two weeks ago, I was first told that Cisco might want to 
come onto (the) stage with me and say a couple words. And I said, 
provided the words aren't something to the effect that "he's a liar," 
I'm OK with it.... It didn't really matter. It lent credence to my 
talk. And it's good because I felt my talk really needed to be taken 
seriously. 

(However, the plan changed even more and Lynn was told to remove any 
mention of reverse engineering from his talk or cancel the 
presentation. If he did neither, he would be fired.) 

Mind you this is a complete reversal. Like a week or so prior, the 
night of the close of the fiscal quarter, and they were all 
celebrating that they hit the numbers, the CEO invited me out for a 
beer, and he just couldn't say enough awesome things about this talk. 


WN: Was Cisco threatening them? 

Lynn: I asked point-blank, "Are you being threatened by Cisco?" They 
said no.... To be perfectly honest, I don't think there was any legal 
threat. I think that it was more of a "scratch our back and we'll 
scratch yours." 

(Cisco asked him to wait a year until it could release a new version 
of its operating system. When he didn't back down, Cisco threatened a 
lawsuit against Lynn and Black Hat. Then with Black Hat's cooperation, 
Cisco arranged to tear out pages with images of Lynn's slides from the 
conference book.) 


WN: You met with the feds after your talk, and someone gave you a 
challenge coin (a special coin created for members of the military to 
commemorate challenging missions)? 

Lynn: Yes, they did, actually. And I didn't know what it was, so I 
didn't thank him properly.... This was a really funny story. (Right 
after my talk, this) guy walks up with a very, very impressive badge 
... and says, "I need to speak with you. Now." 


WN: What agency was it? 

Lynn: Air Force (Office of Special Investigations). NSA, is what I'm 
told, but he wouldn't show me his credentials. There were a lot of 
flashy badges around from lots of three-letter agencies. So they take 
me to a maintenance area and I'm surrounded by people ... and one of 
them says (to another guy), "You've got the van ready?" I'm going, "Oh 
my god." And they go, "Just kidding!... Oh, man, you rock! We can't 
thank you enough." And I'm just sitting there, like still pale white. 
They all shook my hand. 

I get the feeling that they were in the audience because they were 
told that there was a good chance that I was about to do something 
that would cause a serious problem. And when they realized that I was 
actually there to pretty much clue them in on ... the storm that's 
coming ... they just couldn't say enough nice things about me.... 
Also, US-CERT (Computer Emergency Response Team) asked me if I would 
come up to D.C. in a week or two and help them formulate the nation's 
strategy for cybersecurity. 


WN: So this new version of the operating system that they're coming 
out with, that's in beta testing. 

Lynn: It's actually a better architecture ... but it will be less 
secure.... That's why I felt it was important to make the point now 
rather than sweep it under the rug. I think it's something that we can 
fix.... 

The problem now ... is that if you want to attack something ... you're 
going to (have to) hack one machine (at a time) and take control of 
the part of the network (it's on). If you had (the exploit) up running 
against the new version that's in beta now, you can take everything. 
That's the difference between something you can make a worm out of and 
something you can't make a worm out of. 

(Right now) nobody patches Cisco routers because there's been this 
culture (that) there's just never anything that can go wrong (with 
them). So, unless there's some really critical thing that's making it 
crash, people don't install the patches.... We have to change the 
public perception about patching now, and that cause is not best 
served by pretending that there's not a problem and saying maybe you 
can talk about this next year.... The time to talk about this is 
before the critical problem comes around. 


WN: Cisco has said this is not a critical flaw that you found. 

Lynn: I would agree with them in part and disagree with them. In a way 
I would say, yes, it's actually not all that exceptional in that all 
it proved is it's just like any other computer -- they're all 
hackable. Because in any complicated system, people make mistakes. 
It's our very nature. 

But in the sense that the potential impact of something like a router 
worm (attacking the routers) is no big deal, I would strongly 
disagree. Unlike most other vulnerabilities or exploits, when you ... 
take control of another machine, it's very difficult, if at all 
possible, for you to ... destroy the hardware.... But on a router? 

This is (a scenario in which) the network is down, and it's down in a 
way that it's not getting up again. How do you ship the patch when the 
network won't (be up so you can distribute it)? Are you going to mail 
out a CD? But there's no CD drive. 

The real point is there's a ticking clock but we still have plenty of 
time. I wanted people to be afraid a little bit ... because I needed 
people to act. But at the same time, now that I think they already 
are, I will say it's not as bad as you probably think it is. Not yet 
... because the version that makes this an unstoppable critical 
problem is not out yet.