Subscribe via RSS

Archives by Date
October 2008
September 2008
August 2008

See all Archives
Archives by Category
'Canes
Afghan Update
Ammo and Munitions
Armor
Around the Globe
Av Week Extra
Axe in Iraq (and Elsewhere)
Bizarro
Blimps
Blog Bidness
Body Armor Blues
Bomb Squad
Brownshoes in Action
Bubbleheads, etc.
Cammo Green
Catch the "Buzz"
Chem-Bio
Civilian Apps
Cloak and Dagger
Commandos
Comms
Contingency Ops
Cops and Robbers
Cyber-warfare
Data Diving
Defense Tech Poll
Dissent Tech
Door Kickers
Drones
DT Administrivia
Eat DT's Dust
Extra! Extra!
Eye on China
Fast Movers
FCS Watch
Fire for Effect
FOS Files
Friday Funnies
Gadgets and Gear
Going Green
Grand Ole Osprey
Ground Vehicles
Guns
Homeland Security
In the Weeds with Eric
Info War
Iraq Diary
Jarhead Jazz
JSF Watch
Just War Theories
Lasers and Ray Guns
Less-lethal
Logistics
Los Alamos and Labs
M4 Monopoly
Medic!
Mercs
Missiles
Money Money Money
Most Wanted
MRAP Edge
Net-Centric
Nukes
Old Skool
Our Shrinking Planet
Planes, Copters, Blimps
Politricks
Polmar's Perspective
Popular Mechanics
Rapid Fire
Raptor Watch
Red Team
Retro-Futuro
Robots
Roll Your Own
Sabra Tech
Ships and Subs
Snipertech
Space
Special Ops
Star Wars
Strategery
Stray Trons
Tactical Development
Terror Tech
The Deadlies
The Defense Biz
The Peoples' Site
The Sunday Paper
The Tanker Tango
The View from Av Week
Those Nutty Norks
Training and Sims
Trimble on the Case
Video Lounge
War Update
Ward'z Wonderz
You can run...

See all Archives
Newsletters

Edited by Christian Lowe | Contact

That Deaf, Dumb, and Blind Jet . . .

Tommy.jpg

A recent transpac crippled six F-22s as they made their way from Hawaii to Kadena Air Base in Okinawa, Japan. The culprit: The International Date Line.

When the fighters crossed the line, all of their computer systems went Tango Uniform - fuel subsystems, navigation, and some of the comms.

We turn to CNN's John Roberts and retired Maj. Gen. Don Shepperd for expert commentary on the matter:

ROBERTS: Twenty five years from development to deployment, the F-22 Raptor is the most advanced fighting machine in the air. But it was no match for a computer glitch that left six of them high above the Pacific Ocean, deaf, dumb and blind as they headed to their first deployment. So what happened? We turn to a man who's at home in the cockpit, Retired Air Force Major General Don Shepperd. Don, let me set the scene. These F-22s, eight of them, were headed from Hickam Air Force base in Hawaii to an Air Force base in Japan. They were approaching the international date line, pick it up from there.

F22s.jpg

SHEPPERD: You got it right, John. You want everything to go right with your frontline fighter, $125, $135 million to copy. The F-22 Raptor is our frontline fighter, air defense, air superiority. It also can drop bombs. It is stealthy. It's fast and you want it all to go right on your first deployment to the Pacific and it didn't. At the international date line, whoops, all systems dumped and when I say all systems, I mean all systems, their navigation, part of their communications, their fuel systems. They were -- they could have been in real trouble. They were with their tankers. The tankers - they tried to reset their systems, couldn't get them reset. The tankers brought them back to Hawaii. This could have been real serious. It certainly could have been real serious if the weather had been bad. It turned out OK. It was fixed in 48 hours. It was a computer glitch in the millions of lines of code, somebody made an error in a couple lines of the code and everything goes.

ROBERTS: This is almost like the feared Y2K problem that happened to these aircraft. We should point out that computers control almost every aspect of this aircraft, from their weapons systems, to the flight controls and the computers absolutely went haywire, became useless.

SHEPPERD: Absolutely. When you think of airplanes from the old days, with cables and that type of thing and direct connections between the sticks and the yolks and the controls, not that way anymore. Everything is by computer. When your computers go, your airplanes go. You have multiple systems. When they all dump at the same time, you can be in real trouble. Luckily this turned out OK.

ROBERTS: What would have happened General Shepperd if these brand-new $120 million F-22s had been going into battle?

SHEPPERD: You would have been in real trouble in the middle of combat. The good thing is that we found this out. Any time -- before, you know, before we get into combat with an airplane like this. Any time you introduce a new airplane, you are going to find glitches and you are going to find things that go wrong. It happens in our civilian airliners. You just don't hear much about it but these things absolutely happen. And luckily this time we found out about it before combat. We got it fixed with tiger teams in about 48 hours and the airplanes were flying again, completed their deployment. But this could have been real serious in combat.

ROBERTS: So basically you had these advanced air -- not just superiority but air supremacy fighters that were in there, up there in the air, above the Pacific Ocean, not much more sophisticated than a little Cessna 152 only with a jet engine.

SHEPPERD: You got it. They are on a 12 to 15-hour flight from Hawaii to Okinawa, but all their systems dumped. They needed help. Had they gotten separated from their tankers or had the weather been bad, they had no attitude reference. They had no communications or navigation. They would have turned around and probably could have found the Hawaiian Islands. But if the weather had been bad on approach, there could have been real trouble. Again, you get refueling from your tankers. You don't run -- you don't get yourself where you run out of fuel. You always have enough fuel and refueling nine, 10, 11, 12 times on a flight like this where you can get somewhere to land. But again, attitude reference and navigation are essential as is communication. In this case all of that was affected. It was a serious problem.

ROBERTS: So the fact the computers run so much of the systems on these aircraft, General Shepperd, is the -- is the military at risk of over engineering here so if they did have a problem like that when they were going into a hostile situation, they could be, as you said, repeatedly in real trouble?

SHEPPERD: Well, you have redundant systems but it's just a fact of life in the modern computer age. By the way John, you are going to have the same problem coming up on your laptop computer as we conferred from -- from standard time from daylight savings time to standard time. Your program -- your computer is programmed for one thing and we have changed the dates and you are going to have a problem. It's going to have to be dealt with.

ROBERTS: Do me a favor Don. Make sure I'm not on my laptop computer when I'm flying in an F-22 on that day.

SHEPPERD: Absolutely.

And make sure you don't try to conduct any strikes across the International Date Line. One side or the other, war planners; one side or the other.

Full report at DailyTech.

-- Ward

Comments

Mike S,

"You see what happens when you let the bright young men in the think tanks dictate military equipment specs. "

Make that the few inept young men in think tanks as truly bright young men in think thanks or elsewhere would not forget any essential specs. Please think before making a rash comment like that.

Posted by: WR at April 24, 2008 10:03 AM


There has never been delivery of a new major military (or commercial) product like an airplane, helicopter, tank, ship, etc. that did not have some items to be improved, if not corrected. Actually, I am impressed that correction to the "millions of line of code" was accomplished in 48 hours. It is a little sad to see the world's premier fighter - truly fifth generation - berated so heavily over an issue that could have had some consequences, but didn't. In all mock fights to date, F-22 aircraft have not lost a single fight, to any kind or quantity of competitor(s). What I see is common in our society these days - blow off the overwhelming good of an item (or event)and look/dig for something to criticize. Such is obvious from the tone of questions in the article as well as many comments.

Posted by: David at May 11, 2007 08:51 AM


I LOVE THIS JET!!!!!!I WANT TO FLY IT WHEN I GO INTO THE MARINE CORPS!!!! I MISS HEREING THE SOUND OF IT WHEN I WATCH THE VIDEOS OF IT!!!!!!! THATS EXTREMLY CARZZZZZZZZZZZZZYYYYYYYYYYY!!!!111

Posted by: ashleyarmygirl at April 21, 2007 07:44 PM


I used to write test equipment software and firmware to test Avionics Defense systems. This "coulda", "shoulda" been tested thoroughly before the jet went into the air. This is a failure of their test team. They should have a simulation of those conditions for the software and embedded software to "experience." Crossing such locations are known "events" for an avionics system.
In addition, I agree with one of the other responders. There should be a back up, such as an automatic navigational system that comes on- line (maybe something like the Chelton Flight System or something of that type that can be installed completely independent of the rest of the system.) Of course I'm not sure what system they already have for backup in this jet.

Posted by: Kim at March 13, 2007 07:53 PM


As far as I'm concerned, I would have stopped with the invention of the wheel, all we really need is a cart to haul our excess baggage around on
we have way to much excess baggage and a note on those Iron curtains in officials offices In Washington I think bright red velvet would be exquisite.

Posted by: Karen Francis Henneberger at March 3, 2007 09:24 PM


@Hey Skipper -

It's not a time issue, it's a lat/long issue. Crossing the IDL flying due west means you go from -180 degrees Longitude to +180 degrees Longitude basically instantaneously. Not a problem for properly designed/implemented/tested code; a big potential problem otherwise. Similar mathematical weirdnesses can occur near the Prime Meridian and the Equator, and there are also other factors to consider when north (south) of about 80 North (South).

Triva / acronym pedant moment: ADA is not an acronym (well, it is, but the Americans with Disabilities Act isn't what we're talking about here). Ada the computer language was named for Ada Lovelace, daughter of Lord Byron and the author of what is generally considered the first computer program (for calculating Bernoulli Numbers on Charles Babbage's Analytical Engine).

Posted by: Chris of Dangerous Logic at March 1, 2007 06:06 PM


You see what happens when you let the bright young
men in the think tanks dictate military equipment specs. Mac and his young men (who had never been
in or near the military, let alone combat)....."airplanes don't need machine guns, that style of fighting is over. The F-4 doesn't need anything but rockets." Sure.
Well, we let the little men in again and you get the high techy stuff that amazes and amuses them and an aircraft the you can't fly without a computer. When will we learn?

Posted by: Mike S at March 1, 2007 05:58 PM


Not quite as bad as when the Hawker Typhoon entered service in 1941. Six of the brand new, rushed into service planes engaged German Focke-Wulf's over the channel. In the middle of the fighter, five of the six Typhoons' tails fell off in a high g turn. Without ejection seats like they do today, the pilots never had a chance to get out against the tumbling effect.

Still, this Raptor biz seems weird. Anyone feel like they're getting handed a bogus story from the higher ups?

Posted by: mattrmsf at March 1, 2007 05:32 PM


From the article:
SHEPPERD: You got it. They are on a 12 to 15-hour flight from Hawaii to Okinawa, but all their systems dumped.

Hold On there pardner -
Hickam to Kadena comes up as 4040 nautical miles.

Max speed for a KC-10 is a hair over 600mph,
That comes out to 7 hours.
Using the public numbers, the f-22's would have to refuel 5 times en route

If the F-22 could supercruise all the way, it would be only 3.5 hours.

By the way, it doesn't look like there's such a thing as drop-tanks for an F-22

Posted by: Armchair Jet Jock at March 1, 2007 04:23 PM


I write ADA with my defence contractor. I find it odd that the situation was not tested in their SWIL testing.

Posted by: Chuck at March 1, 2007 03:10 PM


Two million lines of hard real-time embedded code, this is not the stuff of trivial permutations but still, as one commenter said, they continued to fly despite the glitch. Our modern society is built on software--think about what we're doing here commenting on a blog, this is 99% the result of software and 1% copper and silicon--your cool phone now has in excess of a million lines of code--and you still can't get a signal. Setting up software test is likewise non-trivial and those who think this is funny ought to try accounting for all possible usage combinations for some trivial software based product...they might just find some humility.

Posted by: Phred at March 1, 2007 11:05 AM


Giggle-giggle...snort...giggle.....MUAHA-HA-HA-HA-HA-HA-HAAAAaaaa....*cough-cough*. Oh dear, this has got to be one of the funniest stories I've read in a very, very long time.

I just knew this would happen when I saw the "Intel Inside" sticker on the avionics bay access hatch.

And the Royal Danish Air Force is looking into spending my tax money on the JSF. I hope they go with Linux instead (or, so help us, DOS).

Regards and all,
Thomas L. Nielsen
Denmark

Posted by: Thomas L. Nielsen at March 1, 2007 02:58 AM


Ummm ... doesn't all aviation work off Zulu?

It isn't at all clear to me why the F-22 would care about the international date line, as it would be keeping time in Zulu.

The reasonably highly automated jet I fly (MD-11) crosses the date line all the time, and it couldn't care less, because all it knows is Z.

Posted by: Hey Skipper at February 28, 2007 08:10 PM


It may be apocryphal, but I once heard from an electronic engineering friend of a similar bug in a guidance system, developed in the eastern part of the Southern Hemisphere, that didn't work quite so well for the North American customers.

The problem was one of coordinate systems: When you're used to getting positive numbers for South and East coords, and sudden you're getting everything in North and West, things don't work so well. To get things resolved quickly, they effectively installed it back to front - i.e. inverting all the numbers. No-one's going to notice if a missile's flying upside down, right? :-)

Posted by: Dr. Curiosity at February 28, 2007 04:59 PM


The F-22 code is definitely not based on commercial code.

Most of it is written in ADA.

http://www.stsc.hill.af.mil/Crosstalk/2000/05/moody.html

Posted by: Alex at February 28, 2007 04:38 PM


Any bets the code is based on something commercial. Maybe an MS product. Anyone remember what happened to the USS Yorktown? I'm just glad the fly by wire system wasn't compromised or they would have neen in deep sierra.

Military systems need to be using proprietary, fault tolerant code - not some offshoot of a badly written commercial product.

Posted by: Tod Glenn at February 28, 2007 02:38 PM


So what happens if these aircraft are hit by an EMP? Were they shielded?

What about some enemy hacking into the software and uploading a virus? Is the enemy going to hit a switch, activate the program and crash all of the F-22's in flight???

To me, this sounds like a case of too much technology and too little common sense. The fast-mover flyboys have left common sense in the dust in order to get a bigger, flashier, shinier new toy that breaks two days after Christmas.

I think this is a prime example of why old, tried and tre technology still has a place in modern warfare. The K.I.S.S. principle is still alive and true --they forgot to Murphy-proof their new super-fighters and almost cost the pilots their lives, and cost the taxpayers Millions of dollars. Way to go, guys!

It also sounds like a prime reason to keep good old aircraft operational, like the A-10, F-4 and F-14. Or at least include backup, non-computerized instrumentation and controls. (How much space would a backup horizon indicator take, anyway?) I'll bet that they could add a small series of backups in there, to prevent just such an issue --but it might mean that they have to remove the Nintendo system.

One thing is for certain: The F-22 is certainly not Cylon proof. And I'll bet that the real-world Cylons are taking notes and trying to figure out how to exploit this as we speak.

Posted by: Obey Wann at February 28, 2007 12:13 PM


yoiks, when they fixed the code to move FORWARD through time -- I hope they also fixed it for moving BACKWARD through time. Otherwise the trip home FROM Japan TO Hawaii could be another doozie...

Posted by: clea at February 28, 2007 12:07 PM


Here's a question. With an aircraft that is so dependant on computer systems to operate, what would happen if a directed energy weapon were fired at it? Let's say a modulated surge of electricity through a couple of ionized pathways burned through the air by powerful lasers? Such a device has been in development for quite a while now. Would the Raptor be vulnerable to this? I know it's been shown to disable motor vehicles...

Posted by: DS at February 28, 2007 11:57 AM


It's impossible to find all the bugs when testing any decent sized software. Even with common data processing business apps, some bugs will only appear after users start using it. This is one such an example, the programmer didn't think of it, and when testing the Raptor no-one flew over the date line so the bug didn't surface. So now they know about it and fixed it.
The important thing is to have some manual backup, to at least stay above water (or in the air) then the crisis hits.

Posted by: africanmuffia at February 28, 2007 11:34 AM


This is rich. I have been up since 3:30AM Eastern patching hundreds of servers for the upcoming Daylight Saving Time change and I read this. I wonder if the on-board computer systems on the F-22's are patched for that changeover?

Posted by: coxch at February 28, 2007 07:02 AM


As a software engineer in the defense industry, I can pretty say for certain that someone is getting fired right about now.

Nasty contracts letters are being written from the Air Force to Boeing.

Software testers are finding the issue, and determining how it slipped through.

Systems Engineers will be looking through their requirements and realizing that no one ever figured that time might go backwards.

Posted by: Alex at February 27, 2007 10:37 PM


Oh, great. If we ever have to fight the Cylons, we are so screwed. I hope we have some Phantoms that are still operational.

Posted by: Scott Free at February 27, 2007 08:25 PM


I just knew that this episode would bring out the "There's no such thing as a bad weapons system" crowd, and that they'd be minimizing the significance of the thing. They might want to ask themselves -- if something like the International Date Line causes such headaches, what about genuine surprises?

In the end it doesn't really matter much, because the F-22 white elephant is truly a solution in search of a problem. If their wings fell off in mid-air, it wouldn't affect American security a jot.

Posted by: sglover at February 27, 2007 08:13 PM


I agree with livio and Moose. The squadron (or whoever the pros from Dover were) did a good job of rewriting the code and getting the Raptors to press on. BUT . . . it's relatively humorous that it happened in the first place. Remember the unit cost of these puppies is $253 million.

Posted by: Ward at February 27, 2007 07:44 PM


"The F-22 Raptor has something like 200 million lines of code"

Do you have a citation for that? I just googled that and all references I've seen peg the number of lines for the F-22's software at 1.7 to 2 million. Sources included Strategypage and Boeing.

Maybe it's splitting hairs, but it's also ten times less of a problem.

Posted by: bigfoot at February 27, 2007 07:23 PM


Cary M brings up a good point.
I mean... as a nerd, I format my computer once a year... while spring cleaning. It's just something you do!

Now... does that mean that the F-22 will need formatting after a year? Probably not... with more use.. the more often you need to re-install the OS.... so are they going to re-format every sorti? lol

However, as a web programmer, I create 100's of unexpected bugs all the time.. and often some get noted and forgotten as the deadline crunch comes. "Gotta stay on budget". I honestly thought that running a Jet on software being a horrible idea. What happens if the "Motherboard" of the jet gets shot out? What's the piolet going to do to control his plane!

Honestly.. Stick with the Tomcat's.. Send the F-22's in for the anti-radar sorti's and let the F-14's/F-18's do the rest...

Posted by: Foreign.Boy at February 27, 2007 07:15 PM


The F-22 Raptor has something like 200 million lines of code, whereas an OS like Windows XP has something in the neighbourhood of 45 million...and we all know how stable XP is ;).
Oh, and clearly, they never lost the "Whole Platform", as the fly-by-wire systems clearly functioned without a hitch, not to mention the entirely electronic throttle and engine management systems, which obviously rely on computer controlled atmospheric sensors and...well, you get the idea.

Posted by: Cary M at February 27, 2007 07:00 PM


Gotta agree with livio. Bugs are a fact of of life in the Info Age. Someone screwed up, and hopefully they'll check more carefully in that area in the future. But all the planes kept flying (speaks well of their redundancy that they did that with their software on the blink) and today they're in Japan.

Posted by: Moose at February 27, 2007 05:38 PM


What does it say about the defense contractor when a fighter plane that costs $135 million is crippled by the International Date Line? Was $28,000,000,000 in development costs and twenty years of development not enough?

Lockie, you're doing a heck of a job!

Posted by: bigfoot at February 27, 2007 05:33 PM


wow a late report on a problem that was taken care of on the same day.
the sarcasm has no more meaning since they are now in japan.

Posted by: livio at February 27, 2007 05:32 PM


wow a late report on a problem that was taken care of on the same day.
the sarcasm has no more meaning since they are now in japan.

Posted by: livio at February 27, 2007 05:32 PM


Post a comment




Remember Me?


Please enter the code as seen in the image below to post your comment.