‘Big data’ poses big problem for Pentagon

FORT LAUDERDALE, Fla. — Data scientists are the most in demand job for the military, according to Reggie Brothers, the deputy assistant secretary of defense for Research.

The military has a problem with “big data” — the problem being that it collects too much of it. The infatuation with unmanned vehicles and the sensors mounted onto them has spurred a wave of data collected on the battlefield.

Using that data has caused military leaders headaches. Brothers said here at the Association of the U.S. Army’s Winter Symposium on Wednesday that the Army and the other services have placed their focus on PED, or processing, exploitation, and dissemination.

He used the ARGUS-IS as an example of the major advances being made in the world of intelligence sensors. The ARGUS-IS can stream up to a million terabytes of data and record 5,000 hours of high definition footage per day. It can do this with the 1.8 gigapixel camera and 368 different sensors all housed in the ARGUS-IS sensor that can fly on an MQ-9 Reaper.

However, the analysis of the data collected by those sensors can’t keep up. Brothers, a former program manager for the Defense Advanced Research Projects Agency, said the military needs more data scientists to mine through these mountains of data streams. The Pentagon is in competition with private industry as these scientists are sought out throughout the commercial industry.

Military service leaders have already reached out to numerous broadcasting companies such as ESPN and National Geographic to learn ways these broadcast teams deal with the large streams of live-motion footage these companies collect.

One of the major challenges the Defense Department faces with this influx of data is dealing with the personnel demands. Many assume that UAVs require fewer people to operate. Brothers said that would be the wrong assumption.

“We just don’t have the personnel right now,” Brothers said.

Defense leaders are trying to reach the appropriate mix of autonomy in order to lessen the work load on the operators. However, balancing the amount of decision making afforded to the machines has delayed those developments.

The goal is to get to a point where, on average, less than one person is needed to operate an unmanned vehicle. Right now, the military averages well over one person operating its unmanned fleet.

“We need to do better than that,” Brothers said.

About the Author

Michael Hoffman
Michael Hoffman is the executive editor at Tandem NSI and a contributor to Military.com. He can be reached at mhoffman@tandemnsi.com.
  • riicky

    So, tech being to much for us huh. Lol.

  • blight_

    Military’s not the only group with big data headaches.

    Mining the raw data is left to algorithms. Support algorithm development, high performance computing, and ways to visualize data product for human analysis.There’s only so much data transformation possible to turn something into an human-understandable 2D/3D plot.

  • Nick

    Good, now all the government needs to do is recruit top programming talent with their sub par salaries and all of our problems are solved. /sarcasm

    But in all seriousness, these sorts of programming/data integration problems are only solved by some of the most sought after talent in the computer industry. Good luck recruiting them with middling salaries and requirements to work in places that not a lot of programmer types live/work.

  • stephen russell

    Do a massive hiring of Internet & IT trained populace for DoD jobs for security alone.
    IE End users, software, hardware alone
    From all states.
    & downsize for rapid use.

  • Jacob

    I imagine the Chinese are probably stealing more data than they can process as well.

  • Musson

    The Hadoop people will be the first to tell you that putting data across 1000 servers means there will always be a few servers that are on the fritz. So, you will only retrieve data from 998 of them.

    So, you never get the same number twice!

  • Brandon

    Any thing recorded that actually has an event worth keeping can be tagged and moved to long term storage. Preset amount of time before and after the even should be kept until it can later be determined how much is needed and what is not. In the case of Argus, only things that are important in areas not even remotely close to the “event” captured would be the moments immediately before and after the event unfolds to watch for reactions and target indicators. All other data can be held more very temporary amounts of time and be automatically deleted if no events are reported. Obviously this is only half of the solution. Most live feeds from predators doing over watch missions can be temporary and automatically deleted if no worth while intel was observed or reported by the operators or ground elements. I think a lot of the problem has to do SIGINT and other communications type intelligence that actually has to saved and processed which is not quick I am sure especially given the language barrier.

  • Rob

    With US Government underminding budget all the time, its going be even more tougher for military to try get these too few of data scientist. I’ve heard alot of US Programmers being retired and outsourcing the work to India since they work for cheap. It will be incredibly difficult to find a good number of technically alligned people to do this work DoD and other organizations are seeking if they keep obsoleting people coming straight out of college. Not giving them chance due to the lack of experience

    • steve

      AMEN to that brother. No experience means no job? Which came first - the job or the experience. The only way to get experience is to get a job but if you can’t get a job….you know where this is going…….

  • @analyticbridge

    One of the problems is that you need clearance or at least US citizenship to qualify for these jobs. It eliminates a large number of great candidates, even worse: these candidates might end up working for enemies.

  • BobsYourUncle

    What we need to do is filter the data coming in and decentralize its analysis. The biggest problem experienced by Air Force intel during the great man hunts of ’03 to ’11 was its centralized nature. Intel was collected, analyzed and funneled as it went up the chain of command and then back down and through conduits to other branches. By the time it reached people in the field it was too old to be of use. And that’s the intel that wasn’t lost in the static. With everything being funneled through the same hole at the top its a wonder anything of use was ever noticed.

    Fighter pilots in Vietnam were known to turn off gizmos in their cockpits and switch to radio frequencies not cluttered by the other flights in their strike packages. Why would they turn of radar and surface to air missile warning alarms, why stop listening to airborne radar controllers and the other flights of fighters doing battle with the SAMs and Migs that were trying to kill them too? It was more info than they could adequately digest. They prioritized and they filtered. Something people today are afraid to do. Not every tweet is worth reading and not every facebook update warrants attention.

    Too much info can do more to rob you of situational awareness than having too little. It is often bandied about that Sept 11 preparations went unnoticed because the intel agencies were taking in more info than they could process. Its pointless to subscribe to every major paper in the world if you don’t have time to read them all.

    We’ve made the Generals into analysts by giving them battlefield tele-presence and by running every juicy bit of paper across their desks. By doing so we have unwittingly set them up to fail. This wiz bang wow stuff is forcing them to micro manage and forcing them to do it poorly.

  • WPG

    This might sound silly and maybe it is, but when i play command and conquer i can run a war from my laptop just fine lol. Its a very simple war tho…..but that template must be worth something?

  • dipanshu mansingka

    A good product using big data / distributed computing is required. UK has 1.85 million cameras, which are now going to be replaced with HD camera.

  • Dwight_Chroot

    Big Data is DU to IT.

    It’s also a tulip bubble…at a time when we need a New Deal.

    IT needs a 0 level makeover. Instead of rooting everything with spychips then adding malware and misappropriating the WWW to Madison Avenue…we need free hardware
    (free as in freedom, not beer, as in OBEDIENT to the owner, transparent and secure.)

    Instead of could/tablet the industry should be building “print to order” custom hardware built to user spec, and the user burns a ROM hypervisor or bootloader at home. That’s how you get network security.

    But Mr. Burns needs to monitor what color of shoes Justin Bieber fans whim today? You can’t milk any more money out of a collapsing middle class…USEFUL COMPUTERS, and a USEFUL WEB is the way to go…Airforce can handle SIGINT for legit snooping.

  • bing.com

    Do you have a spam issue on this website; I also am a blogger, and I was curious about your situation; many of us have
    developed some nice procedures and we are looking to exchange solutions
    with others, why not shoot me an email if interested.