DARPA wants AR goggles to help soldiers with complex tasks

DARPA (Defense Advanced Research Projects Agency) is developing a new system to help military personnel perform complex tasks. Perceptually-enabled Task Guidance (PTG) technology uses sensors to see and hear what the user sees and hears, guiding them through AI-produced instructions displayed in augmented reality (AR).

PTG combines sensors (a microphone and head-mounted camera) with AI and AR headsets to integrate into the user’s environment. The idea is to help soldiers and other military personnel enhance their skills, complete complicated tasks and perform them better. DARPA has narrowed its focus to three areas: battlefield medicine (like untrained personnel helping medics in the field), sustainment (keeping military equipment up and running) and co-piloting (especially helicopters).

However, DARPA’s training demos use something more pedestrian: cooking. Dr. Bruce Draper, the program’s manager, describes it as the ideal proxy task. “[Cooking is] a good example of a complex physical task that can be done in many ways. There are lots of different objects, solids, liquids, things change state, so it’s visually quite complex. There is specialized terminology, there are specialized devices, and there’s a lot of different ways it can be accomplished. So it’s a really good practice domain.” The team views PTG as eventually finding uses in medical training, evaluating the competency of medics and other healthcare services.

The personnel demoing the tech appear to be using a variant of Microsoft HoloLens. The government recently halted plans to buy more “AR combat goggles” from Microsoft, instead approving $40 million for the company to develop a new version. The reversal came after discovering that the current version caused issues like headaches, eyestrain and nausea.

DARPA is the Department of Defense’s “mad science” division. Founded by President Dwight Eisenhower in 1958 to develop cutting-edge tech for the US, many of the agency’s projects have trickled down into non-military products, including GPS, speech recognition, self-driving cars and robotics. Oh, and a minor technology called “the internet” also stemmed from DARPA’s late 1960s ARPANET (Advanced Research Projects Agency Network) program. It’s easy to imagine some form of PTG eventually following their lead into our everyday lives.