Thermal imaging innovation allows AI to see through pitch darkness like broad daylight

Researchers at Purdue University are advancing the world of robotics and autonomy with their patent-pending method that improves on traditional machine vision and perception.

Zubin Jacob, the Elmore Associate Professor of Electrical and Computer Engineering in the Elmore Family School of Electrical and Computer Engineering, and research scientist Fanglin Bao have developed HADAR, or heat-assisted detection and ranging. Their research was featured on the cover of the July 26 issue of the peer-reviewed journal Nature. A video about HADAR is available on YouTube. Nature also has released a podcast episode that includes an interview with Jacob.

Jacob said it is expected that one in 10 vehicles will be automated and that there will be 20 million robot helpers that serve people by 2030.

"Each of these agents will collect information about its surrounding scene through advanced sensors to make decisions without human intervention," Jacob said. "However, simultaneous perception of the scene by numerous agents is fundamentally prohibitive."

Traditional active sensors like LiDAR, or light detection and ranging, radar and sonar emit signals and subsequently receive them to collect 3D information about a scene. These methods have drawbacks that increase as they are scaled up, including signal interference and risks to people's eye safety. In comparison, video cameras that work based on sunlight or other sources of illumination are advantageous, but low-light conditions such as nighttime, fog or rain present a serious impediment.

Traditional thermal imaging is a fully passive sensing method that collects invisible heat radiation originating from all objects in a scene. It can sense through darkness, inclement weather and solar glare. But Jacob said fundamental challenges hinder its use today.

"Objects and their environment constantly emit and scatter thermal radiation, leading to textureless images famously known as the 'ghosting effect,'" Bao said. "Thermal pictures of a person's face show only contours and some temperature contrast; there are no features, making it seem like you have seen a ghost. This loss of information, texture and features is a roadblock for machine perception using heat radiation."

HADAR combines thermal physics, infrared imaging and machine learning to pave the way to fully passive and physics-aware machine perception.

"Our work builds the information theoretic foundations of thermal perception to show that pitch darkness carries the same amount of information as broad daylight. Evolution has made human beings biased toward the daytime. Machine perception of the future will overcome this long-standing dichotomy between day and night," Jacob said.

Bao said, "HADAR vividly recovers the texture from the cluttered heat signal and accurately disentangles temperature, emissivity and texture, or TeX, of all objects in a scene. It sees texture and depth through the darkness as if it were day and also perceives physical attributes beyond RGB, or red, green and blue, visible imaging or conventional thermal sensing. It is surprising that it is possible to see through pitch darkness like broad daylight."

The team tested HADAR TeX vision using an off-road nighttime scene.

"HADAR TeX vision recovered textures and overcame the ghosting effect," Bao said. "It recovered fine textures such as water ripples, bark wrinkles and culverts in addition to details about the grassy land."

Additional improvements to HADAR are improving the size of the hardware and the data collection speed.

"The current sensor is large and heavy since HADAR algorithms require many colors of invisible infrared radiation," Bao said. "To apply it to self-driving cars or robots, we need to bring down the size and price while also making the cameras faster. The current sensor takes around one second to create one image, but for autonomous cars we need around 30 to 60 hertz frame rate, or frames per second."

HADAR TeX vision's initial applications are automated vehicles and robots that interact with humans in complex environments. The technology could be further developed for agriculture, defense, geosciences, health care and wildlife monitoring applications.

Read more at Science Daily

'I know this song!' Evolutionary keys to musical perception

How do we perceive music and sounds? This question is the basis of the research by the Language and Comparative Cognition Group (LCC) of the UPF Center for Brain and Cognition (CBC) published recently in the journal Animal Cognition.

Humans share characteristics that for the time being appear to be unique in the animal kingdom: language and music. "Our group is dedicated to understanding how these skills have evolved in humans and to what extent some of their components are shared with other species," explains Juan Manuel Toro, director of the LCC and one of the authors of the study, together with Paola Crespo Bojorque and Alexandre Celma Miralles.

When we hear a song that we already know, we can identify it even if it is not an exact version of the original. If it sounds higher or lower, faster or slower, or if the instruments are different from the known version, humans can identify it even if there are these superficial changes to the melody. The study by the LLC explores the extent to which this skill is based on skills that are also present in other animals, i.e., not unique to humans.

Hence, they studied 40 laboratory rats (Rattus norvegicus, commonly known as Long-Evans rats), trained to identify a melody, in this case using the second half of the song "Happy Birthday." "It is a thirteen-tone melody that includes all the entire range of pitches of the Western major scales," they explain in the article.

The experiment began with a familiarization phase followed by three test sessions. Twenty familiarization sessions were held, each session lasting 10 minutes per day. At each session, the rats were placed individually in a response box and presented with 40 repetitions of the familiarization melody while being given a sucrose pill as food.

The results suggest that the ability to recognize patterns over changes in pitch and tempo present in humans might emerge from pre-existing abilities in other species

After the familiarization phase, three sessions were held in which modified versions of the song were used. Responses to the following physical changes in the melody were analysed:
 

• Fundamental frequency (pitch): the song was played one eighth above or below the original.
• Speed (tempo)
• Timbre. The original song was played on a piano and the variant on a violin.

"Our results show that the rats recognized the song even when there were changes in frequency and tempo," Toro explains, "but when we changed the timbre they were no longer able to recognize the song. The results suggest that the ability to recognize patterns over changes in pitch and tempo present in humans might emerge from pre-existing abilities in other species."

Some mammalian and bird species can perceive changes in fundamental frequency (rhesus monkeys -Macaca mulatta), tempo (California sea lion -Zalophus californianus- or cockatoo -Cacatua galerita eleonora) and timbre (chimpanzees -- Pa troglodytes). However, Toro explains that humans process music perceiving musical structures in a relative rather than an absolute way; that is, independently of surface changes along features such as pitch, tempo and timbre. It is thus important to understand the extent to which this ability is based on sensitivities already present in other species."

Read more at Science Daily