Machine learning gives users 'superhuman' ability to open and control tools in virtual reality

Researchers have developed a virtual reality application where a range of 3D modelling tools can be opened and controlled using just the movement of a user's hand.

The researchers, from the University of Cambridge, used machine learning to develop 'HotGestures' -- analogous to the hot keys used in many desktop applications.

HotGestures give users the ability to build figures and shapes in virtual reality without ever having to interact with a menu, helping them stay focused on a task without breaking their train of thought.

The idea of being able to open and control tools in virtual reality has been a movie trope for decades, but the researchers say that this is the first time such a 'superhuman' ability has been made possible. The results are reported in the journal IEEE Transactions on Visualization and Computer Graphics.

Virtual reality (VR) and related applications have been touted as game-changers for years, but outside of gaming, their promise has not fully materialised. "Users gain some qualities when using VR, but very few people want to use it for an extended period of time," said Professor Per Ola Kristensson from Cambridge's Department of Engineering, who led the research. "Beyond the visual fatigue and ergonomic issues, VR isn't really offering anything you can't get in the real world."

Most users of desktop software will be familiar with the concept of hot keys -- command shortcuts such as ctrl-c to copy and ctrl-v to paste. While these shortcuts omit the need to open a menu to find the right tool or command, they rely on the user having the correct command memorised.

"We wanted to take the concept of hot keys and turn it into something more meaningful for virtual reality -- something that wouldn't rely on the user having a shortcut in their head already," said Kristensson, who is also co-Director of the Centre for Human-Inspired Artificial Intelligence.

Instead of hot keys, Kristensson and his colleagues developed 'HotGestures', where users perform a gesture with their hand to open and control the tool they need in 3D virtual reality environments.

For example, performing a cutting motion opens the scissor tool, and the spray motion opens the spray can tool. There is no need for the user to open a menu to find the tool they need, or to remember a specific shortcut. Users can seamlessly switch between different tools by performing different gestures during a task, without having to pause their work to browse a menu or to press a button on a controller or keyboard.

"We all communicate using our hands in the real world, so it made sense to extend this form of communication to the virtual world," said Kristensson.

For the study, the researchers built a neural network gesture recognition system that can recognise gestures by performing predictions on an incoming hand joint data stream. The system was built to recognise ten different gestures associated with building 3D models: pen, cube, cylinder, sphere, palette, spray, cut, scale, duplicate and delete.

The team carried out two small studies where participants used HotGestures, menu commands or a combination. The gesture-based technique provided fast and effective shortcuts for tool selection and usage. Participants found HotGestures to be distinctive, fast, and easy to use while also complementing conventional menu-based interaction. The researchers designed the system so that there were no false activations -- the gesture-based system was able to correctly recognise what was a command and what was normal hand movement. Overall, the gesture-based system was faster than a menu-based system.

"There is no VR system currently available that can do this," said Kristensson. "If using VR is just like using a keyboard and a mouse, then what's the point of using it? It needs to give you almost superhuman powers that you can't get elsewhere."

The researchers have made the source code and dataset publicly available so that designers of VR applications can incorporate it into their products.

"We want this to be a standard way of interacting with VR," said Kristensson. "We've had the tired old metaphor of the filing cabinet for decades. We need new ways of interacting with technology, and we think this is a step in that direction. When done right, VR can be like magic."

Read more at Science Daily

M87 in 3D: New view of galaxy helps pin down mass of the black hole at its core

Seen from Earth, the giant elliptical galaxy M87 is just a two-dimensional blob, though one that appears perfectly symmetrical and thus a favored target of amateur astronomers.

Yet, a new, highly detailed analysis of the motion of stars around its central supermassive black hole — the first black hole to be imaged by the Event Horizon Telescope (EHT) in 2019 — reveals that it's not as perfect as it looks.

In fact, M87 is highly asymmetrical, like a russet potato. The galaxy's shortest axis is about three-fourths (72.2%) the length of its long axis, while the intermediate axis is about seven-eighths (84.5%) that of the long axis.

Knowing this, University of California, Berkeley, astronomers were able to determine the mass of the supermassive black hole at the galaxy's core to a high precision, estimating it at 5.37 billion times the mass of the sun. By comparison, our own Milky Way has at its center a massive black hole only 4 million times the mass of the sun.

They also were able to measure the rotation of the galaxy, which is a relatively sedate 25 kilometers per second. Interestingly, it is not rotating around any of the galaxy's major axes, but instead about an axis that is 40 degrees away from the long axis of its 2D image as observed by the Hubble Space Telescope.

The stereo reconstruction of the M87 galaxy and the more precise figure for the mass of the central black hole could help astrophysicists learn about a characteristic of the black hole they've had no way to determine before for any black hole: its spin.

"Now that we know the direction of the net rotation of stars in M87 and have an updated mass of the black hole, we can combine this information with the amazing data from the EHT team to constrain the spin," said Chung-Pei Ma, a UC Berkeley professor of astronomy and of physics who led the research. "This may point toward a certain direction and range of spin for the black hole, which would be remarkable. We are working on this.”

Further analyses to determine the true shape of giant elliptical galaxies — the galaxies with the largest black holes at their cores — will help astronomers understand better how large galaxies and large black holes form and could help astronomers better interpret gravitational wave signals. Ma leads a long-term study of supermassive black holes that is dubbed MASSIVE.

The results were published online March 15 in The Astrophysical Journal Letters (ApJ Letters).

Determining a galaxy's 3D shape

While spiral galaxies tend to be small, rotate quickly and have a well-recognized pancake shape, giant elliptical galaxies rotate slowly and have a blobby appearance, their 3D shape difficult to discern. Like M87, the largest galaxy in the massive Virgo Cluster of galaxies, giant elliptical galaxies have grown from the merger of many other galaxies. That's likely the reason M87's central black hole is so large — it assimilated the central black holes of all the galaxies it swallowed. In all, the galaxy contains about 100 billion stars, 10 times larger than the Milky Way.

Ma, UC Berkeley graduate student and lead author Emily Liepold, and Jonelle Walsh at Texas A&M University in College Station were able to determine the 3D shape of M87 thanks to a relatively new precision instrument mounted on the Keck II Telescope, one of the twin 10-meter Keck telescopes atop Mauna Kea, a volcano in Hawai'i. Called the Keck Cosmic Web Imager (KCWI), the integral field spectrometer allowed Ma and her team to measure the spectra of stars in the center of the galaxy.

They pointed the telescope at 62 adjacent locations in the galaxy, completely covering a region about 70,000 light-years across, and recorded the spectra of stars within that region. The observations span the central region — about 3,000 light-years across — where gravity is largely dominated by the supermassive black hole, as well as the outer part dominated by dark matter. Though the telescope cannot resolve individual stars — M87 lies about 53 million light- years from Earth — the spectra can reveal the range of velocities within each pixel of each image, enough information to calculate the gravitational mass they're orbiting.

"It's sort of like looking at a swarm of 100 billion bees that are going around in their own happy orbits," said Ma, the Judy Chandler Webb Professor in the Physical Sciences. "Though we are looking at them from a distance and can’t discern individual bees, we are getting very detailed information about their collective velocities. It's really the superb sensitivity of this spectrograph that allowed us to map out M87 so comprehensively."

This is the first time KCWI has been used to reconstruct the geometry of a distant galaxy, and M87 is one of only a handful of giant elliptical galaxies whose 3D structure has been determined. Ma’s team had previously determined the 3D structure of two other giant elliptical galaxies, NGC 1453 and NGC 2693, both harboring smaller black holes than M87.

The researchers took the data obtained during four nights of Keck observations between 2020 and 2022, along with earlier photometric data for M87 from NASA's Hubble Space Telescope, and compared them to computer model predictions of how stars move around the center of a triaxial galaxy. The best fit to the data — axial ratios of 1 to 0.84 to 0.72 — then allowed them to calculate the black hole mass.

"The Keck data are so good that we can measure the intrinsic shape of M87 along with the black hole at the same time," Ma said. "We made the first measurement of the actual 3D shape of the galaxy. And since we allowed the swarm of bees to have a more general shape than just a sphere or disk, we have a more robust dynamical measurement of the mass of the central black hole that is governing the bees’ orbiting velocities."

The authors dedicated their manuscript to the late astronomer Wallace "Wal" Sargent, who first suggested that a supermassive black hole lurked at the center of M87 and calculated its mass to be about 5 billion solar masses.

"His number is a twiddle with our error bars, which is very interesting to see after decades of work," said Ma, who credits Sargent with being a mentor when she was a postdoctoral fellow at the California Institute of Technology.

The previous estimate of the mass of the supermassive black hole in M87, published in 2011, was based on a similar analysis of the dynamical movement of stars around the black hole, though that study assumed the galaxy was axisymmetric. The number, 6.14 billion solar masses, is within error bars of the new, more precise estimate. When imaging the black hole four years ago, the EHT scientists estimated the black hole mass to be 6.5 billion solar masses, 21% higher than the new number.

Interestingly, the dark matter within the volume of the galaxy they analyzed is much higher than that of the black hole — about 388 billion solar masses, or 67% of the entire mass of M87. Though the identity of dark matter is still a mystery, it makes up about 85% of the mass of the universe.

Read more at Science Daily

Machine learning generates 3D model from 2D pictures

Researchers from the McKelvey School of Engineering at Washington University in St. Louis have developed a machine learning algorithm that can create a continuous 3D model of cells from a partial set of 2D images that were taken using the same standard microscopy tools found in many labs today.

Their findings were published Sept. 16 in the journal Nature Machine Intelligence.

"We train the model on the set of digital images to obtain a continuous representation," said Ulugbek Kamilov, assistant professor of electrical and systems engineering and of computer science and engineering. "Now, I can show it any way I want. I can zoom in smoothly and there is no pixelation."

The key to this work was the use of a neural field network, a particular kind of machine learning system that learns a mapping from spatial coordinates to the corresponding physical quantities. When the training is complete, researchers can point to any coordinate and the model can provide the image value at that location.

A particular strength of neural field networks is that they do not need to be trained on copious amounts of similar data. Instead, as long as there is a sufficient number of 2D images of the sample, the network can represent it in its entirety, inside and out.

The image used to train the network is just like any other microscopy image. In essence, a cell is lit from below; the light travels through it and is captured on the other side, creating an image.

"Because I have some views of the cell, I can use those images to train the model," Kamilov said. This is done by feeding the model information about a point in the sample where the image captured some of the internal structure of the cell.

Then the network takes its best shot at recreating that structure. If the output is wrong, the network is tweaked. If it's correct, that pathway is reinforced. Once the predictions match real-world measurements, the network is ready to fill in parts of the cell that were not captured by the original 2D images.

The model now contains information of a full, continuous representation of the cell -- there's no need to save a data-heavy image file because it can always be recreated by the neural field network.

And, Kamilov said, not only is the model an easy-to-store, true representation of the cell, but also, in many ways, it's more useful than the real thing.

Read more at Science Daily

Complex three-dimensional kidney tissue generated in the lab from the scratch

A research team based in Kumamoto University (Japan) has created complex 3D kidney tissue in the lab solely from cultured mouse embryonic stem (ES) cells. These organoids could lead the way to better kidney research and, eventually, artificial kidneys for human transplant.

By focusing on an often-overlooked tissue type of organoid generation research, a type of organ tissue made up of various support and connective tissues called the stroma, Dr. Ryuichi Nishinakamura and his team were able to generate the last of a three-part puzzle that they had been working on for several years. Once the three pieces were combined, the resulting structure was found to be kidney-like in its architecture. The researchers believe that their work will be used to advance kidney research and even lead to a transplantable organ in the future.

The kidney is a very important organ for continued good health because it acts as a filter to extract waste and excess water from blood. It is a complex organ that develops from the combination of three components. Protocols have already been established by various research teams, including Dr. Nishinakamura's team at the Institute of Molecular Embryology and Genetics (IMEG) at Kumamoto University, to induce two of the components (the nephron progenitor and the ureteric bud) from mouse ES cells.

In this, their most recent work, the IMEG team has developed a method to induce the third and final component, kidney-specific stromal progenitor, in mice. Furthermore, by combining these three components in vitro, the researchers were able to generate a kidney-like 3D tissue, consisting of extensively branched tubules and several other kidney-specific structures.

The researchers believe that this is the first ever report on the in-lab generation of such a complex kidney structure from scratch. The IMEG team has already succeeded in inducing the first two components from human iPS cells. If this last component can also be generated from human cells, a similarly complex human kidney should be achievable.

Read more at Science Daily

2D material in three dimensions

The carbon material graphene has no well-defined thickness, it merely consists of one single layer of atoms. It is therefore often referred to as a "two-dimensional material." Trying to make a three-dimensional structure out of it may sound contradictory at first, but it is an important goal: if the properties of the graphene layer are to be exploited best, then as much active surface area as possible must be integrated within a limited volume.

The best way to achieve this goal is to produce graphene on complex branched nanostructures. This is exactly what a cooperation between CNR Nano in Pisa, TU Wien (Vienna) and the University of Antwerp has now achieved. This could help, for example, to increase the storage capability per volume for hydrogen or to build chemical sensors with higher sensitivity.

From solid to porous

In Prof. Ulrich Schmid's group (Institute for Sensor and Actuator Systems, TU Wien), research has been conducted for years on how to transform solid materials such as silicon carbide into extremely fine, porous structures in a precisely controlled way. "If you can control the porosity, then many different material properties can be influenced as a result," explains Georg Pfusterschmied, one of the authors of the current paper.

The technological procedures required to achieve this goal are challenging: "It is an electrochemical process that consists of several steps," says Markus Leitgeb, a chemist who also works in Ulrich Schmid's research group at TU Wien. "We work with very specific etching solutions, and apply tailored electric current characteristics in combination with UV irradiation." This allows to etch tiny holes and channels into certain materials.

Because of this expertise in the realization of porous structures, Stefan Heun's team from the Nanoscience Institute of the Italian National Research Council CNR turned to their colleagues at TU Wien. The Pisa team was looking for a method to produce graphene surfaces in branched nanostructures to enable larger graphene surface areas. And the technology developed at TU Wien is perfectly suited for this task.

"The starting material is silicon carbide -- a crystal of silicon and carbon," says Stefano Veronesi who performed the graphene growth at CNR Nano in Pisa. "If you heat this material, the silicon evaporates, the carbon remains and if you do it right, it can form a graphene layer on the surface."

An electrochemical etching process was therefore developed at TU Wien that turns solid silicon carbide into the desired porous nanostructure. About 42 % of the volume is removed in this process. The remaining nanostructure was then heated in high vacuum in Pisa so that graphene formed on the surface. The result was then examined in detail in Antwerp. This revealed the success of the new process: indeed, a large number of graphene flakes form on the intricately shaped surface of the 3D nanostructure.

Read more at Science Daily

Astronomers create 3D-printed stellar nurseries

Astronomers can't touch the stars they study, but astrophysicist Nia Imara is using 3-dimensional models that fit in the palm of her hand to unravel the structural complexities of stellar nurseries, the vast clouds of gas and dust where star formation occurs.

Imara and her collaborators created the models using data from simulations of star-forming clouds and a sophisticated 3D printing process in which the fine-scale densities and gradients of the turbulent clouds are embedded in a transparent resin. The resulting models -- the first 3D-printed stellar nurseries -- are highly polished spheres about the size of a baseball (8 centimeters in diameter), in which the star-forming material appears as swirling clumps and filaments.

"We wanted an interactive object to help us visualize those structures where stars form so we can better understand the physical processes," said Imara, an assistant professor of astronomy and astrophysics at UC Santa Cruz and first author of a paper describing this novel approach published August 25 in Astrophysical Journal Letters.

An artist as well as an astrophysicist, Imara said the idea is an example of science imitating art. "Years ago, I sketched a portrait of myself touching a star. Later, the idea just clicked. Star formation within molecular clouds is my area of expertise, so why not try to build one?" she said.

She worked with coauthor John Forbes at the Flatiron Institute's Center for Computational Astrophysics to develop a suite of nine simulations representing different physical conditions within molecular clouds. The collaboration also included coauthor James Weaver at Harvard University's School of Engineering and Applied Sciences, who helped to turn the data from the astronomical simulations into physical objects using high-resolution and photo-realistic multi-material 3D printing.

The results are both visually striking and scientifically illuminating. "Just aesthetically they are really amazing to look at, and then you begin to notice the complex structures that are incredibly difficult to see with the usual techniques for visualizing these simulations," Forbes said.

For example, sheet-like or pancake-shaped structures are hard to distinguish in two-dimensional slices or projections, because a section through a sheet looks like a filament.

"Within the spheres, you can clearly see a two-dimensional sheet, and inside it are little filaments, and that's mind boggling from the perspective of someone who is trying to understand what's going on in these simulations," Forbes said.

The models also reveal structures that are more continuous than they would appear in 2D projections, Imara said. "If you have something winding around through space, you might not realize that two regions are connected by the same structure, so having an interactive object you can rotate in your hand allows us to detect these continuities more easily," she said.

The nine simulations on which the models are based were designed to investigate the effects of three fundamental physical processes that govern the evolution of molecular clouds: turbulence, gravity, and magnetic fields. By changing different variables, such as the strength of the magnetic fields or how fast the gas is moving, the simulations show how different physical environments affect the morphology of substructures related to star formation.

Stars tend to form in clumps and cores located at the intersection of filaments, where the density of gas and dust becomes high enough for gravity to take over. "We think that the spins of these newborn stars will depend on the structures in which they form -- stars in the same filament will 'know' about each other's spins," Imara said.

With the physical models, it doesn't take an astrophysicist with expertise in these processes to see the differences between the simulations. "When I looked at 2D projections of the simulation data, it was often challenging to see their subtle differences, whereas with the 3D-printed models, it was obvious," said Weaver, who has a background in biology and materials science and routinely uses 3D printing to investigate the structural details of a wide range of biological and synthetic materials.

"I'm very interested in exploring the interface between science, art, and education, and I'm passionate about using 3D printing as a tool for the presentation of complex structures and processes in an easily understandable fashion," Weaver said. "Traditional extrusion-based 3D printing can only produce solid objects with a continuous outer surface, and that's problematic when trying to depict, gases, clouds, or other diffuse forms. Our approach uses an inkjet-like 3D printing process to deposit tiny individual droplets of opaque resin at precise locations within a surrounding volume of transparent resin to define the cloud's form in exquisite detail."

He noted that in the future the models could also incorporate additional information through the use of different colors to increase their scientific value. The researchers are also interested in exploring the use of 3D printing to represent observational data from nearby molecular clouds, such as those in the constellation Orion.

Read more at Science Daily

3D holographic head-up display could improve road safety

Researchers have developed the first LiDAR-based augmented reality head-up display for use in vehicles. Tests on a prototype version of the technology suggest that it could improve road safety by 'seeing through' objects to alert of potential hazards without distracting the driver.

The technology, developed by researchers from the University of Cambridge, the University of Oxford and University College London (UCL), is based on LiDAR (light detection and ranging), and uses LiDAR data to create ultra high-definition holographic representations of road objects which are beamed directly to the driver's eyes, instead of 2D windscreen projections used in most head-up displays.

While the technology has not yet been tested in a car, early tests, based on data collected from a busy street in central London, showed that the holographic images appear in the driver's field of view according to their actual position, creating an augmented reality. This could be particularly useful where objects such as road signs are hidden by large trees or trucks, for example, allowing the driver to 'see through' visual obstructions. The results are reported in the journal Optics Express.

"Head-up displays are being incorporated into connected vehicles, and usually project information such as speed or fuel levels directly onto the windscreen in front of the driver, who must keep their eyes on the road," said lead author Jana Skirnewskaja, a PhD candidate from Cambridge's Department of Engineering. "However, we wanted to go a step further by representing real objects in as panoramic 3D projections."

Skirnewskaja and her colleagues based their system on LiDAR, a remote sensing method which works by sending out a laser pulse to measure the distance between the scanner and an object. LiDAR is commonly used in agriculture, archaeology and geography, but it is also being trialled in autonomous vehicles for obstacle detection.

Using LiDAR, the researchers scanned Malet Street, a busy street on the UCL campus in central London. Co-author Phil Wilkes, a geographer who normally uses LiDAR to scan tropical forests, scanned the whole street using a technique called terrestrial laser scanning. Millions of pulses were sent out from multiple positions along Malet Street. The LiDAR data was then combined with point cloud data, building up a 3D model.

"This way, we can stitch the scans together, building a whole scene, which doesn't only capture trees, but cars, trucks, people, signs, and everything else you would see on a typical city street," said Wilkes. "Although the data we captured was from a stationary platform, it's similar to the sensors that will be in the next generation of autonomous or semi-autonomous vehicles."

When the 3D model of Malet St was completed, the researchers then transformed various objects on the street into holographic projections. The LiDAR data, in the form of point clouds, was processed by separation algorithms to identify and extract the target objects. Another algorithm was used to convert the target objects into computer-generated diffraction patterns. These data points were implemented into the optical setup to project 3D holographic objects into the driver's field of view.

The optical setup is capable of projecting multiple layers of holograms with the help of advanced algorithms. The holographic projection can appear at different sizes and is aligned with the position of the represented real object on the street. For example, a hidden street sign would appear as a holographic projection relative to its actual position behind the obstruction, acting as an alert mechanism.

In future, the researchers hope to refine their system by personalising the layout of the head-up displays and have created an algorithm capable of projecting several layers of different objects. These layered holograms can be freely arranged in the driver's vision space. For example, in the first layer, a traffic sign at a further distance can be projected at a smaller size. In the second layer, a warning sign at a closer distance can be displayed at a larger size.

"This layering technique provides an augmented reality experience and alerts the driver in a natural way," said Skirnewskaja. "Every individual may have different preferences for their display options. For instance, the driver's vital health signs could be projected in a desired location of the head-up display.

"Panoramic holographic projections could be a valuable addition to existing safety measures by showing road objects in real time. Holograms act to alert the driver but are not a distraction."

Read more at Science Daily