Robot 'chef' learns to recreate recipes from watching food videos

Researchers have trained a robotic 'chef' to watch and learn from cooking videos, and recreate the dish itself.

The researchers, from the University of Cambridge, programmed their robotic chef with a 'cookbook' of eight simple salad recipes. After watching a video of a human demonstrating one of the recipes, the robot was able to identify which recipe was being prepared and make it.

In addition, the videos helped the robot incrementally add to its cookbook. At the end of the experiment, the robot came up with a ninth recipe on its own. Their results, reported in the journal IEEE Access, demonstrate how video content can be a valuable and rich source of data for automated food production, and could enable easier and cheaper deployment of robot chefs.

Robotic chefs have been featured in science fiction for decades, but in reality, cooking is a challenging problem for a robot. Several commercial companies have built prototype robot chefs, although none of these are currently commercially available, and they lag well behind their human counterparts in terms of skill.

Human cooks can learn new recipes through observation, whether that's watching another person cook or watching a video on YouTube, but programming a robot to make a range of dishes is costly and time-consuming.

"We wanted to see whether we could train a robot chef to learn in the same incremental way that humans can -- by identifying the ingredients and how they go together in the dish," said Grzegorz Sochacki from Cambridge's Department of Engineering, the paper's first author.

Sochacki, a PhD candidate in Professor Fumiya Iida's Bio-Inspired Robotics Laboratory, and his colleagues devised eight simple salad recipes and filmed themselves making them. They then used a publicly available neural network to train their robot chef. The neural network had already been programmed to identify a range of different objects, including the fruits and vegetables used in the eight salad recipes (broccoli, carrot, apple, banana and orange).

Using computer vision techniques, the robot analysed each frame of video and was able to identify the different objects and features, such as a knife and the ingredients, as well as the human demonstrator's arms, hands and face. Both the recipes and the videos were converted to vectors and the robot performed mathematical operations on the vectors to determine the similarity between a demonstration and a vector.

By correctly identifying the ingredients and the actions of the human chef, the robot could determine which of the recipes was being prepared. The robot could infer that if the human demonstrator was holding a knife in one hand and a carrot in the other, the carrot would then get chopped up.

Of the 16 videos it watched, the robot recognised the correct recipe 93% of the time, even though it only detected 83% of the human chef's actions. The robot was also able to detect that slight variations in a recipe, such as making a double portion or normal human error, were variations and not a new recipe. The robot also correctly recognised the demonstration of a new, ninth salad, added it to its cookbook and made it.

"It's amazing how much nuance the robot was able to detect," said Sochacki. "These recipes aren't complex -- they're essentially chopped fruits and vegetables, but it was really effective at recognising, for example, that two chopped apples and two chopped carrots is the same recipe as three chopped apples and three chopped carrots."

The videos used to train the robot chef are not like the food videos made by some social media influencers, which are full of fast cuts and visual effects, and quickly move back and forth between the person preparing the food and the dish they're preparing. For example, the robot would struggle to identify a carrot if the human demonstrator had their hand wrapped around it -- for the robot to identify the carrot, the human demonstrator had to hold up the carrot so that the robot could see the whole vegetable.

"Our robot isn't interested in the sorts of food videos that go viral on social media -- they're simply too hard to follow," said Sochacki. "But as these robot chefs get better and faster at identifying ingredients in food videos, they might be able to use sites like YouTube to learn a whole range of recipes."

Read more at Science Daily

In spreading politics, videos may not be much more persuasive than their text-based counterparts

It might seem that video would be a singularly influential medium for spreading information online. But a new experiment conducted by MIT researchers finds that video clips have only a modestly larger impact on political persuasion than the written word does.

"Our conclusion is that watching video is not much more persuasive than reading text," says David Rand, an MIT professor and co-author of a new paper detailing the study's results.

The study comes amid widespread concern about online political misinformation, including the possibility that technology-enabled "deepfake" videos could easily convince many people watching them to believe false claims.

"Technological advances have created new opportunities for people to falsify video footage, but we still know surprisingly little about how individuals process political video versus text," says MIT researcher Chloe Wittenberg, the lead author on the paper. "Before we can identify strategies for combating the spread of deepfakes, we first need to answer these more basic questions about the role of video in political persuasion."

The paper, "The (Minimal) Persuasive Advantage of Political Video over Text," is published today in Proceedings of the National Academy of Sciences. The co-authors are Adam J. Berinsky, the Mitsui Professor of Political Science; Rand, the Erwin H. Schell Professor and Professor of Management Science and Brain and Cognitive Sciences; Ben Tappin, a postdoc in the Human Cooperation Lab; and Chloe Wittenberg, a doctoral student in the Department of Political Science.

Believability and persuasion

The study operates on a distinction between the credibility of videos and their persuasiveness. That is, an audience might find a video believable, but their attitudes might not change in response. Alternately, a video might not seem credible to a large portion of the audience but still alter viewers' attitudes or behavior.

For example, Rand says, "When you watch a stain remover ad, they all have this same format, where some stain gets on a shirt, you pour the remover on it, and it goes in the washer and hey, look, the stain is gone. So, one question is: Do you believe that happened, or was it just trickery? And the second question is: How much do you want to buy the stain remover? The answers to those questions don't have to be tightly related."

To conduct the study, the MIT researchers performed a pair of survey experiments involving 7,609 Americans, using the Lucid and Dynata platforms. The first study involved 48 ads obtained through the Peoria Project, an archive of political materials. Survey participants either watched an ad, read a transcript of the ad, or received no information at all. (Each participant did this multiple times.) For each ad, participants were asked whether the message seemed believable and whether they agreed with its main message. They were then shown a series of questions measuring whether they found the subject personally important and whether they wanted more information.

The second study followed the same format but involved 24 popular video clips about Covid-19, taken from YouTube.

Overall, the results showed that video performed somewhat better than written text on the believability front but had a smaller relative advantage when it came to persuasion. Participants were modestly more likely to believe that events actually occurred when they were shown in a video as opposed to being described in a written transcript. However, the advantage of video over text was only one-third as big when it came to changing participants' attitudes and behavior.

As a further indication of this limited persuasive advantage of video versus text, the difference between the "control condition" (with participants who received no information) and reading text was as great as that between reading the transcript and watching the video.

These differences were surprisingly stable across groups. For instance, in the second study, there were only small differences in the effects seen for political versus nonpolitical messages about Covid-19, suggesting the findings hold across varying types of content. The researchers also did not find significant differences among the respondents based on factors such as age, political partisanship, and political knowledge.

"Seeing may be believing," Berinsky says, "but our study shows that just because video is more believable doesn't mean that it can change people's minds."

Questions about online behavior

The scholars acknowledge that the study did not exactly replicate the conditions in which people consume information online, but they suggest that the main findings yield valuable insight about the relative power of video versus text.

"It's possible that in real life things are a bit different," Rand says. "It's possible that as you're scrolling through your newsfeed, video captures your attention more than text would. You might be more likely to look at it. This doesn't mean that video is inherently more persuasive than text -- just that it has the potential to reach a wider audience."

That said, the MIT team notes there are some clear directions for future research in this field -- including the question of whether or not people are more willing to watch videos than to read materials.

Read more at Science Daily