“In spite of the impression many of us have from the media, robots still have a long way to go before they can manipulate so-called compliant objects with proficiency”, says SINTEF researcher Ekrem Misimi.
Misimi believes that a lot more research is needed before this can be realised.
“This is why we’re proud to have developed a method that enables robots to learn to manipulate objects of this kind. This is ground-breaking research offering major innovation potential”, he says.
“The technology currently under development may have many applications across a wide range of industries. But first and foremost in the food sector, where there is no shortage of compliant objects.
An AI-based approach
The new technology has emerged as part of a project, also called BIFROST, which in Norse mythology refers to a ‘bridge’. However, the inspiration for the method started in the kitchen, and more specifically with the stages involved in preparing a humble cod fillet for dinner.
“We’re all familiar with this process without really thinking too deeply about it”, says Misimi. “But if you give it some thought, you find that you’re carrying out a number of different operations on the fillet. In the past, robots were unable to perform such tasks, because up until recently they had only been trained to grip objects that were rigid and non-compliant. Moreover, it is challenging to get a robot to carry out new tasks that it hasn’t been trained to do”, he says.
In the field of AI, robots struggle with what is known as generalisation capability – something at which humans are very adept. In order to succeed, the researchers have developed a new approach based on artificial intelligence. They train the robot by getting it to perform tasks similar to those it would have to perform in the real world. Understandably, these tasks do not involve cod fillets to begin with.
The Bifrost robot in action. With an articulated arm, a cautious ‘finger’ and machine vision, it succeeds in manipulating pliable objects. It has been trained using AI and may have many useful applications, especially in the food industry. Photo: Henriette Louise Krogness.
Time-consuming tasks are the most difficult
On the table lies an elongated cloth bag filled with rice, here simulating a cod fillet. It lies in the shape of an ‘I’, but in response to a command from the researchers, the robot manipulates it into a ‘C’.
“We have thus succeeded in getting the robot to manipulate a compliant three-dimensional object to order”, says Misimi. “On its own, this may not seem very impressive. However, the robot has not previously been trained using this particular shape”, he says.
Misimi adds that it is easy for humans to perform tasks for long periods until they finish what they intend to do, but this is much more challenging for a robot.
“A combination of learning and perception is essential if a robot is to complete such manipulative tasks.
“The more complex a task, the longer a robot will take to learn how to complete it”, says Misimi. “And the more time it takes, the more demanding it becomes for the robot”, he says.
New opportunities
The research team has recently published an article on the method in connection with the prestigious International Conference on Robotics and Automation (ICRA). The team explains that AI-based robotic manipulation offers new opportunities that have previously been beyond our imaginations.
“What is so cool about this method is that the robot is trained exclusively using simulation”, says Misimi.
The knowledge is then carried forward into the real world, without any additional training. This is completely new. The researchers’ main focus has been on training the robots in skills inspired by what humans can do.
“In a way, we’re making them better placed to perform tasks that require human dexterity – tasks that today can only be performed by humans, both in terms of learning and perception”, says Misimi.
The term perception is used here to describe an ability to be aware of what is going on in our surroundings, so that a task can be planned ahead and then carried out. A combination of learning and perception is essential if a robot is to complete dexterous manipulation tasks.
Successive manipulations are tricky
When a robot is tasked with manipulating a given object it has to combine its visual capabilities with perception.
“In such situations, it may also have to work for some time to complete the task”, says Misimi. “To manipulate the object into a new shape, the robot has to complete many different actions and, up until now, it’s been challenging to use robots for such tasks”, he says.
He goes on to explain that a robot commonly requires between 20 and 60 seconds to manipulate an object into a new shape, depending on the complexity of the process.
“It’s a challenge to get a robot to concentrate on a task for an extended period, but we have succeeded”, says Misimi.
Ideal for the food industry
The technology currently under development may have many applications across a wide range of industries.
“But first and foremost in the food, and especially the seafood, sectors, where there is no shortage of compliant objects for manipulation”, says Misimi. “This is important because the use of robots can help towards enabling Norway to maintain its food production industry, and in so doing may also promote more sustainable and locally-sourced food of better quality”, he says, going on to add:
“At an even higher level, knowledge obtained from this project may help us to address some of the fundamental challenges we face in modern robotics. This in turn will lead to the development of new technologies that will benefit both the industrial sector and wider society.
The research team has also succeeded in further developing the BIFROST robot such that it is now able to perform pushing as well as gripping actions. This enables the even more gentle manipulation of compliant objects and structures.
Are you curious to find out more about AI? Why not check out the website Elements of AI, where you will find more information and a free online course (in Norwegian).
References: Sivertsvik, M., Sumskiy, K. and Misimi, E. (2024); Learning active manipulation to target shapes using model-free, long-horizon deep reinforcement learning. IEEE Conference on Robotics and Automation, 13-17 May 2024, Yokohama, Japan.
