The opinions expressed in this article are those of the authors alone. This article was first published in the financial daily Dagens Næringsliv and is reproduced here with the permission of the paper.
Tesla has been testing self-driving cars for many years, and NTNU was able to participate behind the scenes as part of a recently published study.
Tesla has produced several versions of its autopilot AI tool, which has become better and better with every new release. The tool is trained by its users. In other words, by drivers that use the autopilot function.
It is now clear that some of these drivers have been highly risk-tolerant techno-optimists.
Dead – asleep at the wheel
Some test drivers have had so much confidence in the autopilot function that they have fallen asleep at the wheel. This has resulted in accidents, two of which were fatal because the sleeping drivers were unable to intervene and override the autopilot function.
In one case, a car drove straight into a fire engine parked in the road.
The research in which NTNU has participated shows that these drivers, who have the greatest confidence in the AI autopilot, in fact serve only to put a spanner in Tesla’s works.
This is because when Tesla owners with low levels of risk tolerance read about the accidents, they lose confidence in the AI tool and never use the autopilot function. This in turn removes major opportunities for the AI tool to learn.
Ask the risk averse first
As research scientists, we believe that many other enterprises with ambitions to utilise AI can learn some important lessons from the findings of this research. Not least in the health sector.
By far the most important lesson for all enterprises planning to utilise AI tools is that the role of pilot users should first be offered to the cautious and risk averse.
Such advice is in direct opposition to what has usually been practised when it comes to testing new digital systems.
The reason for this, of course, is that during the recruitment of pilot users (individuals or teams), whether in connection with AI or something else, it is always the risk-tolerant who put up their hands. In other words, those who are the most positive and optimistic on behalf of new technology.
Such people are typically also those who are willing to take chances during the testing process. It is exactly because of this that they are preferred in the role of pilot users.
Same pattern among doctors
Today at SINTEF and NTNU, we are studying medical doctors and engineers who are cutting their teeth on machine learning models. And we are seeing the same tendencies as recorded in the Tesla study.
Risk-averse users tend to steer away from AI when they encounter examples of tools making mistakes. Their initial scepticism is confirmed, and they lose the motivation to try again.
A probable key reason why it is so easy to ‘follow the lead’ of the sceptics is that people in general have a strong aversion to things they cannot explain or don’t understand.
‘Black boxes’ make us nervous
The problem with so-called subsymbolic AI (algorithms that recognise everything from shapes to complex patterns with the aid of a process called classification) is that it is perceived as a ‘black box’.
Research in the field of so-called Explainable AI, as conducted by workers such as Inga Strümke at NTNU, addresses this issue in particular. Findings from such work show that when something unforeseen happens, people need an explanation.
We accept that people in general, including doctors, make mistakes. This is because we can understand the reason for what has happened. However, we do not know how a subsymbolic AI tool has arrived at its classification. This makes us feel nervous and insecure.
AI can offer key assistance in the health sector
This is probably an important reason why, when ‘risk-tolerant’ people make fatal mistakes based on poorly founded AI-based advice, any cautiously established confidence held by others can be demolished in minutes.
We argue that society has to get a grip on this issue. Not least because of ideas published by the medical doctor Ishita Barua in her recent book ‘Kunstig intelligens redder liv’ (Artificial intelligence saves lives, in Norwegian). Barua also has a PhD in the use of AI in medicine. She argues that the health sector must take the plunge and utilise machine learning models because of the enormous potential they offer.
So, what can the health sector do to avoid ending up in the ‘Tesla trap’?
Key data from the risk averse
In our opinion, when it comes to testing AI models, ask less risk-tolerant doctors. Ask them to experiment with the tool – to look, learn and disseminate their experience without, at least at first, using AI-based advice in the treatment of their patients.
This approach has the potential to offer the models better and more reliable training data. The Tesla study tells us why. When the car’s autopilot AI did something peculiar, the ‘cautious’ drivers took the wheel and corrected for the error. This provided the model with valuable machine learning, enabling the AI tool to adjust its driving style.
In more recent versions of the Tesla autopilot, the AI tool has performed much better in situations similar to those in which it had been previously corrected.
But society risks losing benefits such as this if ‘cautious’ AI users are scared away by the mistakes made by techno-optimists.
