Some of the world’s biggest venues, including Manchester Arena, use gun scanners that are “unable” to detect some large knives.
Evolv, a company that sells artificial intelligence (AI) scanners, claims they can detect all weapons.
But documents shared with BBC News by research firm IPVM suggest they may not recognize certain types of knives, as well as some bombs and components.
Evolv told BBC News it had informed venues of all “capabilities and limitations”.
Marion Oswald, from the government’s Center for Data Ethics and Innovation, told BBC News: “There needs to be more public information and a more independent assessment of these systems before they are rolled out in the UK.
“At the end of the day, they may replace established methods of metal detection and physical searches.”
AI and machine learning allow the scanners to create unique “signatures” of weapons that distinguish them from items like computers or keys, Evolv says, reducing manual checks and avoiding long queues.
“Metallic composition, shape, fragmentation — we have tens of thousands of these signatures for every weapon that’s out there,” Chief Executive Peter George said last year, “every weapon, every bomb and every big tactical knife.” “
For several years, independent security experts have questioned some of Evolv’s claims.
The company has previously refused to let IPVM test its Evolv Express technology.
But last year it gave permission to the National Center for Spectator Sports Safety and Security (NCS4).
NCS4’s public report, released earlier this year, gave Evolv a score of 2.84 out of three – detecting many types of weapons 100% of the time.
But a private report was also produced, obtained via a freedom of information request from IPVM and shared with BBC News along with emails between Evolv and NCS4.
And there was Evolv’s ability to spot large knives, a score of just 1.3 out of 3.
In 24 walkthroughs, Evolv Express failed to detect large knives 42% of the time.
“The system was unable to detect every knife at the sensitivity level observed during the exercise,” the report said.
“Recommend full transparency to prospects based on the data collected.”
IPVM’s Conor Healy said: “With certain categories of knives, the system did not recognize them at all when they were brought through. And that totally contradicts what Evolv has said to the public.”
NCS4’s report doesn’t say, and BBC News doesn’t report, what types of large knives the technology failed to detect
And for security reasons, BBC News does not report further details on the documents’ suggestion that certain types of bombs and their components may not be detected either.
ASM Global, which owns Manchester Arena, said the deployment of Evolv Express was the “first such deployment at an arena in Europe” and plans to expand the technology to other venues.
In 2017, a man detonated a bomb at an Ariana Grande concert in the arena, killing 22 people and injuring hundreds more, including many children.
When asked if the conclusions of the private report had been shared, an ASM Global representative told BBC News he did not want to comment on security issues.
Emails received from IPVM show that Evolv employees were allowed to make “tracked changes” to the report – deleting certain sections.
In a January 19 version, the ending “Knives were not consistently recognized” was deleted.
An Evolv employee using “track changes” also deleted an indication that the system was “unable to recognize each knife” and one on the 1.3 score.
When asked why Evolv was able to edit a report marked as independent, NCS4 told BBC News it “did not allow Evolv to edit the report directly”.
“The ‘track changes’ feature was used to collect feedback,” said an official.
And NCS4 “stands by its process, which has proven effective in informing and educating solution providers and practitioners.”
Evolv did not dispute the conclusions of the private report.
“We work closely together to communicate sensitive safety information, including our system’s capabilities and limitations, to enable safety professionals to make the most informed decisions for their particular venue,” an official told BBC News.
“We believe that providing a blueprint on how to bypass the security screening process and technology to the public will result in the venues being secured by our clients being less secure.”
The NCS4 report is important because very little is publicly known about how well the Evolv technology is performing.
Evolv told BBC News it had also been tested by the Center for the Protection of National Infrastructure – a body affiliated with the Home Office.
But when asked by BBC News, the Home Office refused to confirm or deny it.
Add Comment