Pinchas Gutter survived a Nazi death camp – and now his story will live on through a hologram that can answer your questions.

Pinchas Gutter goes out of his way to find me biscuits. In a sun-baked living room in his north London home, he opens a packet of Rich Tea, sits down and tells me about the Holocaust.

Gutter was seven years old when the second world war broke out. He lived in the Warsaw ghetto for three and a half years, took part in its uprising, survived six Nazi concentration camps – including the Majdanek extermination camp – and lived through a death march across Germany to Theresienstadt in occupied Czechoslovakia.

“Remembrance is the secret of redemption, while forgetting leads to exile,” he says, quoting the Baal Shem Tov, the founder of Hasidic Judaism.

“For me this whole thing is about not forgetting. Because if you forget, you repeat it over and over again. To me, the great importance of testimony is not to forget what people are capable of.” Sitting opposite him here in Hampstead, meeting his eyes and hearing his voice, the idea of forgetting this extraordinary story seems impossible.

One week later I am staring into Gutter’s eyes again. But these eyes are on a screen in the Alternate Realities strand of Sheffield’s international documentary festival. The version of Gutter projected on the monitor is a prototype for a responsive hologram that will be wheeled into classrooms, lecture halls and museums. The idea is that the audience asks questions and pre-recorded memories from Gutter will respond – much as if you’re talking to the real person. This virtual Gutter meets my gaze and tilts his head when I speak into the microphone.

The project, called New Dimensions in Testimony, was thought up by concept developer Heather Maio, and was made in a collaboration between the University of Southern California’s Institute for Creative Technologies (ICT) and the Shoah Foundation – an organisation dedicated to making and archiving interviews with survivors and witnesses of genocide.

The prototype involved filming extensive interviews with Gutter using an array of cameras and a specialised light stage. The team at ICT then used natural language processing software to help create an interactive version of the video footage, with vocal cues triggering responses from the pool of recorded speech.

Complex algorithms dictate which responses are played back – faintly reminiscent of virtual assistants such as Apple’s Siri and Microsoft’s Cortana, except that while those pieces of software are primed to answer your questions about emails or the weather, New Dimensions in Testimony is based on communicating one man’s experience of genocide.

We don’t consider it an AI. I think of it more like a smart search engine which gives you a conversation-like experience

“We wanted the visitor to experience the discussion and what that means to them,” Maio tells me. “Not the technology that goes around it and makes it work. In fact, we didn’t want them to think there was technology around them at all.”

Maio says 12 Holocaust survivors have so far been interviewed in this manner, adding that there are plans to advance the intelligence of the computer learning – giving the program the ability to track what the recording has said and tailor the order of its responses to fit. Directing the content in this way enhances the sense of talking to a real person, but it also raises questions about authenticity.

When algorithms are choosing the answers and the order in which they are spoken, are the responses those of Gutter or of an AI?

“In a way it’s both,” says David Traum, director for natural language research at the ICT. “We have to distinguish two concepts of response. Everything that is said comes directly from Pinchas. However, the decision of which recording to present as a response to a specific question is made by the technology, based on its training data of how Pinchas and other people have answered questions. In the best cases, the answers are exactly how Pinchas himself would have responded. In some cases, it may seem that he is addressing a somewhat different question, but the response itself is still his.”

Instead of AI, Stephen Smith, executive director of the Shoah Foundation, says we should consider New Dimensions in Testimony as a voice-activated search engine. He emphasises that no new content has been added beyond that in the 25 hours of video footage with Gutter.

“As time goes by and many more users use the content, machine learning will enhance its intelligence – it will understand a wider range of questions as each interaction helps train the system,” he says. “We still don’t consider it an AI, though, because AI assumes the machine is thinking independently. I think of it more like a smart search engine which gives you a conversation-like experience.”

That conversation-like experience is the crux of New Dimensions in Testimony. The project is winning many plaudits including the interactive award and the audience award at the Sheffield documentary festival this week. Smith says audience members have applauded the virtual Gutter, thanked him, argued with him.

“One woman apologised to him for the negligence of the world, telling him she would have willingly traded places with him had she been alive then.”

My personal experience with the project hinged on eye contact. Having Gutter staring at me as he answered my question on the Warsaw uprising, or about whether he still believed in God, was affecting on an instinctive level. Though I was aware that this was an assortment of clips directed by speech recognition software, it was hard not to feel the impression of an interaction in real time. One comparable forthcoming project in this sense is Anne, a VR film of Anne Frank, which will place viewers in a version of the annex where she spent two years hiding from the Nazis.

While New Dimensions in Testimony aims to represent a person’s individual testimony, Anne aims to represent a particular environment.

Both want to keep the memory of the Holocaust alive through interaction, imparting a sense of tangibility on to events rapidly passing from eyewitness testimony. While there are pressing ethical questions – not least the danger of representing real situations as a form of immersive entertainment – previous projects have shown serious scope for documentary use of VR.

Nonny de la Peña’s Project Syria recreates a rocket attack in the Aleppo district of Syria to shocking effect; the National Theatre-produced Home: Aamir is a 360° video account of life in the Calais Jungle camp; Jane Gauntlett’s Dancing with Myself subtly and effectively communicates the experience of living with epilepsy; and the Guardian’s own 6×9 combines VR with more traditional testimony about the effects of solitary confinement.

These projects succeed largely due to their focus on upsetting the viewer’s sense of comfort. Instead of easing you into a cosy virtual playground, these experiences jolt another world into your own. Elsewhere in the Sheffield festival is an example taken to its extreme. The Enemy, an augmented reality piece directed by war photographer Karim Ben Khelifa, involves holding an iPad and seeing Palestinian and Israeli fighters projected on to the surrounding floor. The avatars follow your gaze as you walk around, and speak to you as you approach. The audio responses are all taken from actual interviews, and the 3D figures are modelled on the interviewees.

Once again, the power of the experience comes from the sense of physicality. Like interacting with New Dimensions in Testimony, eye contact and body language communicate something unlike film documentary. And yet, as with the virtual Gutter, there’s also the question about representation. When you make a version of a person that can watch, listen and respond, do you ultimately do more than show a recording of them in a particular moment of history? Do you have a responsibility to show them as a whole person?

“VR invites us to break taboos around how close we get to people, how directly we observe them,” says Mark Atkin, curator of the Alternate Realities programme at the festival. “It has no code of conduct – that might have to come. And it has no rating system – that will come. For now, the honour system seems to be working well but once the technology to create this sits on all of our phones and anyone can manipulate a 3D scan of a human and put it in any environment, a new form of abuse could well emerge. We need to imagine these futures now, in order to be able to deal with them if and when they do happen. »

The team behind New Dimensions in Testimony has been rigorous in its conduct about recording and representing its subject. The responses given are all clear in context, and having spoken to both the real Gutter and the recording, I didn’t feel that he was misrepresented. But when Gutter and his generation are gone, that point of comparison will be harder to define. The Shoah Foundation Institute is aware of its responsibility to portray Gutter truthfully, but when I speak to Maio she accepts that other organisations might not be as meticulous. In Hampstead I ask Pinchas Gutter how it felt to see the virtual version of himself.

“I saw myself sitting there, moving my hands and legs, being a presence and speaking to people,” he tells me. “Not just speaking but answering. In other words, I’m listening. I’m not just speaking, I’m listening.”

Whether it is Gutter listening or a computer, the most important thing for the man who survived the Holocaust is that the outcome stands as evidence that these events happened, and they happened to a real human. Gutter tells me he didn’t talk about the Holocaust at all in the years following the war, but when he did finally speak, he found it cathartic. When he is no longer alive to speak to audiences, he wants that two-way process to carry on.

“It helped me alleviate some of the pain and the difficulty of relating to people,” he says. “I tell them when they speak to me: now you have a piece of me and I have a piece of you. We’re joined together now, in both the good and the bad that can happen.”

Can I give a piece of myself to a hologram? Can it give a piece back?

Perhaps not. Perhaps there’s no way to replicate the power of talking to another body – a body that stands as physical testimony to the events it lived through. Projects such as New Dimensions in Testimony are trying though, and Gutter tells me it’s the closest thing we have.

“To me, this is such a vitally important project. It’s beyond value. There’s almost something holy about it.”

• New Dimensions of Testimony won both the Alternate Realities interactive award and audience award at the 2016 Sheffield Doc Fest

• This article was amended on 21 June 2016 to clarify the quotes of David Traum and Heather Maio.