Machines That Listen

We spent a week with a small team of engineers building voice assistants that deliberately forget what they hear.

This is part two of our Quiet Tech series, about people who build technology that does less on purpose. Part one followed a man who repairs landline telephones in a garage. This one takes place mostly in a converted dairy outside the market town of Wenlow, where a company of five called Sill has spent three years on a single object: a speaker about the size of a bar of soap that answers your question and then forgets you asked it.

Fig. 2. Always on, always listening. Drawing by Almanac.

Designing for amnesia

Most listening devices are built to remember. Every request is stored, labelled and used to improve the next one. This team is doing the opposite: their device processes speech on the spot, answers, and then discards the recording entirely. There is no history to review because there is no history.

In practice this means the listening happens on a chip inside the device, with no connection to a server. The audio sits in a small buffer for as long as it takes to work out what was said and then the buffer is overwritten. Noor Hadley, who leads the team, showed us the function that does it. It is eleven lines long. Most of the last three years, she said, has gone into making sure nothing else in the device quietly keeps a copy.

Her version of the whole idea is one sentence, which she has written on more whiteboards than she can count. The most respectful thing a machine can do with a conversation is to let it end.

The most respectful thing a machine can do with a conversation is to let it end.

It is harder than it sounds. Ask it to set a timer for the pasta and it will, but it will not remember tomorrow that you always cook pasta at seven. Several people we spoke to outside the company thought this made the thing pointless. The team mostly agreed that it made it smaller, and seemed quite content with that.

The list of what it can do is short: timers, units, arithmetic and about four hundred recipes stored on the chip. It cannot tell you the weather, because finding out the weather would mean asking a server, and the server would then know that somebody in a kitchen near Wenlow wanted to know. Sill plans to sell it for £140, roughly three times the price of a cheap smart speaker. Nobody at the company pretends that isn’t a problem.

The people in the film

Noor came to the idea from hearing aids, where she spent eleven years designing the software that separates a voice from the noise of a restaurant. Nobody in that industry, she says, would dream of recording what a wearer hears.

The idea arrived, she says, at her father’s seventieth birthday, in a restaurant, watching him adjust a hearing aid she had helped to design so that he could hear his grandchildren over the next table. It occurred to her that the device in his ear knew more about that evening than anyone at the table, and had the decency to forget all of it. She left her job about a year later.

Tomasz Wierz builds the hardware: the soap-shaped case, turned from ash by a joiner in the next village, and the microphone shutter, a small brass slider on the back that physically covers the microphones when it is closed.

He keeps a drawer of perhaps two hundred switches, salvaged from radios, cars and one old lift. His rule is that a switch should make a sound you can hear from across a room, so that you know, without looking, that the microphones are covered. The final brass slider makes a short, dry click that Felix measured, out of curiosity, at a little over sixty decibels. Tomasz was very pleased with that number and mentioned it twice.

The design of silence

Elsie Brandt is the interaction designer, which at Sill mostly means deciding what the device does with light and silence. She came from museum audio guides, where she spent years working out how long a visitor will stand in front of a painting before they get bored of being spoken to (about forty seconds, she says, unless the painting has a dog in it). It answers in a plain, unhurried voice she describes as “a reasonably competent neighbour”, and then it stops.

The question on the whiteboard, the week we filmed, was whether a user should be able to see the forgetting happen. Elsie’s answer is a thin ring of light that fades over two seconds once an answer is given. Tomasz thinks any signal at all is a kind of performance. They do not resolve it on camera. (By the end of the week the light was in, but dimmer.)

The empty room is a test Noor runs on every new build. The device is left on for three hours in a room with nobody in it, and the team counts how many times it wakes up. On the Thursday we filmed, it woke twice: once for a pigeon on the window ledge, and once, as far as anyone could tell, for the fridge in the next room.

Felix at home

Felix Oduya is seventy-one, a retired sound engineer who spent thirty years recording radio drama, and the company’s only tester who is not on the payroll. He takes each new prototype home and lives with it for a fortnight. He is paid, as far as we could establish, in coffee beans.

He has a radio that has been on, he estimates, since about 1985. That radio is the reason the device now copes with background speech, because Felix refused to turn it off for testing on the grounds that nobody else would either. During the Sunday lunch we filmed, he asked the prototype forty-seven questions and kept a tally in pencil on the back of an envelope. It misheard him six times. Three of those were “thyme”.

He is the only person at Sill who openly misses what the device won’t do. For most of last winter he asked it to convert gas mark five into Celsius every Sunday, and every Sunday it answered as if he had never asked. He found this, he says, “half restful and half insulting”. But he keeps taking them home. When I asked him why, over the washing up, he put it plainly. He spent thirty years making sure microphones caught everything, and it is a relief, at the end of all that, to work with one that is trying not to.

The friendly sceptic

We also spoke to Hannah Corwen, who researches speech recognition at the University of Esk and describes herself as a friendly sceptic. She thinks the device is admirable and that most people will not pay for it. Both of those may be true.

Her objection is more specific than the usual one about convenience. Memory, she says, is where accuracy comes from. A device that learns from what it hears gets better at your voice, your accent and the names of your children. A device that forgets is stuck, for ever, at whatever it knew when it left the factory. That is less fine for someone with a strong regional accent, or a stammer, or speech that has been changed by a stroke. “Five people in a dairy is not a test population,” she said, which the team did not enjoy hearing but did not dispute.

On money she was blunter. Privacy, in her experience, is something people say they want in surveys and then decline to pay for at the till. She would like to be wrong, but doesn’t think she is.

When we put this to Noor, she agreed with the first point and has since started working with a speech therapy clinic in Carrow, testing with forty volunteers in supervised sessions where, she stresses, nothing is kept. On the second point she said Hannah was probably right, and then went back to her notebook.

From the interview

This is a shortened excerpt from our interview with Noor, recorded in the dairy’s old cold room, now the quietest room in the building.

AMARA: Most people would say they’ve got nothing to hide. Why does it matter if the machine remembers?

NOOR: Because it isn’t about hiding. When you speak in your kitchen you are not publishing anything. You’re talking. You might say something cross about your sister, or practise a difficult phone call out loud, or sing badly. The whole assumption of a room is that what’s said in it goes when you leave. We built machines that broke that assumption, and then we asked people to feel relaxed around them.

AMARA: But the remembering is what makes them useful.

NOOR: Some of it. A lot of it is what makes them useful to the company that sold them to you. I’m not against memory. I’m against memory you didn’t agree to and can’t see. If someone wants a notebook, I’ll sell them a notebook. They’re very good.

AMARA: What happens if nobody buys it?

Then we’ll have made one good thing that a few hundred people own. And the eleven lines are public. Anybody can use them. If one bigger company copies how we empty the buffer, I’d count that as a win, though I’d be furious for about a week.

Noor Hadley, in the dairy’s old cold room

When you speak in your kitchen you are not publishing anything.

Chapter notes

The workshop and the kitchen

The people and the argument

Credits and thanks

With thanks to everyone at Sill, to Felix Oduya for the roast potatoes, and to Hannah Corwen for disagreeing with us so pleasantly.

This is part two of our Quiet Tech series. It runs eighteen minutes and is best watched somewhere without a smart speaker in the room.