---
title: "Transcript: Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI"
category: "transcripts"
videoId: "mFLlVpnGpds"
sourceLabels: ["YouTube transcript", "Cached transcript markdown"]
wordCount: "4117"
---

# Transcript: Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

## Source Video
- [YouTube](https://www.youtube.com/watch?v=mFLlVpnGpds)

## Local Cache
- `raw/sources/youtube-transcripts/mFLlVpnGpds.txt`
- 4,117 words

## Transcript

Uh good morning everyone. Thanks for being here to the voice and vision session. Uh so I'm Hervé Bredin, chief science officer and co-founder at PyAnnote. Uh so I'm going to talk to you today about um conversations, understanding conversations, and what you can do on top of transcription. Uh so a quick word about myself. So I've been an academic researcher all my life until two years ago when I started this company. Oops, sorry. Uh what happened? Yeah. Uh so basically yeah, I worked on this topic called speaker diarization which I'll introduce a bit later for those of you who don't know this word word that is tricky for me to pronounce.

But basically over the years I built an open source toolkit called PyAnnote which focuses on speaker diarization and that became quite popular over the years in particular since OpenAI released the Whisper speech-to-text open source models. Basically Whisper was some kind of revolution in terms of STT, the fact that it was free, that it was very good, and but it didn't provided the actual didn't provide the actual names and tags of the speaker. So people naturally turned to PyAnnote to combine the two. And you can see that on the inflection points on the GitHub star history.

And just a few words, I happen to turn 45 uh uh in just one week from now and we all almost at 10K stars on the GitHub. So please give me a birthday gift by just going to the website and to the GitHub website and add your star. That will make my birthday. Um, so uh let's go to the to the core of the presentation. So you all know what transcription or speech-to-text is. Basically, that's answering the question what was said from the recording or streaming of a conversation. Basically, you get from the audio and you get a sequence of words as output. That's for instance what Whisper does.

But without the actual names of the speaker, the usually the conversations on the transcripts are not really understandable. So the next step after transcription is basically to attribute speaker tag to each of the words. So that's I call that here speaker attributed transcription. And so that that answer the question who said what. But for some application that's enough like for instance for maybe for meeting note takers to make a summary of a of of a meeting that might be enough to assign an action point to the right people to to know exactly that Air V said that and John said said this this other thing.

But we to understand the conversation a bit better, we we might want to go slightly better. Sorry, I forgot about this slide. So basically there are many cases where knowing who said what is as important as what was said actually. So I have here a few examples like for instance video dubbing, automatic video dubbing. Knowing who said what is actually really important to put the actual uh uh uh good correct voice to the right speaker. So if you want to automatically translate a video from one language to another, you want the voices to be consistent as well. So for that you need to know who's who who spoke when.

Same for meeting note takers and there are other applications like for instance podcast what I call podcast intelligence like being able to track a speaker across different episodes of a podcast or across different podcasts finding the same guest across multiple podcasts. But yeah, so I was a bit over just before but just knowing who said what is sometimes not enough to really understand the conversation the conversation how it goes. So knowing who said what and when actually brings even more information. For instance in this example without knowing exactly when each words were pronounced you can't detect that actually the black speaker interrupted the green one.

You can't really detect that maybe this small back channel that the green sorry the black speaker does during the speech of the of the green speaker if you miss it basically you you don't really understand that maybe the black speaker is actually agreeing noting at what the green speaker says. So knowing exactly when this small word has been pronounced is very important. And same without precise timestamps you can't really know whether I made a I make a a pause in between two speech turns which might convey additional information about my state of mind my I don't know the point that I'm trying to to make in the conversation.

And we can go even a step further. Like we we might want to know who said what and when and how they said it. Like I have a few examples here if some of you are laughing at some point is it because I don't know I said something stupid or I was actually funny. Same for coughing maybe there's something in the conversation that makes me uncomfortable and I'm coughing for some reason to to hide behind that. And yeah so I also mentioned stress disfluency prosody um fact that I uh stress on particular words in a sentence might actually change completely the meaning of the sentence.

Uh I have this example in mind like uh I don't know in the sentence the dog uh ate uh the cake. If uh I don't know you're talking to a child and the the child tells you the dog ate the cake or the dog ate the cake or the dog ate the cake uh the the meaning might be slightly different. So uh stressing particular words, uh being able to have a a voice AI actually understand the stress and and these kind of of low-level details might bring some more uh information to the to any downstream LLMs or or whatever tools that you use after this first enriched transcription step.

And if we take even a step a step back uh um knowing who's uh talking to who uh am I uh here in this example and probably can be seen as these uh gray speaker addressing to all of you. But at some point if uh at the end of this talk I uh you do have questions then uh I will be probably the the green one talking to one particular person answering the particular question. So knowing that, having a higher-level view of the conversation uh also opens up a new bunch of applications.

And all of this uh also happens in uh acoustic environment that gives you information about the context whether it's in this quiet room uh for now uh or if I'm in the street uh this bring more context to to to the game and you can actually uh take a better decision about what's going on, why a person is actually saying what what they are saying and and etc. So all of this is basically the what we are trying to work on at PianoTel AI, but we uh uh, started with a uh uh, smaller problem called a speaker diarization and that's really what I want to talk about, uh, today.

So, speaker diarization is the, I, I, yeah, is is usually, uh, what I what I wanted to say about this slide is that, uh, who said what is usually just as important as what was said and you can tell by instance by going through, uh, to the hugging face, uh, uh, model repository and filter, um, all the models there uh, with the ones that have the audio tag. Uh, where's my mouse here? Here. So, this is what I did and then you sort them by, uh, by number of downloads.

And basically, uh, the among the the top, uh, here seven models that, uh, are at the top of this list, three of them, the the first three are actually related to speaker identity and speaker diarization and the the the other ones are STT. So, that's one way of showing that, uh, speaker diarization is actually quite, uh, quite important on on in the community. So, uh, going uh, into a bit more detail about what speaker diarization is. So, as I said, it's answering the question who speaks when. So, starting from the recording of a conversation, uh, basically the first step that people usually do is, uh, basic voice activity detection.

So, you basically tell whether someone is speaking at at one time or not. Anyone is speaking at one time or not. And then you can go a step further. Uh, you can actually segment those speech regions into smaller speech turns. Um, so this is what I call segmentation and it includes finding, uh, speaker change points between, uh, in into those longer in those long speech regions, you might find, uh, you know, a speaker change point and, uh, also find regions where, uh, people are actually interrupting each other like, uh, in, uh, in the first example here.

There's definitely here someone interrupting, uh, someone else and it's possibly hear some kind of back channel where someone is actually nodding or saying "Mhm, okay." This kind of stuff. And this is the kind of small speech turn that uh you don't want to miss because sometimes they actually convey the most important information of the conversation. If it's a small yes here uh and if you miss it uh then you have no idea what's the the state of mind of the speaker. But that's not yet speaker diarization. Speaker diarization goes all the way to actually assigning a speaker identity to each speech turn.

And so here in this example, the the speaker diarization system automatically detected that there are two speakers, the green one and the black one. Um but we usually don't have any prior knowledge on the actual number of speakers in the conversation. You might have some kind of guidance on I don't know if if you are meeting a note taker for instance, you you have the list of attendees.

So you might have an idea of the number of people that were invited but that doesn't mean that uh someone two people joined from the same channel or or that an attendee that was not invited actually joining finally and uh so that makes the problem uh kind of difficult in the sense that we don't know in advance the number of classes that we're supposed to detect contrary to other let's say classical machine learning problems. And also we don't know the identity of the speaker that we're supposed to diarize.

In the sense that speaker diarization does not really output John, Hervé, and and and Jack but basically it's more like speaker one, speaker two, speaker three and they can be permuted and it's still correct. For example, in this example if I just change green into black and black into green, the diarization is still exactly the same from the point of view of the evaluation metrics that I'm going to talk about and uh uh from the point of view of the actual speaker diarization task.

So, that makes the uh this problem kind of difficult and that's why even though uh the community has been working on this topic for a long time now, it's still not solved. Uh there are other reasons like the fact that we have to detect overlapping speech to handle overlapping speech, uh that we have to uh uh take into account very short speech turns, that don't have to take into account imbalance between the speech time of multiple speakers in the in the conversation. All of that makes the problem difficult. On top of the usual acoustic conditions uh problems and everything that any speech processing uh um uh task have to deal with.

So, uh I'm going to switch to a demo to to to show you a a bit more how the diarization system are evaluated. Um maybe when you you you stumble in into benchmark, you you find this DER acronym, which is a stands for diarization error rate. And uh so, hopefully um this will work. So, uh I have here a Python notebook uh that I've prepared for for this talk. It's actually already available on GitHub. I'll show you the the QR code at the end of the presentation, so you can play with it. But, so in this example, I have a conversation between two women talking over the phone. >> Hello. >> Oh, hello.

I didn't know you were there. >> Neither did I. >> I thought, you know, I heard a beep. This is Diane in New Jersey. >> And I'm Sheila in Texas, originally from Chicago. >> Oh, I'm originally from Chicago, also. I'm in New Jersey now, though. >> Okay, so you you get the idea. A 30-seconds conversation between two women. So, this is the expected output of a perfect speaker attribute either transcription. This was manually labeled. And now what I'm going to to to do this here is actually run piano open source community one model on this same file. So it's actually running right now on my Mac.

So that's why here I'm using this MPS PyTorch backend. So how it works is that you first download the model from hugging face with this line of code and simply apply it on on the audio file. So now it's finished and there's the this prediction variable that contains um uh sorry, I should have done that on the other tab because this was the one that was already processed. I wanted to do it live. So let's do it live now. So it's running. And so the the next step is to actually visualize the errors that this system made. So at the top here you have the reference annotation.

So the expected output of diarization and here is the output of community one uh speaker diarization uh pipeline uh that is open source and free to use. So it makes three kinds of mistake as I was saying like a confusion here it got the speaker wrong. Uh it can also make uh false alarms. Uh so false alarm is when it detects speech when there's actually no speech in the ground truth. And misdetection is the other way around. And misdetection can actually uh happen for instance here during overlapping speech when we actually detect only one of the two speakers here.

And then once we have these basically errors of false alarm, confusion, and and misdetection we we can actually compute the diarization error rate. Uh so this is with this library called PyAnnotate metrics. And basically it gives us in this example 5% diarization error rate which is basically the sum of the confusion uh uh false alarm and misdetection divided by the the total duration of uh of speech in in this in this file. And so we have this other model that I'm currently running. Hopefully it work it works. So it's running right now on our cloud API. Basically it's a a better model that community one that we call precision two that is a running here.

Hopefully it will work but as for every demo it might fail. If no it worked. And this is the kind of output that you can get. So you see that it got the speaker right here and makes slightly less mistakes in terms of false alarm and misdetection. And overall in this example I think we got a 3% diarization error rate here. So let me switch back to the to the presentation. So this was about benchmarking speaker diarization model and I'm often asked how well does state of the art speaker diarization works today. And it's a difficult question to answer because really depending on the use case it might vary completely.

For instance in this example we have a here conversation telephone speech like the one we just listened to like two person over the phone. And we we can go down to 8% diarization error rate. The best system does that. But if if you are not in a restaurant with many friends with lots of background noise even the best system reaches like 41% diarization error rate. So it's far from being a solved problem but we are working on that. And so once you have speaker diarization and you have transcription shouldn't be that hard to actually find the speaker attributed transcription right?

It's just a matter of you know assigning a word to a speaker speaker to a word. But actually that's not that easy. And I'll there there are many reason why that's not that easy. The first reason is that most speech to text models are actually trained on single speaker data. As soon as you apply them on multi-speaker data with overlap, with speaker change, with all kind of mess, they completely they fail miserably.

So, for instance, when you look at the open ASR leaderboard from the Hugging Face, if we look for instance at NVIDIA Parakeet that we are using that I will be using in the in the demo right after that, they report 11.4 % word error rate. When we apply the very same model on the same AMI um on on our side, we we get 26%. So, the question is why is there a difference? Actually, it's in the way that those benchmarks are set up here on the uh uh for for example, in this AMI data set, that's a meet uh data sets with meetings uh between four to five people, I think, in meeting rooms.

And they are basically microphones like the one I'm I'm having here, so headset microphone, as well as one microphone in the middle of the table. And and those numbers here uh on the open ASR leaderboard are based on the headset microphone. Those numbers here uh are based on the microphone that is in the middle. And so, on one side you have a single speaker speech, on the other side you have multi-speaker and even distant microphone speech. So, that's why it degrades uh a lot. And so, that that's that was my point of the of this slide.

So, when doing speaker attributed transcription, the um the reason why it it might go wrong is either because STT doesn't work great and uh it's usually the case that they don't generalize uh very well to multi-speaker recordings. For, as I was saying, distant microphone, uh speaker change, crosstalk, interruptions, you name it. Uh maybe code-switching as well when you change language in the middle of a sentence. But also, it's it might uh be because of the actual reconciliation between diarization and STT timestamps.

Though it may sound uh uh uh obvious how to do that as I was saying, I will hopefully show in this live demo once again that it's it's not such a an easy problem. Because uh STT does not transcribe overlapping speech well because the timestamps disagree between STT and diarization, and because sometimes diarization will detect speech that the transcription will not transcribe and the other way around. So, um second part of the demo about speaker attributed transcription. So, uh first I'm going to apply this Parakeet model from Nvidia that does a by the way a great job at the transcribing uh uh this sample. And so, Parakeet gives us this kind of output.

So, we have this sequence of words and for each word we actually have the corresponding timestamps. Okay? So, I can play it quickly. >> Hello. Oh, hello. I didn't know you were there. >> Neither did I. >> Okay, then. I thought, you know, I heard a beep. This is Diane in New Jersey. Um >> Well, you get the idea. And then, um the question is um now we have our best diarization. This is Precision-2 output here. This is Parakeet output here. And the question is now we need to assign a speaker to each word. So, let's do it step by step for instance.

So, in this example, even though the timestamps of the the words are a bit shifted on the left here, it's obvious that it's probably the yellow speaker who spoke the the gray word. But if you go just the third word. So, now we have this. So, we have this word the word O here that is in between those two speech turns according to diarization. Which one do you assign it to? Um so we we I I can listen to it. >> hello. Oh hello. >> Why? It's It's not quite quite sure. And there are many other uh position where it happens here same.

There's a word here, okay, that apparently according to diarization there are two speakers speaking and only only one word from the from the transcription. So that that's why at the Pianoteq we came up with this new STT orchestration thing that basically does its magic and and does the job for you. So here I I just submitted to our cloud API um a a job to transcribe and and and use both precision two diarization model and Parakit transcription model and take care of the what I call this reconciliation between the two. And you end up with this kind of with this kind of output.

Uh what's nice is that so now you it did the the job for you, but in particular I I like to go to this particular example even where there was overlapping speech before it actually managed to uh interleave the the words from the two speaker. Let me play the just this part and focus on this area maybe. >> in New Jersey. And I'm Sheila and tech. >> And there's actually a end uh that this is the um here that is actually uh the the yellow lady actually has this um at the end of the speech and the the blue one actually interrupts her.

But they they they're actually overlapping and we managed to get them right. Um yeah, so this is the end of my talk. I wanted to give you a few minutes. I have two minutes left for questions, but I just wanted to say that this demo that I did is already available on the in the Pianoteq tutorials GitHub repo. So in there you you'll be able to to with Pianoteq open source toolkit for diarization with piano metrics for evaluation. Uh with i-piano for this nice visualization widget that we've implemented. I stands for not iOS, iMac, or whatever, but really for interactive. And also the SDK to to play with our premium models.

And I'll stop here and happy to take few questions. We have one more MINUTE LEFT. YES, PLEASE. >> SO, WHAT'S THE TRICK THAT you use to resolve the conflicts? >> So, >> Is it part of the model training or is it some sort of like heuristics that you have around to identify who to attribute the word to? >> Yeah, so the question is what's the trick to actually solve this problem of a reconciliation. So, that's proprietary trick, but what what I can say is that we there's already part of this trick that is available in the community one model, which we call exclusive diarization.

Basically, what we do is that we find a way when there is overlap to actually select the most likely of the two speaker that will be transcribed by the STT model. So, that simplifies actually the reconciliation between the two. So, that's one major part of the of the approach. >> The model training is like a couple of heuristics you have to >> Yeah, so so the question is about I'm repeating because I've been told I need to repeat questions. Uh so, it's not part of the model training in the sense that we really plan to support any kind of STT without having to change the STT model itself.

So, really it's supposed to work with any STT, even fine-tuned ones that you might have internally for because you you fine-tuned them for your particular use case and nobody has it. You can combine it like that. I 18 seconds, so I guess we need to stop. I'm sorry, maybe we can talk offline, otherwise they'll beat me, I guess. Thank you very much. I'm sorry.