LSTM Based Speaker Diarization

Bachelor thesis research implementing LSTM and BiLSTM networks for speaker diarization — the task of labeling who speaks when in an audio recording. Compares d-vector generation architectures and four clustering algorithms (k-means, agglomerative, spectral, VBx) to find the best diarization error rate.

This bachelor's thesis project tackles speaker diarization — labeling who is speaking when in an audio recording — using LSTM and bidirectional LSTM networks to generate d-vectors, fixed-length embeddings that capture a speaker's voice characteristics from a short audio window. Rather than picking one clustering approach and stopping there, the project systematically compares four clustering algorithms — k-means, agglomerative clustering, spectral clustering, and VBx (variational Bayes HMM clustering) — against the generated d-vectors to find which combination of embedding architecture and clustering method minimizes the diarization error rate (DER) on the evaluation set, framing the result as an empirical comparison rather than a single fixed pipeline.

Architecture

graph TD
  Audio[("Audio recording")] --> Features["extract_features.py
(MFCC / spectral features)"]
  Features --> LSTM["LSTMModel
(nn.LSTM + linear projection)"]
  LSTM -->|d-vector embeddings| Loss["GE2E loss
(centroids + cosine similarity)"]
  Loss -->|trains| LSTM
  LSTM -->|inference| Embeddings["per-segment d-vectors"]
  Embeddings --> Clustering["clustering.py
affinity matrix + spectral/agglomerative"]
  Clustering --> Diarization["speaker-labeled segments"]

Under the hood

The model reduces each segment to a single d-vector embedding (last LSTM hidden state, linearly projected), trained with a GE2E loss that pulls embeddings toward their speaker's centroid and away from the most confusable other speaker.

model.py

Python

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, projection_size):
        super(LSTMModel, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=False)
        self.projection = nn.Linear(hidden_size, projection_size)

    def forward(self, x):
        _, (hidden, _) = self.lstm(x)
        d_vector = hidden[-1]  # Get the last hidden state
        d_vector = self.projection(d_vector)  # Apply linear projection
        return d_vector


def ge2e_loss(embeddings, labels, w, b, device, epsilon=1e-8):
    # Calculate centroids for each speaker
    unique_labels = torch.unique(labels)
    centroids = torch.stack([embeddings[labels == speaker_label].mean(dim=0) for speaker_label in unique_labels])

    centroids = centroids / (centroids.norm(dim=1, keepdim=True) + epsilon)
    embeddings = embeddings / (embeddings.norm(dim=1, keepdim=True) + epsilon)

    similarity_matrix = torch.mm(embeddings, centroids.t()) * w + b
    mask = labels.unsqueeze(1) == unique_labels.unsqueeze(0)

    positive_similarity = similarity_matrix.masked_select(mask)
    positive_loss = -torch.log(torch.sigmoid(positive_similarity) + epsilon).mean()

    masked_similarity_matrix = similarity_matrix.masked_fill(mask, -float('inf'))
    most_similar_incorrect = masked_similarity_matrix.max(dim=1)[0]
    negative_loss = -torch.log(1 - torch.sigmoid(most_similar_incorrect) + epsilon).mean()

    return (positive_loss + negative_loss) / 2