Agent skill

Automatic Speech Recognition (ASR)

Transcribe audio segments to text using Whisper models. Use larger models (small, base, medium, large-v3) for better accuracy, or faster-whisper for optimized performance. Always align transcription timestamps with diarization segments for accurate speaker-labeled subtitles.

View SKILL.md on GitHub Repository
Stars 897
Forks 232

Install this agent skill to your Project

npx add-skill https://github.com/benchflow-ai/skillsbench/tree/main/tasks/speaker-diarization-subtitles/environment/skills/automatic-speech-recognition

SKILL.md

Automatic Speech Recognition (ASR)

Overview

After speaker diarization, you need to transcribe each speech segment to text. Whisper is the current state-of-the-art for ASR, with multiple model sizes offering different trade-offs between accuracy and speed.

When to Use

  • After speaker diarization is complete
  • Need to generate speaker-labeled transcripts
  • Creating subtitles from audio segments
  • Converting speech segments to text

Whisper Model Selection

Model Size Comparison

Model Size Speed Accuracy Best For
tiny 39M Fastest Lowest Quick testing, low accuracy needs
base 74M Fast Low Fast processing with moderate accuracy
small 244M Medium Good Recommended balance
medium 769M Slow Very Good High accuracy needs
large-v3 1550M Slowest Best Maximum accuracy

Recommended: Use small or large-v3

For best accuracy (recommended for this task):

python
import whisper

model = whisper.load_model("large-v3")  # Best accuracy
result = model.transcribe(audio_path)

For balanced performance:

python
import whisper

model = whisper.load_model("small")  # Good balance
result = model.transcribe(audio_path)

Faster-Whisper (Optimized Alternative)

For faster processing with similar accuracy, use faster-whisper:

python
from faster_whisper import WhisperModel

# Use small model with CPU int8 quantization
model = WhisperModel("small", device="cpu", compute_type="int8")

# Transcribe
segments, info = model.transcribe(audio_path, beam_size=5)

# Process segments
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

Advantages:

  • Faster than standard Whisper
  • Lower memory usage with quantization
  • Similar accuracy to standard Whisper

Aligning Transcriptions with Diarization Segments

After diarization, you need to map Whisper transcriptions to speaker segments:

python
# After diarization, you have turns with speaker labels
turns = [
    {'start': 0.8, 'duration': 0.86, 'speaker': 'SPEAKER_01'},
    {'start': 5.34, 'duration': 0.21, 'speaker': 'SPEAKER_01'},
    # ...
]

# Run Whisper transcription
model = whisper.load_model("large-v3")
result = model.transcribe(audio_path)

# Map transcriptions to turns
transcripts = {}
for i, turn in enumerate(turns):
    turn_start = turn['start']
    turn_end = turn['start'] + turn['duration']
    
    # Find overlapping Whisper segments
    overlapping_text = []
    for seg in result['segments']:
        seg_start = seg['start']
        seg_end = seg['end']
        
        # Check if Whisper segment overlaps with diarization turn
        if seg_start < turn_end and seg_end > turn_start:
            overlapping_text.append(seg['text'].strip())
    
    # Combine overlapping segments
    transcripts[i] = ' '.join(overlapping_text) if overlapping_text else '[INAUDIBLE]'

Handling Empty or Inaudible Segments

python
# If no transcription found for a segment
if not overlapping_text:
    transcripts[i] = '[INAUDIBLE]'
    
# Or skip very short segments
if turn['duration'] < 0.3:
    transcripts[i] = '[INAUDIBLE]'

Language Detection

Whisper can auto-detect language, but you can also specify:

python
# Auto-detect (recommended)
result = model.transcribe(audio_path)

# Or specify language for better accuracy
result = model.transcribe(audio_path, language="en")

Best Practices

  1. Use larger models for better accuracy: small minimum, large-v3 for best results
  2. Align timestamps carefully: Match Whisper segments with diarization turns
  3. Handle overlaps: Multiple Whisper segments may overlap with one diarization turn
  4. Handle gaps: Some diarization turns may have no corresponding transcription
  5. Post-process text: Clean up punctuation, capitalization if needed

Common Issues

  1. Low transcription accuracy: Use larger model (small → medium → large-v3)
  2. Slow processing: Use faster-whisper or smaller model
  3. Misaligned timestamps: Check time alignment between diarization and transcription
  4. Missing transcriptions: Check for very short segments or silence

Integration with Subtitle Generation

After transcription, combine with speaker labels for subtitles:

python
def generate_subtitles_ass(turns, transcripts, output_path):
    # ... header code ...
    
    for i, turn in enumerate(turns):
        start_time = format_time(turn['start'])
        end_time = format_time(turn['start'] + turn['duration'])
        speaker = turn['speaker']
        text = transcripts.get(i, "[INAUDIBLE]")
        
        # Format: SPEAKER_XX: text
        f.write(f"Dialogue: 0,{start_time},{end_time},Default,,0,0,0,,{speaker}: {text}\n")

Performance Tips

  1. For accuracy: Use large-v3 model
  2. For speed: Use faster-whisper with small model
  3. For memory: Use faster-whisper with int8 quantization
  4. Batch processing: Process multiple segments together if possible

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

benchflow-ai/skillsbench

csv-processing

Use this skill when reading sensor data from CSV files, writing simulation results to CSV, processing time-series data with pandas, or handling missing values in datasets.

897 232
Explore
benchflow-ai/skillsbench

pid-controller

Use this skill when implementing PID control loops for adaptive cruise control, vehicle speed regulation, throttle/brake management, or any feedback control system requiring proportional-integral-derivative control.

897 232
Explore
benchflow-ai/skillsbench

yaml-config

Use this skill when reading or writing YAML configuration files, loading vehicle parameters, or handling config file parsing with proper error handling.

897 232
Explore
benchflow-ai/skillsbench

simulation-metrics

Use this skill when calculating control system performance metrics such as rise time, overshoot percentage, steady-state error, or settling time for evaluating simulation results.

897 232
Explore
benchflow-ai/skillsbench

vehicle-dynamics

Use this skill when simulating vehicle motion, calculating safe following distances, time-to-collision, speed/position updates, or implementing vehicle state machines for cruise control modes.

897 232
Explore
benchflow-ai/skillsbench

web-interface-guidelines

Vercel's comprehensive UI guidelines for building accessible, performant web interfaces. Use this skill when reviewing or building UI components for compliance with best practices around accessibility, performance, animations, and visual stability.

897 232
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results