Clips AI

Clips AI is an open-source Python library whose main function is to automatically convert longform videos into clips and dynamically resize video aspect ratios (e.g., from 16:9 to 9:16).

Main Features:

1. Video Transcription: Transcribes videos using WhisperX (an open-source wrapper on Whisper) with additional functionality for detecting start and stop times for each word.
2. Clip Creation: The clipping algorithm analyzes a video's transcript to identify and create clips.
3. Video Resizing: The resizing algorithm dynamically reframes videos to focus on the current speaker, converting the video into various aspect ratios (utilizing Pyannote for speaker diarization).

Target Users: Designed for developers, it is tailored for audio-centric, narrative-based videos such as podcasts, interviews, speeches, and sermons.

Usage Workflow:

1. Installation: Install Python dependencies (clipsai and whisperx), libmagic, and ffmpeg. Using a virtual environment (like venv) is highly suggested to avoid dependency conflicts.
2. Creating Clips: The video must first be transcribed using Transcriber. Then, use ClipFinder to find clips based on the transcription, obtaining StartTime and EndTime for trimming.
3. Resizing a Video: Call the resize function with the video file path, Pyannote auth token, and desired aspect ratio (e.g., 9:16) to get crop segments. A Hugging Face access token is required for Pyannote speaker diarization, but users won't be charged for using it.

Core Advantages: Open-source and free, allowing developers to segment videos and dynamically reframe focusing on the speaker with just a few lines of code.