AudioFP

Project Overview

AudioFP is a self-hosted audio fingerprinting service. You index your own audio and video files into a library. After that you can hand it a few seconds of audio and it tells you which recording the clip came from and where in that recording it sits. Or you can turn it around: index a short pattern, say a jingle, a compliance disclaimer or a piece of hold music, search with a long recording, and it lists every place that pattern turns up, with start and end times.

It works on the sound itself. There is no speech recognition and no cloud service in the loop, so it is just as happy with music as with IVR prompts, adverts or anything else that was recorded once and played many times. Everything runs on your own machine. The first version was a weekend Shazam clone; version 2 is the same idea rebuilt so that it can be run for real, with a REST API, a command line, a web UI and the operational plumbing you need when other people depend on it.

Why I Built This

The first version came out of wanting to understand how audio fingerprinting actually works. The Wang (2003) paper behind Shazam is short and the idea is clever, but most explanations stop at the theory. I wanted to read the paper, implement it from scratch and end up with something I could point a clip at.

Version 2 came from a different place. I kept running into the same question in contact centre work: was the mandatory disclaimer played on this call, how many times did the hold music loop, which calls contain this advert. Those are acoustic questions, the same audio recurring in many recordings, and a fingerprint index answers them well. But the first version could not be trusted for that. It found the best match whether or not there was one, it loaded whole files into memory, and it had no notion of a pattern occurring several times. So I rewrote it.

What changed in 2.0

• A second search mode, occurrences, that returns every alignment between the query and a track, with spans in both, instead of one best match.
• Scores that mean something. Every match carries confidence, aligned_hashes and peak_ratio, and matches under the thresholds are dropped. A clip that is not in the library now comes back with nothing.
• A streaming pipeline. Decoding, the STFT and the peak picking all run in chunks and give the same result as processing the whole file, so an hour-long recording needs a few tens of MB rather than gigabytes.
• Storage that holds up. A new SQLite schema with clustered fingerprint rows, batched sorted writes and crash repair, a PostgreSQL backend on psycopg 3, and one contract test suite that every backend has to pass.
• Background jobs with progress, cancellation and a history that survives restarts. Duplicate detection. A folder allow-list, API key auth, uniform JSON errors with request ids, JSON logs.
• A proper CLI (audiofp), an OpenAPI document, a rebuilt web UI with match timelines, and a documentation set that covers architecture, tuning, deployment, performance and troubleshooting.
• librosa is gone. The STFT is a few lines of numpy and produces bit-identical output, and the package installs without a compiler.

Key Features

• Reads WAV, FLAC, OGG/Opus, MP3 and AIFF directly through libsndfile. With ffmpeg installed it also reads M4A, AAC, WMA and the usual video containers.
• SQLite by default, PostgreSQL for multi-process deployments, an in-memory store for tests.
• REST API with pagination, search, sort and filter for tracks, metadata editing, bulk delete, audio streaming with Range support, and an interactive /docs page.
• Web UI with drag and drop upload, microphone recording, match timelines, play from the matched position, a library with a metadata drawer, an activity page for jobs, and dark, light and system themes.
• Docker image and compose file, CI on Linux, Windows and macOS, and a test suite that generates all its audio so the repository stays small.

Screenshots

Search results with match timelines

Library (light theme)

Library (dark theme)

The shape of the pipeline

How It Works

Indexing

1. The file is decoded in chunks to 11,025 Hz mono. libsndfile handles the native formats and resamples with soxr; anything else goes through an ffmpeg pipe.
2. Each chunk is turned into spectrogram columns with a Hann-windowed STFT (2048-point window, hop of 512 samples). The extractor keeps just enough context between chunks that the result is identical to a whole-file STFT.
3. Local maxima in that time-frequency map become peaks. A cell counts when it is the largest in a 20 by 20 neighbourhood and above an amplitude threshold that is scaled to the file's loudest sample.
4. Each peak is paired with the next ten peaks in time. Two frequency bins and the frame gap between them are packed into one 36-bit hash.
5. The hashes go into an inverted index together with the frame they occur at. SQLite buffers them and writes large sorted batches, which keeps indexing speed flat as the database grows.

Searching

1. The query goes through the same pipeline.
2. All of its distinct hashes are looked up in one batch. Hashes that occur thousands of times in the library are treated as stop words and skipped.
3. Every shared hash votes for a time offset between the query and the track. Real matches pile their votes on one offset; coincidences spread out.
4. Per track, the vote histogram is smoothed by one frame, the spike is measured, and the track is scored. In identify mode the best spike per track is kept; in occurrences mode every spike that clears the thresholds is kept.
5. Offsets are signed. A positive offset means the clip starts inside the track. A negative one means the track was found inside the query, which is the pattern search case.

The three scores

aligned_hashes is the number of distinct query hashes that vote for the winning offset. Distinct matters: a sustained tone repeats one hash for many frames and would otherwise fake an alignment. peak_ratio is that count divided by the average votes per bin away from any spike, so it says how sharp the spike is compared with the noise floor. On unrelated audio it lands somewhere between 3 and 9 whatever the library size, and real matches sit well above that. confidence is the aligned count divided by the number of distinct query hashes inside the matched region, so a short pattern found inside a long recording still scores high.

Tech Stack

My Contribution

I designed and built both versions end to end.

• Implemented the landmark fingerprinting algorithm from the Wang (2003) paper: peak picking, combinatorial hashing and time-offset voting
• Rebuilt the pipeline to stream: chunked decoding, a chunked STFT with context carry-over that is bit-identical to librosa's, and single-pass normalisation
• Designed the scoring (distinct aligned hashes, spike-excluded background, region-local confidence) and calibrated the default thresholds on synthetic corpora at several noise levels
• Added the occurrences mode with span-based suppression, and made it run in under a second on a 30-minute query
• Designed the storage contract and the SQLite schema (clustered rows, batched sorted writes, crash repair), and rewrote the PostgreSQL backend on psycopg 3 with COPY loads
• Built the job manager, the indexer with de-duplication, the typed configuration, the error and logging conventions, the API key and stream token auth
• Wrote the CLI, the REST API and OpenAPI document, the web UI, the test suite, the CI matrix, the Docker image and the documentation

Challenges and Learnings

Chance matches on repetitive audio
The original scoring counted raw votes, and repetitive material broke it in three different ways. A held note repeats one hash for hundreds of frames, so it piles up votes at an offset that means nothing. A pattern that really does occur five times in a recording makes five spikes, and a background computed as a plain mean over all bins is then dominated by the matches themselves. And a single shared chord can align perfectly for one frame. The fixes were to count distinct hashes rather than votes, exclude every strong bin from the background, require an alignment to span at least three distinct frames, and measure confidence over the matched region only. Each of those came out of a specific failing test.

An occurrences search that took 157 seconds
The first implementation of occurrences mode looped over every candidate offset bin and touched every vote for each, so a 30-minute query against a busy track was quadratic. Two things fixed it: a cheap upper bound that rules out most bins before any real work, and keeping the votes sorted by offset so each candidate reads a slice instead of scanning. The same query now takes about 0.7 seconds. A test pins that with a five-minute loop of the same tone.

Memory on hour-long files
Loading a whole file, normalising it and computing one big STFT cost 1.35 GB for an hour of audio. The streaming version needs the STFT of chunk N+1 to see exactly the same neighbourhood as a whole-file STFT would, so the extractor keeps a margin of columns on each side and only emits the ones that are final. Getting that bit-identical, including the zero padding at both ends, took a while, but the test that compares it against a whole-file reference at every chunk size is the one I trust most in the suite.

SQLite write amplification
Inserting one track at a time touches a random leaf page for every hash, and on a large library each track rewrites hundreds of megabytes of B-tree. Buffering rows and writing them in large sorted batches fixed the throughput, but it created a new failure: a crash between batches would leave tracks in the library with no fingerprints. So each track carries a flushed flag, and on start-up the store removes anything that never made it to disk and says so in the log.

Shutting down cleanly
With a write buffer in play, how the process dies matters. Python's default SIGTERM handling kills the process without running any cleanup, which is exactly what systemctl stop and docker stop send. The server now turns SIGTERM into a normal shutdown: running jobs stop at the next file, the buffer is flushed and storage is closed. Small change, but easy to miss.

Core Logic: How the Code Works

A few of the pieces that matter most, trimmed to their essentials. The full source is on GitHub.

Hash layout

A landmark is an anchor peak, a target peak and the frame gap between them. Twelve bits each is enough for a 2048-point FFT and gaps up to 4095 frames, and the whole thing fits in one int64.

FREQ_BITS = DELTA_BITS = 12
FREQ_MASK = DELTA_MASK = (1 << 12) - 1

def encode_hash(f1, f2, dt):
    """Pack (anchor bin, target bin, delta frames) arrays into int64 hashes."""
    f1 = np.asarray(f1, dtype=np.int64) & FREQ_MASK
    f2 = np.asarray(f2, dtype=np.int64) & FREQ_MASK
    dt = np.asarray(dt, dtype=np.int64) & DELTA_MASK
    return (f1 << (FREQ_BITS + DELTA_BITS)) | (f2 << DELTA_BITS) | dt

Pairing is vectorised across all anchors at once: for k in range(fan_value), the target is the peak k + 1 positions later in time order. Pairs further apart than max_hash_time_delta are dropped.

Chunked STFT

The extractor keeps a sample buffer. Each push frames as many complete windows as it can, keeps the tail the next window still needs, and returns the new spectrogram columns. Half a window of zeros goes in before the first chunk and after the last, which puts frame i at sample i * hop, the same grid as a centred whole-signal STFT.

def _consume_buffer(self, final: bool) -> np.ndarray:
    n_frames = 0 if self._buf.size < self.n_fft else (self._buf.size - self.n_fft) // self.hop + 1
    if n_frames == 0:
        return np.zeros((self.n_fft // 2 + 1, 0), dtype=np.float32)
    usable = (n_frames - 1) * self.hop + self.n_fft
    cols = self._spectrogram_columns(self._buf[:usable])   # sliding_window_view + rfft
    self._buf = self._buf[n_frames * self.hop :]
    return cols

Peak picking needs neighbours on both sides of a column, so the columns are not emitted straight away. A margin of neighborhood columns waits for the next chunk, and the last emitted columns are kept as context, which is what makes the chunked result identical to the whole-file one.

Offset voting and the spike

Per track, the votes are a histogram of track_frame - query_frame. Clips rarely start on a frame boundary, so each bin also absorbs its neighbours within the tolerance. The best bin is then measured by how many distinct query hashes voted for it.

def _smoothed_histogram(offsets, tolerance):
    bins, raw = np.unique(offsets, return_counts=True)
    smoothed = raw.astype(np.int64).copy()
    for delta in range(1, tolerance + 1):
        for sign in (-1, 1):
            shifted = bins + sign * delta
            idx = np.minimum(np.searchsorted(bins, shifted), bins.size - 1)
            hit = bins[idx] == shifted
            smoothed[hit] += raw[idx[hit]]
    return bins, raw, smoothed


def _count_distinct(values):
    """Distinct values by sort + diff; several times faster than np.unique here."""
    ordered = np.sort(values)
    return int(np.count_nonzero(np.diff(ordered))) + 1

peak_ratio is the distinct count divided by the mean votes per bin among bins that sit away from every strong spike. Keeping all spikes out of the background, not just the best one, is what lets a pattern that occurs five times still look sharp.

Turning numbers into a label

def quality_label(confidence: float, peak_ratio: float) -> str:
    if confidence >= 0.15 and peak_ratio >= 30:
        return "strong"
    if confidence >= 0.05 and peak_ratio >= 18:
        return "likely"
    return "weak"

The defaults (10 aligned hashes, confidence 0.02, peak ratio 12) came from running two families of synthetic audio through the matcher at several signal-to-noise levels. True matches stayed above a peak ratio of 15 down to 0 dB for clips of three seconds or more, and chance matches never went past about 7.

Installation and Setup

git clone https://github.com/inboxpraveen/Audio-Fingerprint.git
cd Audio-Fingerprint
python -m venv .venv && . .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

audiofp doctor        # checks Python, libsndfile, ffmpeg and the storage
audiofp serve         # UI and API at http://localhost:5000

audiofp index ./my-recordings --tags campaign-a
audiofp search clip.wav
audiofp search call.wav --mode occurrences

ffmpeg is optional. Without it you can't decode video or M4A/AAC/WMA; everything else works. There is also a Docker image and a compose file, and the deployment guide covers a reverse proxy, systemd and PostgreSQL.

docker build -t audiofp -f docker/Dockerfile .
docker run -p 5000:5000 -v audiofp-data:/data -e AUDIOFP_API_KEY=s3cret audiofp

Where it goes next

Right now AudioFP searches sound. The plan is to add keyword and phrase search on audio, with speech recognition running next to the fingerprint index, and on top of that the QA workflows a contact centre needs: saved pattern sets, batch evaluation of a folder of calls, rules like "this phrase must occur in the first 30 seconds", and reports per call. The storage contract, the job manager and the UI were built so that a transcript engine can slot in beside the fingerprint one without a second set of plumbing.

Closing Note

AudioFP is open source under the MIT licence. It started as a way to learn one algorithm properly and turned into something I would run in production. If you are curious about audio fingerprinting, the architecture and tuning docs in the repository go through the whole pipeline with the real constants, and the test suite is a decent place to see how each scoring rule was arrived at.

Questions, bug reports and pull requests are welcome on GitHub.