Mathematical Foundations: From the Fourier Transform to the STFT

Continuous Fourier Transform (FT)

The Fourier Transform (FT) of a continuous signal x(t) is defined as:

This representation indicates which frequencies are present in the signal but does not show when they occur. For stationary signals (whose properties do not change over time), the FT is sufficient. However, audio is a non-stationary signal, requiring a different approach.

Short-Time Fourier Transform (STFT)

The STFT introduces a sliding window $w(t)$ that isolates short-time segments of the signal before applying the FT. Mathematically:

Where:

• w(t) is a window function (e.g., Hann, Hamming, Blackman).

• t is the time instant being analyzed.

• $f$ is the frequency.

The STFT produces a time-frequency representation, which can be visualized using a spectrogram.

Key Parameters in the STFT

The quality and resolution of the STFT depend on:

• Window size ($N$):

  • Larger windows → Higher frequency resolution, lower time resolution.

  • Smaller windows → Higher time resolution, lower frequency resolution (Heisenberg-Gabor Uncertainty Principle).

• Overlap:

  • To avoid information loss, an overlap of 50%-75% is typically used.

• Window type:

  • Rectangular: Simple but with high side lobes (spectral leakage).

  • Hann/Hamming: Reduces leakage, better for spectral analysis.

Spectrogram: Visualizing the Time-Frequency Domain

A spectrogram is a graphical representation of the STFT magnitude ($|X(t,f)|$):

• X-axis: Time.

• Y-axis: Frequency.

• Color/Intensity: Energy (dB) or magnitude.

Example features in an audio spectrogram:

• Formants: High-energy bands corresponding to vocal tract resonances.

• Harmonics: Periodic components in musical instrument signals.

Types of Spectrograms

• Amplitude Spectrogram: Displays $|X(t,f)|$.

• Power Spectrogram: Displays ∣X(t,f)∣2∣X(t,f)∣2.

• Log-Scale Spectrogram: Useful for audio, as human hearing perceives frequencies logarithmically (e.g., Mel scale).

Applications in Audio Processing

Audio Compression (e.g., MP3, AAC)

• The STFT identifies irrelevant components (auditory masking) for selective compression.

Speech and Music Recognition

• Extraction of MFCCs (Mel-Frequency Cepstral Coefficients), based on filter banks applied to the spectrogram.

Audio Synthesis and Modification

• Time-Stretching & Pitch-Shifting (e.g., Phase Vocoder).

• Noise Reduction: Time-frequency filtering (e.g., Wiener Filtering).

Musical Instrument Analysis

• Identification of harmonic partials and transients.

Limitations and Alternatives

Limitations of the STFT

• Time-frequency trade-off: Cannot achieve high resolution in both domains simultaneously.

• Smearing effect: Due to window convolution.

Alternative Methods

• Wavelet Transform: Better time resolution for high frequencies.

• Spectral Component Analysis (PCA/ICA): For complex signals.

• Wigner-Ville Distribution: Higher precision but with cross-term interference.

Conclusions

Fourier analysis in the time-frequency domain (STFT + Spectrogram) is a powerful technique for audio processing, enabling:

• Visualization of frequency evolution.

• Feature extraction for machine learning.

• Sound modification and synthesis.

However, its effectiveness depends on proper parameter selection (window type, overlap, FFT size). For advanced applications, methods like wavelets or neural network-based models (e.g., TF-GAN, Wavenet) can be explored.

References

• Oppenheim, A. V., & Schafer, R. W. (2010). Discrete-Time Signal Processing.

• Smith, J. O. (2007). Spectral Audio Signal Processing.

• Mallat, S. (2008). A Wavelet Tour of Signal Processing.