Singing Voice Synthesis through Diffusion Probabilistic Model
Open Access DepositedDownloadable Content
a forward diffusion process that gradually transforms data into noise, and a learned reverse process that reconstructs data by iterative denoising. Unlike generative adversarial networks (GANs), which rely on adversarial training and can suffer from mode collapse, training instability, and sensitivity to hyperparameters, diffusion models optimize likelihood-based objectives and typically achieve stable convergence and superior sample diversity. The flexibility of the diffusion framework has enabled advances in large-scale applications, most notably through models such as Stable Diffusion, which demonstrate that diffusion approaches can generate visually coherent and semantically rich outputs. Audio processing has become a fundamental component in modern machine learning, enabling the development of applications such as text-to-speech (TTS) and singing voice synthesis (SVS). TTS systems aim to convert written text into natural-sounding speech, supporting applications from virtual assistants to accessibility tools. In contrast, SVS focuses on generating expressive and musically coherent singing voices from symbolic musical scores, requiring precise modeling of pitch, rhythm, and timbre. A prevailing approach in both TTS and SVS is the two-stage architecture, where the system first predicts intermediate acoustic representations such as mel-spectrograms from symbolic inputs, and subsequently reconstructs the audio waveform using a neural vocoder. Although recent advances in deep generative modeling have significantly improved the fidelity and expressiveness of both TTS and SVS, current SVS systems continue to face challenges related to the presence of artifacts, such as phase inconsistencies and background noise. These artifacts often arise from the complexity of modeling long-term musical structure, the limitations of two-stage architectures that separate acoustic feature generation and waveform synthesis, and the difficulties of aligning symbolic and acoustic information. Addressing these problems remains essential for producing high-quality, artifact-free singing voices and advancing the broader field of audio generation. Music scores and music notes provide structured representations of melody, rhythm, and expressive intent, serving as the primary symbolic input for singing voice synthesis and enabling precise modeling of vocal performances. However, conventional SVS systems conditioned solely on musical scores often suffer from unnaturalness and artifacts, largely due to limitations in capturing the richness of real-world acoustic features. To address these challenges, we introduce a reference-based diffusion model architecture that leverages reference audio to incorporate more detailed acoustic information than musical score conditioning alone. This approach was first implemented at the acoustic modeling stage with RDSinger, enhancing expressive control through reference guidance. Building on this foundation, we extend the framework by integrating multi-resolution processing directly at the audio level, resulting in the design of SmoothSinger. Unlike traditional two-stage pipelines, SmoothSinger operates directly on audio signals and applies postprocessing within a unified model, effectively eliminating artifacts associated with intermediate representations and producing natural, artifact-free singing voices. In this work, we systematically evaluate these methods using the Opencpop dataset, a large-scale Chinese singing corpus with high-quality audio and detailed musical annotations. Our results demonstrate the potential of reference-guided and diffusion-based approaches for advancing the quality and realism of computational singing voice synthesis.
Diffusion models are a class of probabilistic generative frameworks that have attracted considerable attention for their ability to synthesize high-quality and diverse data across a range of domains, including vision, audio, and natural language. These models operate by simulating a two-step stochastic process
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.