Budding speciation is a process wherein a new species arises from a small, isolated population within or at the margin of an ancestral species. Well-documented cases of budding speciation are rare, and the roles of various evolutionary factors in this process remain controversial. Based on whole-genome resequencing data from 272 individuals across 27 populations, we reconstructed the evolutionary history of Rhodiola sect. Trifida and explored the relative contributions of natural selection, genetic drift, and chromosomal rearrangements as drivers of lineage divergence. We found that all samples of R. chrysanthemifolia (including R. alterna and R. sinuata) were clustered into three clades. Rhodiola liciae was sister to all other samples in the section, likely due to post-divergence gene flow and the minimal population structure of the progenitor species, while it shared the same ancestry with R. ch-I in population structure analyses. The two populations of R. sinuata were not monophyletic, instead clustering with geographically proximate populations of R. ch-III. Demographic analyses revealed that R. liciae underwent a contraction in population size following its divergence from R. ch-I approximately 0.34 million years ago (Mya), and has remained stable since around 0.1 Mya. Genomic islands and genotype-environment association analyses suggested that genetic drift and the assorting of ancestral polymorphism may have played a more significant role in the speciation of R. liciae than nature selection or chromosomal rearrangements. We propose that R. liciae diverged from R. chrysanthemifolia through budding speciation, although post-divergence gene flow has obscured its phylogenetic signal. Additionally, we identified two potential parallel budding speciation events in R. sinuata at an earlier stage than R. liciae. Our study highlights budding speciation as a prevalent yet poorly characterized mode of plant speciation, with assorting of ancestral polymorphism as a key stochastic mechanism in the process.