Deep learning-based genomic prediction (DL-based GP) has shown promising performance compared to traditional GP methods in plant breeding, particularly in handling large, complex multi-omics data sets. However, the effective development and widespread adoption of DL-based GP still face substantial challenges, including the need for large, high-quality data sets, inconsistencies in performance benchmarking, and the integration of environmental factors. Here, we summarize the key obstacles impeding the development of DL-based GP models and propose future developing directions, such as modular approaches, data augmentation, and advanced attention mechanisms.