Objective To explore the application of multimodal model in diabetic retinopathy (DR) referral by integrating computer vision (CV) and structured data. Methods Diabetic patients from Lixiang Eye Hospital of Soochow University and Suzhou Municipal Hospital were enrolled. Fundus photographs and clinical data were collected, and patients were divided into the DR group without referral and the DR group requiring referral according to the relevant criteria of clinical guidelines. EfficientNetV2S was used as the posterior bone framework for transfer learning, and the CV model α was constructed. And then integrated with the output of the CV model α and the patient's clinical structured data, the multimodal model β was established by using H2O's AutoML platform to determine whether diabetic patients need DR referral or not. Results In the validation dataset, the accuracy of the model α was 0.918, the sensitivity was 1.000, the specificity was 0.891 and the area under ROC curve (AUC) was 0.946. In the external test dataset, the accuracy was 0.879, the sensitivity was 1.000, the specificity was 0.831 and the AUC was 0.918. In the internal validation set, the accuracy of the multimodal model β was 0.965, the specificity was 0.953, the sensitivity was 1.000, and the AUC was 0.977. In the external test set, the accuracy was 0.985, the specificity was 0.986, the sensitivity was 0.983, and the AUC was 0.984. Furthermore, the XGBoost model results were visualized by SHapley Additive exPlanation, which can observe the distribution of the top variables in patients with binary outcomes in the optimal model based on the XGBoost algorithm. Fasting blood glucose, insulin, AST, systolic pressure and triacylglycerol were positively correlated with DR referral, while high-density lipoprotein cholesterol was negatively correlated with DR referral. Conclusion Compared with the single modal model, the multimodal integration model based on CV and structured clinical data can significantly improve the accuracy of DR patients referral.