Aflatoxin B1 (AFB1) in food is a threaten to food safety and human health. In this study, a colorimetric and fluorescent dual-mode aptamer sensor was constructed for rapid and accurate detection of AFB1 based on polyvinylpyrrolidone-gold-copper bimetallic nanoclusters (PVP-Au/CuNCs) and Mn3O4 nanoparticles (Mn3O4 NPs). In the presence of AFB1, the AFB1 aptamer on the surface of Mn3O4 NPs specifically bound to AFB1. The oxidase-like activity of Mn3O4 NPs was restored and 3,3',5,5'-0tetramethylbenzidine (TMB) was oxidized to oxidized TMB (oxTMB), which quenched the orange fluorescence of the PVP-Au/CuNCs. With AFB1 concentrations increasing, the blue color of oxTMB deepened, the fluorescence quenching efficiency was enhanced. The working range was 0–56 μg/L with a limit of detection (LOD) of 0.36 μg/L (colorimetric), and 2.86–40.00 μg/L with a LOD of 0.29 μg/L (fluorescent). In addition, the established sensor was successfully applied to determine AFB1 in milk, rice, oats, and corn, with a recovery rate of 91.1%–107.5%. This study provides an insight for fabricating a sensitive, efficient, and accurate analytical platform for rapid detection of AFB1 in food.