This study investigated the association of hypertension with cognitive impairment and determined whether better adherence to plant-based diet (PBD) patterns may modify the association. We analyzed 4 485 elderly individuals with normal cognition at baseline from the Chinese Longitudinal Healthy Longevity Survey (2011−2018). Cognitive function was assessed by the Mini-mental State Examination (MMSE), and cognitive impairment was defined as an MMSE score below 18 points. Diet patterns were assessed using the overall plant-based diet index (PDI), healthful plant-based diet index (hPDI), and unhealthful plant-based diet index (uPDI) derived from a simplified food frequency questionnaire (FFQ). Blood pressure was measured through physical examination. Cox proportional hazard regression and restricted cubic spline were performed. Participants with hypertension had an increased risk of cognitive impairment compared with normal group. The associations between hypertension and cognitive impairment were more pronounced among participants with lower PDI than those with higher PDI, among participants with lower hPDI than those with higher hPDI, and among participants with higher uPDI than those with lower uPDI. A healthful PBD pattern may attenuate hypertension-induced cognitive impairment, while an unhealthful PBD pattern may exacerbate hypertension-induced cognitive impairment in elderly. Adherence to healthful PBD pattern could be used to prevent or mitigate adverse neurological effects in the hypertensive population.