Type 2 diabetes mellitus (T2DM) is a multifaceted metabolic condition characterized by chronic hyperglycemia (elevated blood sugar levels) and insulin resistance (IR). We have isolated a new homogeneous polysaccharide, LYTP-3 (7.22 × 105 Da), with anti-diabetic activity, from large leaf yellow tea (LYT) by biological activity-guided sequential separation and activity evaluation. Using a T2DM mouse model, we demonstrate that oral administration of LYTP-3 could reduce blood lipid levels, alleviate hyperglycemia and glucose tolerance, and inhibit IR. LYTP-3 ameliorated T2DM treatment by activating the liver X receptor alpha (LXRα)-ATP-binding cassette (ABC) transporter (LXRα-ABCA1/ABCG1) signaling pathway, which is regulated by the gut microbiota. The strain Lachnospiraceae bacterium A4 is potentially the key to this improvement. Additionally, we have solved the molecular structure of LYTP-3 using nuclear magnetic resonance (NMR), and methylation analyses. The main backbone of LYTP-3 consists of 1,2,4-α-L-Rhap, 1,4-α-D-Galp, 1,6-α-D-Glcp, with branches of 1,4-β-D-Galp, 1,6-β-D-GalpA, 1,3,6-β-D-Manp, 1,2-α-L-Rhap, T-β-Manp, and T-β-Galp. The findings of this study suggest that LYTP-3 may serve as a potential gut microbiota modulator in the treatment of T2DM.