Scoop the batter evenly into the prepared muffin cups filling each one about 3/4 of the way full with batter. Sprinkle the reserved streusel topping on top of each muffin.
Stir together the powdered sugar, milk, and vanilla extract in a small bowl using a whisk or a fork. Immediately drizzle the vanilla glaze over top of each slightly warm muffin.