Put the sugar, cocoa, pumpkin pie spice, salt, and water in a small saucepan and bring to a boil. Let cook for a minute or so until mixture is thickened and whisk in the milk.
Let simmer until hot, but don't boil. Add vanilla and pour into mugs. Top with whipped cream and marshmallows.