In a small saucepan over medium heat, warm milk until it begins to simmer, approximately 2-3 minutes, stirring occasionally.
Reduce heat to low. Stir in pumpkin puree, brown sugar, cinnamon, nutmeg, all spice, and vanilla extract. Whisk gently until the ingredients are combined.
Stir in chopped chocolate, whisking until the chocolate is melted. Remove from heat and stir in bourbon.
Divide the hot chocolate between 4 glasses. Top with 3 large marshmallows and using a torch, toast the tops of the marshmallows.
Serve immediately.