Preheat the oven to 400°F (200°C). Line a large baking sheet with parchment paper.
Stir the raspberry jam and cornstarch together in a small bowl until evenly combined. Set the filling aside.
Place the cold puff pastry sheets on a lightly floured work surface. Cut each sheet into eight equal rectangles, creating 16 pieces in total. Refrigerate the pastry briefly if it becomes soft or sticky.
Arrange eight pastry rectangles on the prepared baking sheet. Place approximately 1 tablespoon of raspberry filling in the center of each rectangle, leaving a clean 1/2-inch (1.25 cm) border around the edges.
Beat the egg with 1 tablespoon of water. Lightly brush the exposed pastry borders with the egg wash.
Place the remaining pastry rectangles over the filling. Press the edges together with your fingers, then crimp them firmly with a fork. Cut two or three small steam vents into the top of each tart.
Brush the tops with the remaining egg wash and sprinkle with coarse sugar, if using. Refrigerate the assembled tarts for 10 minutes to help them maintain their shape.
Bake for 17 to 20 minutes, until the pastry is puffed and deeply golden. Rest the tarts on the baking sheet for 5 minutes before transferring them to a wire rack to cool completely.
Combine the powdered sugar, 1 tablespoon of milk, and vanilla extract in a small bowl. Stir until smooth, adding a few drops of additional milk at a time until the glaze is thick but spreadable.
Spread or drizzle the vanilla glaze over the completely cooled tarts. Immediately garnish with crushed freeze-dried raspberries or sprinkles, if desired. Allow the glaze to set before serving.