1. In a large bowl, whisk together the sweetened condensed milk, whole milk, and instant vanilla pudding mix until smooth with no lumps. Refrigerate this pudding mixture for about 5 minutes to allow it to set slightly.
2. In a separate large bowl, whip the heavy cream and vanilla extract at medium-high speed until stiff peaks form. Reserve half of this whipped cream to use as a topping. Gently fold the remaining whipped cream into the chilled pudding mixture until combined.
3. Using a 3-quart trifle dish or similar clear glass bowl, arrange a layer of vanilla wafer cookies on the bottom. Spread one-third of the pudding mixture evenly over the wafers.
4. Add another layer of vanilla wafers followed by a layer of banana slices. Repeat the layers—pudding, wafers, and bananas—two more times, finishing with a final layer of pudding on top. Reserve some cookies to crumble for garnish.
5. Refrigerate the assembled banana pudding for at least 3 hours, or preferably overnight, to allow the flavors to meld and the layers to set properly.
6. Before serving, stir the granulated sugar into the reserved whipped cream and dollop it over the pudding. Sprinkle crumbled reserved vanilla wafers on top for garnish.