Turn on the thumbnails
Once the video has been added to the submenu, open the settings of the button by selecting it from the tree view on the left side panel. You can also hover over the button and select the settings icon.
In the lower-right of the button settings, select your preferred option for the thumbnail (Upload Image, Use Current Frame, Reset to Default). The default image is the poster frame of the video. If you choose "Use Current Frame", it will capture a frame from the current time in the player and use it as the thumbnail.
You can learn more about the buttons settings here: Collection button settings