When exporting images, you need to be aware that most image formats have limitations. A limitation of jpeg images is, that it does not support transparency. When exporting to jpeg, GIMP fills the transparent areas with the background color, which by default is white.

To fix this, you will have to choose a different image format that does support transparency, such as png or tiff.