So, while working on the RetroPi-Setup, I finally found the reason for the slowness (50kb/s) when transferring games.
It's actually a common Raspberry Pi issue: When connecting a USB 1.1 device to the 2.0 hub of the Pi, there are packet drops... which result in the slow speed.
Forcing the Raspberry Pi to run in 1.1 mode fixes the slowness issue!
Of course, 2.0 devices will also run slower then (so USB Sticks, harddisks, etc.), but the Retrode runs full speed then.
To do that, add the following kernel parameter to the boot-script:
dwc_otg.speed=1
(Add that into /boot/cmdline.txt)
That's it
