They need to put a visa in your passport, so I would expect them to keep it until your application is approved.

If you need it, you'll have to contact them to return it, but this will delay your application further. On the other hand, it could push them in to an early approval!!