Read the manual about creating a Voucher type Item and a Voucher Tender Type. In terms of using a voucher for store credit, you create your return transaction, and add the Voucher item to it for the balance of the return. This makes the total transaction balance $0, you tender it and just hit enter - you don't give any money back... the receipt will show the voucher number you issued and it's value.
To redeem this, they can just bring back the receipt (or you can give them some other form or Gift Certificate with the voucher ID). You ring up their purchases and enter use the Voucher Tender Type to redeem the item. If they don't redeem the full amout of the voucher, the receipt will again show the remaining balance. If they try to purchase more that the voucher amount, the system will alert you and re change the voucher tender amount to the voucher balance...