I pay my Citibank credit card, AT&T, Verizon, Verizon Wireless and Comcast this way and have been for a year or more. I've had no problems at all. Most of them will send an email confirming the payment authorization. All the sites have a payment history view and I can look at my credit union checking account online. The only recurring bills I pay by check are my mortgage (they don't have any other way to pay) and the water bill (they charge a $3 convenience fee for online payment).
My credit union does offer online bill paying but I'm too cheap to pay for it.