I'll leave a discussion of the ethics of withdrawal to someone else, but the main consequence of withdrawing your paper will be just that: Your paper will be withdrawn. It's unlikely to be something that's held against you.
It may be a slight inconvenience to the organisers, particularly if you were accepted for an oral presentation, to have to find someone to replace you (or to leave a gap in the programme) but part of taking on the role of organising a conference is accepting that these things happen.
You say that you've not submitted the camera-ready paper, so the proceedings will not have been completed. Therefore I'd suggest that there'd be no real damage done if you withdrew.
To summarise. It might be a nuisance for the organisers (one they may well have accepted as par for the course), but there's no real harm been done.
Edit: This isn't in answer to your question, but if the issue with the submission is simply financial then it may be worth contacting the organisers to ask if they know of any financial support that might be available, as well as conducting your own search for funds, before withdrawing.