"Why would I be angry? Have your affairs," he replied. Despite the question it was clear he was, in fact, angry.
Other options:
no subject
"Why would I be angry? Have your affairs," he replied. Despite the question it was clear he was, in fact, angry.