If he finds it rude, it certainly doesn't show. Really, he's just glad to have skipped the can you talk approach. He debates the answer for only a little time less than Ellie took to debate asking it.

My voice would destroy this building.

He doesn't look particularly grim as he writes it; maybe it's a joke. He knows it's unbelievable to someone who isn't familiar with the sort of power that can be found in the greater universe, after all, and doesn't mind if she doesn't believe him.