How little is too little to tip a delivery guy?

I know, that sounds pretty cheap of me to say. But, my wife and I order take in quite a bit from this Chinese restaurant on the Upper West Side. Sometimes like every other week. This place is six blocks from our apartment. Our last order that we placed we spent about $20 dollars, and gave the delivery person a $3 dollar cash tip. He gave me this look like I had just told him his mother died. Then I closed my door without saying anything to the guy, he then comes back to our front door and rings the bell. I open the door, and he says with a thick and broken accent, "is that all you are going to tip?" I landed up giving him another dollar, but that is not only unprofessional but also extremely rude. But what can we expect, it's New York City, right? So, I don't know, call me cheap, but I don't think the delivery guy needs to be tipped 20% for walking six blocks with a bag of food. There really isn't a whole lot of service involved with that.