I get asked this question so often that I thought I'd blog about it. So...
In a word: yes
If the casting director doesn't need your headshot once you are there, then just put it back in your portfolio, and all is well. However, it often happens that a casting director will ask for one to remember to call you in for another job. We do this all the time.
At this point, you may be making snarky remarks about my short-term memory, or my ability to take notes. But a casting day is made up of many moving parts, and they are all kept in our heads, so a visual reference of you is so much better than just a scribbled note that reads "Bring mrphpmrhpmr in for the role of mrphmprh "Jamie'.
Sometimes, when more than one casting director shares offices, someone you are NOT auditioning for may see you working in the lobby and ask you for a picture for an upcoming project.
The real point here is that it shouldn't even be an issue. Your pictures and resume are your business cards! You will never catch a lawyer or a realtor without their business card, and you are coming to what amounts to the market place without yours. You don't need to carry hundreds of them, just as many as you have auditions on that day. They are very light, even with the extra weight of the resume on the back and the staple you've attached it with. So carry a bunch with you. Bring them into the audition, and see if the casting director wants one. It can't hurt.