To your first question: Obviously you don't have to complete this to get a job. You didn't, and so you probably will not get the job.

To your second question: I find this particular challenge to be unreasonable. I have provided coding challenges many times for people who are interviewing for companies I work for or own. Sometimes they are "very large project" type challenges. But I always stipulate in writing and over the phone that "we would like you to time box your efforts to 30 - 60 minutes total. Focus on some aspect of the project you would like us to see. We don't mind if you only write a single function. We want to understand your methods and thought processes, that is all." I find this works exceptionally well. Unfortunately, a number of technical companies really get this interviewing thing wrong.

Now for some friendly advice, which, of course, you can take or leave. You must be looking for a job or at least you are attempting to have a relationship with this recruiter, because it seems you have spoken to companies that this recruiter has found for you. It is good to not burn bridges with recruiters. You just burned a bridge.

You need to realize that the recruiter is never going to pass your message along to the client. The recruiter is right now crafting a message to the client saying "Chad really doesn't have time to complete the challenge because he has a few other offers on the table. Would you like to give him an offer? Of course if you don't, I have 30 other engineers for you to look at who would love to complete the challenge and will be a great fit for your company!" Yes it is a blatant lie, and your recruiter may make up an entirely difference excuse for why you won't do it. But the client will certainly never receive your message. Never.

In the future when a job comes up which fits your skill set and price range, do you think this recruiter is going to think of you first? Not a chance. "Chad looks like a perfect fit for this. But, what if they give him a code challenge? That's a lot of hassle to have to save face with the client. Hmmm Susan and Johnny are both pretty good fits as well, I'll send them."

Your point is good but your presentation is the problem. Maybe you could have tried:

"I am super busy right now, I just can't complete such a large challenge with my family life and work schedule. Please let your client know I am so sorry and I don't want to cause them trouble, I just can't complete it at the moment."

This would have been a great way to get out of it that would have left the recruiting thinking of you in the future. Alternatively you could have tried something like:

"Wow! I really like this client and I would love to work with them. It sounds like the Director really wants to find the right candidate. However, I am a bit concerned. This is not just a lofty challenge but a time consuming one, especially with family, kids, and work. I estimate this would take me 8 hours to complete. Maybe I am way off though. I would love to show them my abilities, but would they object to me time boxing this to 1-2 hours? How can we work together to make this happen?"

Your milage my vary of course, but don't burn too many recruitment bridges, or you might find you have no way to cross over to the other side.