"... Anyone have any suggestions? We're on mostly uncharted territory here ..."

This is a radical idea probably without merit but small incremental steps to improve the quality of submissions & comments are short term fixes to deeper problems. What are the root cause(s) of poor quality responses?

Identity

Good behaviour in any group is important if you encourage identity. I tried hard in any sites I've joined to stick by the spirit of the group because my identity is tied to anything I say. What would joining HN be like with no identity and zero reputation. A place where there is high competition for submissions and few examples of what is really expected of you? The only sign post I see is karma some FAQ's on behaviour - but who reads those? My behaviour is effected by those around me who in all reality want to improve their standing through karma. Progress is measured by a score that is derivative of what I do, who cares about the outcome. Make identity meaningful. SO does this well. Users are recognised and rewarded. The hard bit is HN isn't binary.

Utility

I join sites like HN because of the quality gap on the web. The only other way I can do this is directly interact with fellow entrepreneurs. HN fulfils this purpose. HN also is about things that interest hackers. That was the intent, discuss new ideas, intelligently. HN is a lot like the LME discussing the effects of X on Y, substituting copper for ideas, effects of conflict on price for execution of product. What happens when the purpose is subverted or unfulfilled?

Audience

Who reads and contributes in HN matters. I don't recognise the readers I started with. As the audience drifts the early adopters leave as the utility of HN drops. A lot of good hackers started here but will probably leave or have left. This is a real problem. Hackers leaving is a signal that things are broken or that the usefulness has been reached. Hackers are really sensitive to certain types of audiences, especially non-technical. Like frogs, Hackers leaving HN might be a sign the audience is polluted with the wrong type of users.

Broken

HN is fundamentally broken. We already know this. It's not a new problem. But something has to fundamentally change to address user identity and utility. Encourage good behaviour by looking at [Identity]: the need to fit in, contribute, improve and [Utility]: the reason users contribute and not get bored or get up to mischief, leave.

Induction

Entry needs to be set higher than it currently is. Where else of value is entry a handle, email and time enough of a measure of worth? I would put a concrete intellectual challenge in the form of some writing, say 500 words in their profile. For extra credit a link to a site the post exists. The purpose is twofold. Create a baseline set of information that can be classified
through code and used to judge the quality of the HN user. Users could game this if they wanted but a quick check against a post on a users website could avert this. This benchmarks each user.

Evaluated

All subsequent posts are measured against their score. Submission scores are scored against their benchmark.

Purpose

Make a real purpose for staying on at HN. Encourage interested HN users to also submit to apply to YCombinator, even if they think they don't fulfill the criteria to make them improve. Tie identity to purpose by making contributing to HN a part of submitting to YCombinator. Give some real purpose. Make being on HN way beyond just submitting links, making stupid comments and watching your score.