Thank you for the detailed explanation of what happened to the post and your thoughts on the topic in general. I'm happy to hear that what's been going on with some of the bigger surveillance stories is the result of the difficulties of moderation and not external manipulation. I now have a greater understanding of why efforts to stop the topic from dominating the site end up supressing it.

As it seems you noticed, my main frustration is that surveillance stories with new information, and other particularly insightful pieces about the topic, often aren't staying on the front page, which essentially means that no one sees or discusses them at any level of depth. HN is generally so good at what it does that for most topics of interest to the community, it can be one's daily news source for an overview of what's going on. Readers come to expect that topics that interest HN will be covered well and that important developments will be posted. So it's particularly jarring after reading tech-related news only from HN for a few days to see you missed a major surveillance story, especially one that deals with the tech or tools. When you search HN and see you missed it because it was submitted but probably buried before anyone discussed it, you start feeling that moderators are parting ways with the community or that something sinister is going on.

I think a number of other difficulties arise as a result of the major stories not showing up. I imagine users start getting the sense that an important topic isn't being discussed, so they submit and upvote even more "me-too" stories. On the stories that do make it, people don't bother writing long and substantive comments because of the risk of the post dropping to oblivion at any moment and no one ever seeing the comment. Those who know a lot about the topic become less likely to write about it insightfully here, increasing the noise further and giving the topic a worse reputation for quality.

How to improve things is a difficult questions, as this topic's scenario really doesn't play to the site's strengths. It seems there's really no feature for "best-of-a-type" other than penalties.

Never-the-less, here are a few ideas (some of which are assuredly overly vague or simplistic, but since I don't know which might be helpful...):
- Consideration could be given to how many surveillance stories are currently on the front page. Very few people want an entire front page filled with these stories every day, but given the level of interest and the continuing developments one would still expect to see a number of stories every day or two. Maybe only resort to giving a penalty to an article that isn't obviously low quality / blogspam if there are already surveillance stories on the front page?

- Using heuristics concerning novelty, and whether the particular novel issue has been discussed on HN, could supplement length

----- A quick check of whether the article claims it's new information, the date of the article, and a general search for the main claim of the article

----- Less scrutiny of publications with a history of breaking stories or that have access to documents, like theguardian.com, firstlook.org, washingtonpost.com, and spiegel.de

- A way to know a topic isn't going to quickly disappear after investing time in writing a comment would be nice. Some indicator that a topic has been "blessed" by a mod and won't receive a penalty would be helpful for this.

- Having a moderator who follows the topic might be helpful. I definitely understand that not every moderator does, and that it's probably really hard to make judgments if you're not following it. Someone a bit more familiar with the topic would be a good resource and could keep an eye out for sub-optimal penalties being given out.

- In general, visible transparency about what's going on would be helpful. I'm not sure how to achieve that because the site's not really designed for it. Talking to you here is great. Having other people with the same concerns get the information is important too.

- The root issue isn't really that interest in the topic is too high--it's that there's a low signal-to-noise ratio in the stories that are submitted and the passion for the topic is so high that even the noise is upvoted. Some users probably want to make sure everyone knows what's going on. Once a normal level of the best stories are making the front page and there's an outlet for that passion, perhaps the submission problem will decrease and a virtuous cycle will begin.

I'll email first in the future for anything I'd like a response about or if I have more suggestions--I took the comment approach because in the case of a forum or many other online communities, it would be better to have the conversation publicly, but I agree that HN's not really set up well for that (as evidenced by my multiple requests for a response).

Again, I'm grateful for your response, and I greatly appreciate the work you're doing here and your commitment to addressing users' concerns.