skysight-inc/HackerNews huggingface dataset:
I am very familiar with SQLite internals. The answer is already in there. SQLite stores each row as each column value encoded sequentially corresponding to the declared order of the columns. Changing column order or deletions/inserts require a rewrite of every row. The one special case that is allowed is adding a column on the end of the schema providing it has a default value.

A SQLite provided ALTER TABLE implementation would do exactly what was stated - start a transaction, rename the existing table to a temporary name, create a new one with the desired schema, and copy data across mangling as appropriate before deleting the old table and finishing the transaction. For plain tables this is no big deal, but for more complicated ones there are a lot of issues such as foreign key references, constraints, indices. The majority of the code would be dealing with all these conditions and interactions.

It also wouldn't be any more efficient than code the developer writes - ie there are no shortcuts available to the SQLite developers that aren't available to developers using SQLite. The net effect would be a larger library (they limit to a 250kb library size so something else would need to be chopped), some complex documentation and a heck of a lot of testing code. For something that is relatively rarely needed (see requests on the mailing lists).

The chance of something like this ending up in the SQLite core is extremely slim, so you could never depend on it being there anyway.

SQLite does have several things to help. There is a user_version pragma you can use to keep track of the schema version and use for upgrading. You can temporarily disable foreign key and constraint enforcing. There are numerous pragmas to get table metadata. The table definitions are stored as SQL strings in a sqlite master table, and a pragma allows you to make that writeable.
In case you were confused, I've never compared Xinhua to CNN. In fact, I never even MENTIONED Xinhua. What I said was that CNN, NYT etc all conform to the official Washington party line.

But now that you brought it up, I can also say this: CNN is as much an instrument of the US state, as Xinhua is.

Of course CNN is privately run, and also pays lip service to journalistic independence and stuff, but that's just how this thing works over there. That's like Google saying "Don't be evil" or McDonalds touting the quality of it's products: i.e. only for very naive people.

You might like this:

"As Major Thomas Collins, of the U.S. Army Information Service acknowledged: "Psyops personnel, soldiers and officers, have been working in CNN's headquarters in Atlanta through our programme 'Training With Industry'. They worked as regular employees of CNN. Conceivably, they would have worked on stories during the Kosovo war. They helped in the production of news.""

Or this:

"In an extraordinary directive to its staff, Cable News Network has instructed reporters and anchormen to tailor their coverage of the US war against Afghanistan to downplay the toll of death and destruction caused by American bombing, for fear that such coverage will undermine popular support for the US military effort".

Or this:

In a second memo leaked to the Post, CNN’s head of standards and practices, Rick Davis, expressed concern about reports on the bombing of Afghanistan filed by on-the-spot reporters. Davis noted that it “may be hard for the correspondent in these dangerous areas to make the points clearly” about the reasons for the US bombing. In other words, the CNN official feared that overseas correspondents might be intimidated by local opposition to the US military intervention and allow such sentiments to influence their reports.

To ensure that every CNN report always includes a justification of the war, Davis prescribed specific language for anchors to read after each account of civilian casualties and other bomb damage. He suggested three alternative formulations:

* “We must keep in mind, after seeing reports like this from Taliban-controlled areas, that these US military actions are in response to a terrorist attack that killed close to 5,000 innocent people in the US.”
* “We must keep in mind, after seeing reports like this, that the Taliban regime in Afghanistan continues to harbor terrorists who have praised the September 11 attacks that killed close to 5,000 innocent people in the US.”
* “The Pentagon has repeatedly stressed that it is trying to minimize civilian casualties in Afghanistan, even as the Taliban regime continues to harbor terrorists who are connected to the September 11 attacks that claimed thousands of innocent lives in the US.”
Davis concluded with an ultimatum to journalists concerned that they may sound like parrots for the White House: “Even though it may start sounding rote, it is important that we make this point each time.”

http://www.wsws.org/en/articles/2001/11/cnn-n06.html

http://en.wikipedia.org/wiki/Operation_Mockingbird

http://www.counterpunch.org/2000/03/26/cnn-and-psyops/
I'd need to see the context to know if the Guardian was wrong where they used that exact word, or were simply describing something that exists with some companies or at another level of tapping.
Sony isn't just a corporation, they are a zaibatsu. Essentially many companies with a strategic partnership.
Haskell: Do Notation Considered Harmful (2006)
I think you are making some poor semantic arguments yourself.

The simple case is that of the NSA document ("collection directly from the servers of these US service providers") against google ("The U.S. government does not have direct access or a “back door” to the information stored in our data centers"). These are two competing claims, neither of which have been supported by evidence and hence the burden of proof rests equally with both cases.

Anything else (By the Guardian, Slate, the 'mainstream view on HN' or yourself) is purely speculative.

You seem to place the burden of proof on one claim over another and hold a very specific view on what 'direct access' means.

You also seem to hold a strange interpretation of the Guardian's reporting, but I won't get into that as its largely immaterial to the real subject.
I downvoted this because you linked to an entry without reading it, which drives me insane. Strike "probably" from your post and you have an accurate statement.
SMTP between Google Mail and any server that supports TLS SMTP is encrypted. We seem to have identified one case --- inbound SMTP to a Yahoo MX --- where that TLS connection does't happen.

SSL and TLS are for the purposes of this discussion the same thing; the distinction between the two is actually less important in SMTP than it is with HTTP.
No, it's a private corporation in bed with the state.
Yes. My suspicion is that your first two assumptions are incorrect, and that (1) FISA requests are not personalized under PRISM, and that consequently (2) there is no manual review or check against the abuse of power by providers on an ongoing basis.

This is the only reasonable explanation I can think of for why the NSA would be trying to hide its request volume in the larger volume of overall requests from law enforcement: an attempt to massage the average user-accounts-compromised-per-request downwards when reported to the public. If there are any other explanations you can think of for why it matters how the aggregate statistics are reported, I would be curious to hear of them.

And obviously, abuse of the FISA process renders splitting hairs about what constitutes direct/indirect access meaningless. FISA abuse plus an automated dropbox provides exactly the sort of data access that Snowden and the NSA repeatedly insist they have, while reconciling Google's claims with those of the NSA.
For those keeping score, this is another of the kind of comment that lead me to believe that most HN commenters take the obvious interpretation of "direct access".
Fears of National ID With Immigration Bill
This is a defeatist attitude. But you are voicing your disapproval if it makes it. They know if they are doing wrong if you tell them. Of course it will be like that. But it is the principle of it. They'll probably raise signatures to 250k if it passes, and we should take it past that as well. I am of the ideal that you should never let freedoms go away easily, voice your disapproval.

If it is legal, made by executive orders only via Patriot Act proxies, then we aren't really in a free society anymore. Make a public congressional vote if it is legal, until then is it really legal? What precedent does this set for 1-2 decades from now, not just if it is ok now?
Interesting that now North Korea wants to open talks with the U.S., according to a news tweet tonight (June 15th):

https://twitter.com/NewsBreaker/status/346074606270038017

Strictly coincidence?
Well, I'm not agreeing with you and other yellow peril things at all.

Was just trying to show you that since the dawn of time the "other" has been considered dangerous, evil, etc. by many people. And you reflect very well this mentality.
>I'd say what Snowden is doing fits fits their core values and goals perfectly. He's spreading gross misinformation and propaganda that's damaging to the US government and major US based technology companies.

Err, what? Perhaps you haven't been following the news. Or 10,000 HN stories on this and related issues.

As for it being "damaging US government and major US based technology companies", OK, I get what your priorities are...
I don't understand the (1) and (2) thing. What does it matter whether the requests are "personalized"? In fact, I think they probably rarely are; you can look at Facebook's numbers to see the aggregates suggesting that most requests are for sizable numbers of accounts, not just one.

My point isn't that NSA's FISA directives are surgical; like you, I doubt that they are. My point is that upon receiving them, a lawyer at Google approves or rejects them, not a SQL query.

If you read all my comments on this whole annoying story I think you'll find that I'm rarely (maybe never?) sticking up for NSA, but I am happy to stick up for Google anywhere that I can. Google is actually (in this instance) fighting for your privacy, and then getting shellacked on message boards like this; what's worse, they're prevented by the USG from explaining what's happening. They're being equated with companies like AT&T, companies that appear to be sharing bedding with NSA. That belief is wrong, it's unfair, and it's counterproductive.
I've noticed a lot of downtime on this site too. I agree it's not suspicious. I imagine this site gets a lot of traffic. I'd be curious to see uptime stats over time. Like "downforeveryoneorjustme" except more like Pingdom. Is there a service like Pingdom for checking uptime stats on public sites like this?
This article tackles a lot of confusion that has come with the word over the past couple decades: http://www.thedailybeast.com/articles/2013/06/05/neoconserva...
What? I'm not putting forth any interpretation of 'direct access'.

I appreciate that it seems you are responding to a lot of comments but I really find this comment lazy and borderline offensive.
I doubt that he has a professional conflict in this case. He is nothing if not consistent, his point of view here is exactly what you would expect from reading his past posts on other issues.
All caps are great! They make your message more compelling.

Please keep defending America with your caps lock key on the internet.
I agree with you. This kind of functionality is much better implemented in a third-party tool like sqlalchemy-migrate or alembic that OP mentions in the article.

I much rather a smaller sqlite library than one that is burdened with functionality you just don't use 99% of the time.

The benefit of making an "official" migration library for SQLite is probably outweighed by the opportunity cost of having it developed, tested and kept up-to-date.
It's really unfortunate that most people read 1984 as teenagers (I did). I went back to it as an adult, somewhat expecting it to be a let down. What I found was quite the opposite, there's a lot more nuance in that book than would really be absorbed by high school students being forced to read it.

"The object of persecution is persecution. The object of torture is torture. The object of power is power. Now do you begin to understand me?"
If "anything you say will be used against you", does it mean everyone is under arrest, we just don't know it yet?
Thanks for that helpful reply! Interesting to see that what I wrote out isn't far from what reality would be. I appreciate your informed input here.

But still, as I described in the post, all these bits of tooling that other developers write seem to break all the time anyway. If you look at the quote from the Alembic developer, you can read the level of frustration and spent time that's been put into this. If this is how it's going to work, why not just do it in one location, as I said? All this tooling that's trying to add workarounds seems to be constantly breaking. I think that sqlite could do a better job implementing that logic itself. At the moment, with MediaGoblin, we're trapped and our developers and users are fighting these problems on a very regular basis.

I believe that the constant reimplementation of this in migration frameworks is indication enough that it really is desired and would be useful. Again, I encourage: if there's an experienced developer of sqlite, try testing my theory by running a crowdfunding campaign to add the feature. I bet you'll have proof enough through donations that this is something people want.
The threat was real but it had about as much to do with the mutual distrust and, yes, paranoia of two nuclear-armed superpowers as with anything else.

A case in point is the mythology you mention - Khrushchev's shoe-banging incident was separate from the famous 'We will bury you' line and 'We will bury you' means something significantly less aggressive in context rather than when excerpted and literally translated into English.

"Whether you like it or not, history is on our side. We will dig you in"

This is an expression of the Communist belief in Communism's inevitable supremacy as a political system all over the world, by 'historical necessity' - it's not Khrushchev threatening the Russians are about to come over, crush their enemies, see them driven before them and hear the lamentations of their women.
A graf from Greenwald's original story:

When the FAA was first enacted, defenders of the statute argued that a significant check on abuse would be the NSA's inability to obtain electronic communications without the consent of the telecom and internet companies that control the data. But the Prism program renders that consent unnecessary, as it allows the agency to directly and unilaterally seize the communications off the companies' servers.

Every part of this graf appears to be false!
Less than it takes to keep them flying in other countries. In Death Valley there would not be any reason to spend money on equipping them with missiles.
One bit of evidence supporting that view is that COINTELPRO targeted not only King, but basically the entire civil-rights movement. The NAACP, CORE, the SCLC, etc. were all targeted as potentially subversive. Which in a way they were, since their purpose was to oppose the prevailing social policy in much of the country.
I'm sure the lying part comes automatically when you want to keep your job and you can't tell your boss that you just broke every rule the company has.
Not that these people care, but marketers' inability to see how vulgar they come off is highly entertaining. Who volunteers to be compared to a cult?
>Either way: the original notion that NSA had direct access to the servers that actually operate Google Mail has been found to be unsupported by the evidence published thus far.

Evidence being "Google denies it"?
>This story needs some historical context. This is pointed out in the blog post, but the main reason for wiretapping King wasn't his civil rights advocacy, but his communist leanings.

Marin Luther King, Jr. leaned "communist" in the sense that he was an opponent of the Vietnam war and a proponent of social and economic equality.

The US has a history of branding dissidents as traitors. It's no different than today when the government repeatedly uses "because terrorism" to justify any action it takes.
> Edit: sorry, I reversed the polarity wrt. your question. This confirms the other finding, i.e., Yahoo sends via TLS but doesn't accept.

I am. I sent this message to my personal domain from my Yahoo Mail account just now:

Jun 16 01:46:01 shell postfix/smtpd[29319]: connect from nm4-vm6.bullet.mail.gq1.yahoo.com[98.136.218.165]
Jun 16 01:46:01 shell postfix/smtpd[29319]: Anonymous TLS connection established from nm4-vm6.bullet.mail.gq1.yahoo.com[98.136.218.165]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)

However, it appears that Yahoo's MX did not accept TLS for my outbound reply. That's concerning....
The two are not really competing claims; the argument over whether they are is the argument that NSA could in fact have direct access to the servers operating Google Mail.

I'm not trying to single you out, sorry.
The problem though is that Alembic doesn't want to engineer itself around implementing the tooling because it's so painful (and the author of that tool is himself asking for this kind of alter table support) and sqlalchemy-migrate breaks all the time. And sqlalchemy-migrate breaks all the time, and when I used South and django-migrations (granted, that was years ago), those also broke all the time too. The external migrations path isn't working.
Ha, that has been public knowledge in Europe since the early seventies...
And that The Guardian ran a story contradicting it, and that Marc Ambinder reported a story suggesting that PRISM was a dropbox system and not direct access, and that Declan McCullagh ran a story with sources saying that NSA does not have access to Google Mail, and the NYT running a story with an attached court order from the FISC showing Yahoo availing itself of the opportunity to try to deny NSA access to its servers, and the idea that NSA having direct access to Google is an extraordinary claim requiring not just some evidence, but extraordinary evidence.
If there's no exotic location and free golf involved, fact finding doesn't appeal to them.
How Canada’s shadowy metadata-gathering program went awry
Most Smith Act convictions were overturned by the Supreme Court because people were convicted when they hadn't done anything besides being a member of a political party. That's not the sort of thing that makes anti-communists look good.
Here's two more classics:

http://en.wikipedia.org/wiki/Operation_Mockingbird

http://en.wikipedia.org/wiki/Guatemala_syphilis_experiment

And, strange enough:

http://www.independent.co.uk/news/world/modern-art-was-cia-w...
What Dell tells us about the entrepreneur’s endless fight
Isn't this idea just uShip.com? They raised $18mm recently from Kleiner Perkins, although they are pivoting towards competing directly with UHAUL, from what I hear.
If you're saying some lower-level employee of Google was "turned" by NSA and then lied about it to their manager, we're back into "illicit access" territory.
vulgar sells
I've witnessed first-hand three start-ups slowed or stopped by "ninja" developers using the start-up as an excuse to try some cool new technology that will look good on a resume. In two of them, they were paid the going rate. If you're not getting paid, or getting paid a stipend, then it may be fair to use the start-up as a sort of experiment - the cool experience may be all you get out of it - but this should at least be agreed upon. However, if you're getting paid the going rate, then this is a job, not a party, and your job is to get the job done.

Lets be honest, this sort of thing goes on at large companies too for two reasons: resume stuffing, and job security code. My favorite is when someone builds some really "cool" tech that only they understand, uses it to get a new job, and then quits. If a manager lets this happen, they ought to be fired too, in my opinion.
Why is ALTER never useful in production?
I expanded this comment into a blog post, and corrected some of the factual errors in the comment:
http://projectgus.com/2013/06/sonys-open-source-smartwatch/
Is this actually a bad idea? We won't know until someone tries it, and deals with the issues. The author's attempts at prognostication cloud what is otherwise a well written and perceptive post.
If you did not care for anything other than running Linux why did you update firmware? It's only needed to play newer games and blu-rays, things you say that did not matter to you.

This the essential problem that I see (usually put in ridiculous context like "UASAF used a PS3 cluster and NOW THEY CANNOT!!!!!1!") - if you run Linux you don't care much about the games and even if you do - all the games you already had before the update don't go away.

There is no guarantee that new games will ever come again so your ability to run newer games is not a right. If Sony pulled PS3 off the market instead of issuing an update you would not get any games as well.

But it's still hard to imagine somebody in need of both Linux and newest games simultaneously. If you are one of such people - you are member of very unique and small group. No company size of Sony is ever going to carter to such a small group.
>If I say "pay me 50 dollars, and I will give you X", and then you pay me and I just grin at you, and the WORST you can do to me is to say "fine, then we won't do that again", that's not accountability.

Sure it is. You're presumably selling X to me at a profit, so if you screw me now, you're losing all of your future profits, which keeps you honest when that's worth more to you than the benefit of screwing me out of $50 once. Obviously that only works in cases where we're talking about repeat players, but it's still miles away from nothing.

>Their own consciences AND each other.

What do you think that buys you?

Here is how this would play out. A hundred honest people go to Washington. Someone suggests that they get $250,000 for their local community center. That's nothing out of the federal budget, and it seems like a worthy cause, so OK. Somebody else is a teacher and wants a million dollars for her school, same deal.

Soon people are exchanging support for each others' "worthy" pet causes. Somebody's pet cause is a sister who needs a heart transplant. It sure would be nice to get a heart transplant for everybody who needs one, though it turns out there just aren't enough donor hearts for that. But we can make an exception for this guy's sister, can't we? We need his vote to get the community center and the school money.

Eventually somebody asks for something that costs a billion dollars instead of a million and some people have reservations about spending that much on something that smells like favoritism, but the first one will be something sympathetic like a billion dollars for cancer research (which so happens to target the rare type of cancer that the bill's sponsor has). So that one gets the green light because who is going to tell a cancer patient you won't fund cancer research? And once you set the precedent for a billion dollar personal project the next thing you know federal laws are being passed which give a regulatory advantage to the company of the chosen one who works in that industry and major government contracts are being awarded on the basis of "I don't trust these low bidders I've never met but I know this guy."
I wonder how many technologies out there exist, which would allow us to go far beyond our current state of the world, but we simply don't know about them. In a such an interconnected world, it sure feels that innovations are not very well combined to move us forward. Good article, btw.
And then get the raft of restricted drivers questions, the assorted breakages...