skysight-inc/HackerNews huggingface dataset:
Oh, that is dastardly, I didn't see that. I also have every FB domain set to null in my hosts file, but I think it's time to extend that to my internal DNS too!
I disagree slightly. One of the interviewees listed some reasons why people leave reviews, but I think it's a little more emotional driven. IMO people leave reviews for three reasons: Very happy, very unhappy, or because they're doing it for some other intrinsic reasons - trying to become elite, ego, doing a public service, etc.
I agree with your argument but you should stop using grandma as an example of a non technical user. I am a grandma.
> Those folks live in Excel and CSV becomes the de facto import format.

Which is an error, and you'd save yourself much pain if you just worked directly with excel files. Most languages have libraries allowing that.
Seconded. I like combining a rather small ORM (for simple access patterns like "get object by ID" or "list all related objects") with hand-crafted SQL for complex reports that join many tables together.
When I am logged on to Google I see companies details in a right hand side pane (when the search term is unambiguous)

For IBM it says

IBM
Computer manufacturing company
Image result for ibm
IBM is an American multinational technology company headquartered in Armonk, New York,
United States, with operations in over 170 countries. Wikipedia
Stock price: IBM (NYSE) USD155.39 +2.71 (+1.77%)
10 Apr., 4:00 pm GMT-4 - Disclaimer
Founder: Charles Ranlett Flint
Founded: 16 June 1911, New York City, New York, United States
Headquarters: Armonk, North Castle, New York, United States
Subsidiaries: Trusteer, FileNet, IBM Global Services, Ustream, MORE
Executives: Ginni Rometty (CEO, President, Chairperson), MORE
Did you know: IBM is the world's eighth-largest information technology company by revenue.
wikipedia.org
I've spent a lot of time thinking about a better format that is close enough to CSV to be practical, but has more precisely defined semantics and structure, also to support better usability (decreasing the need for manual integrity checks after parsing). I wanted at least a defined encoding and defined schema (table definitions with fixed number of typed columns). Optionally Unique Keys and Foreign Keys, but that quickly leads to a situation where there are more possible features with diminishing returns to consider.

I ended up with this [1] and a python implementation [2], and it turned out not too bad. I've also done a more pragmatic C implementation (couple hundred LOC) in a toy project [3] (wsl.c and *.wsl files), and it turned out quite usable.

I think what prevents adoption of such a thing is that it's very hard to standardize on primitive types and integrity features.

[1] http://jstimpfle.de/projects/wsl/main.html
[2] https://github.com/jstimpfle/python-wsl/
[3] https://github.com/jstimpfle/learn-opengl
There are so many companies that collect and sell data on Americans that I don't understand why our Congress is grandstanding on Facebook.

Those other companies haven't arguably violated a consent decree with the FTC as Facebook has.
Then they should look for gainful employment in a stable industry that pays them consistently for their work.
Is the problem that facebook is just too good at what they do? Lots of companies build profile databases of people for various reasons. Lots of the "people finder" websites scrape or buy data directly from municipalities. I'm no fan of facebook, but why on earth do we allow the DMV to sell their data? I'm more upset about that than granny. It seems as though our culture and laws don't have much of a foundation for respecting any right to data privacy of individuals.
>from the providers' point of view, subscriptions are absolutely desirable

It's fair to say that businesses mostly like subscriptions but balance against that a few factors.

- Consumers often do not which can make it harder to acquire a customer in the first place.

- It's an ongoing revenue stream that can be canceled at any time rather than a larger up-front purchase.
I don't think this is clickbait, since the entire point of the video is the one quote. However, I do think the entire post is useless.
If it's that big a deal just tether it to the phone.
Access control should be in the DB itself, or anyone can connect directly bypassing the ORM and have full access!
Huh, it's almost like we need ads in order to have free content
Because "pretty extensively specified" does not mean "fully specified and perfectly implemented".

The equivalent page for CSV would not have an end.
The title is the whole point of the video.
hstore querying is quite slow (and GIN indexes on hstores are pretty massive). I'd always go jsonb over hstores these days, but jsonb has the same indexing problem. JSON has a well-optimized/spec compliant serializer/deserializer in every language you can imagine as a baseline, whereas hstore does not.
> doesn't play well with Pip

The focus of my current right-now dev work is to teach conda to better understand non-conda-installed python packages so we respect what's already there even if it's not a conda package.
I think the one phrase we all need to learn these days is

"Problems with your business model is not society's problem"
Out of the same league, where "If it costs nothing, it can not be worth much":

Me working for a large multinational: I would like to support this OpenSource project that we regularly use, can we donate some money?

Boss: OpenSource? NO WAY. We're not commies.

Me: I would like to buy 100 CD's with software to distribute among my colleagues since it will give them the change to learn about some really cool security tools, all installed on a bootable linux live CD.

Boss: 10k Only? GO FOR IT!

That same night, we spend burning 100 CD's and printing 100 labels.

And that, dear readers, was a (really small) push into the development of Auditor, the Linux Penetration Testing Distro. we all learned to love (Hi Max!).
Who are these people who, before today, thought that Zuckerberg actually did care about users? Didn't they know about the "dumb fucks" thing?
In a way Freebase and DBpedia were/are practical applications of the concept. Now when you search on Google they try to understand a simple query and send you the answer. In Freebase you could write queries about facts retrieved from many sources.

The utopia is more than this but I assume that few people will used these tools directly.
Ok, credit card knows what you buy but not the last time you talked to your friends and what you said to them.
Chrome extension for finding the cheapest flights to awesome travel destinations
Do you mean you read a book and then asked people when you got stuck?
Yeah, I mean those are not always true but they're also not always false.
As much as I loathe too I have to defend Facebook. Not to say the data collection practices are not
egregious ,let's not forget that these are industry-wide practices and that the hearings have less to do with protecting users than it is political grand standing. Experian, anyone?

It's almost as if we've collectively forgotten the data breaches of the last 12 months and how exposed American and EU citizens are now to fraud because of lax security practices by credit scoring companies.

I guess the difference is that Facebook is actively sharing this data and Experian was hacked, because their breach was the action of an external agent, they are less culpable.

From the UK it looks like this. Trump used data, Facebook helped, politicians are upset that Trump was helped by the Facebook platform (conveniently forgetting that every campaign before that has used large amounts of Facebook user data), let's punish Facebook.

Facebook doesn't care about users, but politicians who have the power to change this care even less.
>That's a problem of incentives, technical difficulties, UI, bitrot of invisible metadata, etc

Perception, culture, linguistics, time, reality.
It will end up being a shoot-em-up clone of Star Wars. All the psychological aspects of the books just won't be interesting enough to mask the fact that there is essentially no action or excitement or romance. They'll probably turn Harry Seldon into sort of a Yoda character. Using the precepts of psychohistory, what a few people will think about the film is unpredictable but as a population of movie goers, it will be boring and unintelligible.
Cached: https://archive.is/wFAL4
Project-based learning, or designed thinking, doesn’t just help students “get” the material in time for a good grade on the test; it also helps deepen their appreciation for what they learn.

The article is a little slanted, IMO, towards tech understanding. My husband studied polysci. There are not many projects, and hands-on methods for getting all of that, and I would say that for MOST degrees. Tech, on the other hand, totally hands on, project based for me.
Because FB is the most visible example of it. c.f. fraud and embezzlement enforcement. Regardless, whatever comes out of the case w.r.t. FB will determine policy for the other personal-data companies.
The Inequality of Pedestrian Deaths
Stargate: Universe spent its brief run exploring desolate planets and bickering while hurtling through uninhabited space. It didn't work too well.
It's the age old trap of trying to fix a format or a protocol, only to end up with yet another format or protocol.
> This is unacceptable in large business settings today. A list of assets alone can be tens of millions of lines. A list of transactions?

The original comment specifically mentioned excel. Excel can not work on "tens of millions of lines" and your objection is thus irrelevant.

If your data does not come from excel, there is obviously no reason to make it go through excel. There is also no reason to make it go through CSV, make it go through an actual serialisation format which is not mostly unspecified garbage since you're programmatically serialising it for interchange and hopefully not expecting human edition of the intermediate megabytes of data dump.
Ah, good point. I avoided the movie because I have a great disdain for Mark.
The determining factor is hardware decode support, not efficiency. H.264 is worse than H.265, but if that movie was H.264, it would have played back flawlessly due to the Intel hardware decoder.
I'd suggest that people popularize the phrase "Exxon Valdez of Privacy" but that may be old enough that people have forgotten it. Maybe substitute "Deepwater Horizon" or "Fukushima".
If you want even more prescient, go back fifteen years to see him post a link to https://www.wired.com/2003/01/google-10/

"The company's growth spurt has spawned a host of daunting questions that no data-retrieval system can easily answer. Should Google play ball with repressive foreign governments? Refuse to link users to "hate" sites? Punish marketers who artificially inflate site rankings? Fight the Church of Scientology's attempts to silence critics? And what to do about the cache, Google's archive of previously indexed pages? "

Whenever a colossal disaster happens, there is nearly always someone who was telling anyone who would listen what the problem was years in advance.

Felten is not as famous as Schneier but just as important, and has been doing lots of important work such as fighting against electronic voting systems.
Absolutely! It's complete grandstanding. I really wish someone would go Howard Hughes on them.
XML, JSON, any specified format is a better machine-to-machine format than CSV.

I've seen the JSON-object-per-newline pattern a few times, but why not use a JSON array then? Then the whole file remains valid JSON.
No, I wasn't trying to be anyways. It just seemed from your comment that you were very amped up, and jumping around a bit.
The market makes up its own mind
Normally, I kinda think these sorts of articles are a bit defeatist because reality isn't as bad as they suggest. When you set out to put an address widget on your web page, you are not necessarily bound to create something that can correctly ship a package to some random location in Wakanda. Usually you're just shooting for a single country, which has a de facto accepted layout which even if it is technically "wrong" the local delivery services already deal with. So there isn't a great need to worry about 80% of the "falsehoods". Software gets pretty damned expensive if every single address field has to be free of all errors of that sort, and every name field has to handle all 7 billion people in the world with equal fluidity, etc.

But this is one case where I'd say the style conveys an accurate impression. The problem is basically that there is no such thing as "CSV". You can't build a parser that handles all the issues because a lot of the mistakes made during generation lead to fundamentally ambiguous file formats. In the simplest case, how do you parse:

Quote, Speaker, Time
I came, I saw, I conquered, Julius Caesar, 47BC

A generic library can't really help with that. You're going to have to do some heuristic munging. And bear in mind this is just a simple example; when you've got megabytes of vaguely CSV-inspired ASCII octets it gets bad fast. You'll yearn for CSV files generated by someone who thinks

"\n".join(",".join(str(y) for y in x) for x in [[1,2],[3,4]])

is all you need to write a CSV output function; the real world gets much more perverse than that.

As file sizes increase and the errors increase, while you get an increase in the number of issues that a library may be able to help with, you also get an increase in the number of errors in which the resulting CSV is simply fundamentally ambiguous.
> What's your preferred approach to store table definitions and migrations?

Schema versioning and ORM can be quite orthogonal, unless your ORM decides to be as opinionated about stuff as e.g. ActiveRecord. In Go, I use migrate [1] and Gorp [2]. Granted, I have to tell the column names to Gorp manually, but that is a minor annoyance which I'm happy to pay for clean separation of concerns.

> Doesn't [using SQL for table definitions and migrations] make them more susceptible to mistakes?

What mistakes do you have in mind? Typos etc. will be caught by tests that access the database. That leaves actual design errors, which can happen just as easily in ORM code as in SQL code.

[1] https://github.com/mattes/migrate

[2] https://github.com/go-gorp/gorp
I just learned that SQLite doesn't accept any patches, so the argument that Git is the industry standard doesn't matter to them, and they just use what they prefer.
Or give your data to, I don't know, companies like Equifax...
I use `git config pull.rebase true` too, but that doesn't mean you _have_ to stash first, just as rebase manually wouldn't - depends if there's a conflict.

Same is true of merge-based pull.
Sure, but do you know them? If you don't, how will you judge if their reasons are valid for you?

In any case, the simple fact that the reasons are unknown speaks pretty heavily for the case that the reasons aren't universally valid, so you would be learning a dialect of C++.
Something that's being lost here is that this article is in the same vein as the seminal "Falsehoods programmers believe about time" [0] which contains quite a few statements which describe user error that developers still have to deal with (e.x. "The system clock will always be set to the correct local time.")

0. http://infiniteundo.com/post/25326999628/falsehoods-programm...
JPEG or PNG or GIF? Which Image Format Should You Choose?
Not only can they be, they are. Or they stop accepting CSV.
Most politicians care more about people than Facebook does, if for no other reason that they need to be reelected.
The Mule should be played by Donald Trump.
Facebook never was part of 'the free internet', in fact it is a force strongly against the free internet.
Lot's of companies compete in lots of sectors. If you buy a Ford van you're not going to go to chevy. But that doesn't change the fact that Ford's primary business is normal cars and chevy competes in that industry. If I ask who MS's biggest competitor is it's Apple. If I ask whose Google's biggest competitor it's probably MS Bing.

Facebook is overwhelmingly known for its social network and it's biggest competitor is probably twitter who problem has ~10% of it's customers.

The point is that none of facebook's competitors come even close to facebook's size no matter which portion of Facebook's core functionality you're talking about. Plus, facebook is not diversified in the way that SAP or Honeywell is.
The real problem is when the user complains "but it works fine in Excel." If it does work elsewhere then in the user's eyes it's your app's fault that it cannot read a working file.

Yes, it does work in Excel but it's because of 40 years of various edge case fixes that you just cannot replicate in their entirety in your app.
I've always maintained that the Will Smith movie captured the spririt of the Asimov Robot stories, in an accessible way. I never understood the hate it gets.
No, we don't. Content on the web was free for years, and on the internet for decades, without a single ad anywhere.

The only things that need to be paid for are the physical infrastructure and power usage, employee salaries, etc (typical business costs,) but not the content, and not necessarily through advertising.
That would be a humongous boon, especially for Windows. I really wanted to use Conda, but I really, really needed Pip for packages. I don't really remember what blew up when I tried it, but it wasn't pretty.

Glad to know that soon enough that will be just a memory :)
Time to come clean about my first CD burner story I guess. In 1998 they were pretty common already but I was a poor student in a foreign country so could not afford one. Office Depot (I think that's what it was called) had a good return policy, so the scheme was as follows : I would buy the CD burner and a friend would return it a few days later. Only the box would contain my old plain cd rom drive and not the burner. We relied on the incompetence of the Office Depot staff and hope they would not recognize the difference. Why was my friend to return it then ? Well simply because he could claim ignorance of the matter in case they discovered the swap, and I could later choose to return the real CD burner or keep it, depending on my finances. Well as it turned out the staff were really as useless as we had hoped and we burned many, many cds in the months thereafter. (Most of which came from HMV which also had a pretty good return policy until they caught on)
So hmv and Office Depot there you have it : my sincere apology for my youth mistakes !
> Arguing with some random dude who thinks he knows more than me is not really fun.

That comment is not very nice and does not serve your otherwise interesting comments.
Warranty void stickers 'not valid' in US, says FTC
In most instances you are stealing views from the original creator and as a result of those lost views, revenue.
something about society too, how much users really care ?

people may complain, rant, scream, walk, strike .. but do they really want privacy ? and do they know how to define it ? at what cost (going off facebook/web ? paying for guarantees?, doing paper work to ensure political class write these down as laws ?)
Totally agree, I guess the apologists have business model problems.
This is one of the reasons I really like uMatrix. In addition to giving really fine-grained controls over what you load (by type of asset, by domain given current domain), it gives you really great visibility into what a webpage is doing, and where it's sending data. I have mine set up to act like NoScript by default -- to block first-party JS.

You end up noticing some surprising things. For example, yesterday I read an article (http://www.lowtechmagazine.com/2015/10/how-to-build-a-low-te...) from Low-Tech Magazine, about how to build a "low-tech internet" (an article I really liked, BTW). Happily, the page loaded just fine without any JavaScript, but I still noticed in my uMatrix panel that the page was trying to load various assets from 20-some domains, about half of which uMatrix recognizes as tracking domains. A little disconcerting for a site called Low-Tech Magazine.
That should have been the target of his rant not that they won't turn on LTE updates. He comes off sounding like first world problems in the bulk of this article.
The original statement was about coverage of the breadth of the specification: if you're consuming data you need to accept everything the specification allow, but if you're generating data you should be producing as little of the specification's productions as you can.

This does not work when the format is mostly unspecified and the vast majority of what you'll find in the wild is a tire fire.
Sessions: The VIM Feature You Probably Aren't Using
Russian trolls hit Reddit, too
Good catch:

https://www.gnu.org/software/taler/

I'll contact Christian Grothoff today and make sure nothing suspicious is going on and ask him to update the GNU forward if the domain has indeed changed.
Quant Loader Trojan Spreads via Microsoft URL Shortcut Files
Which sites is that on? When webmasters use Facebook comments or authentication that gets inserted?
The WHOIS data for gnutaler.org seems possibly fishy too. Lists an address in Germany, but the city/state name are not actual cities/states and the country code is turkey.
> They can get map updates for free, all they need is WiFi access... am I missing something here? There is free public WiFi all over.

The updates can't be triggered by the user, so you have to leave your car somewhere (or your phone in your car) for a extended period of time to hope to get an update.

There's also not free public parking everywhere (much less free public parking with WiFi coverage), especially in the urban areas where “my parking is remote from my living space and doesn't share wifi with it” is most likely to be an issue.

Also, outside of certain urban areas (and this can still be a problem outside of urban areas) there isn't free public WiFi everywhere.
Untracked files are not stashed, that is true.
> But for how long?

usually hype cycles last a few months, until the next outrage takes over to keep people busy.
when I got my first home PC in 1993 there was IQ test software on it. I took the test tired, drunk, happy, caffeinated, all sorts of ways. I had huge swings in the score . +- 40 points for me personally.

I came to the same conclusion as you - that it only ranks how well I can take an IQ test.
I don't disagree with you. That is why I said positively correlated, rather than perfectly. My comment intended to concur with the parent comment: "the two parties here have fairly well aligned goals -- less freedom on the Internet. The senators want to show that they're doing something, and Facebook welcomes regulation in order to suffocate would-be competitors with it."

We most likely will not see any trust-busting as you're describing here. We'll more likely just see more side-effect legislation like the net neutrality repeal or FOSTA/SESTA that will just consolidate power for a few parties and the government at the expense of small businesses.
I would say “curating” of political ideas is a big issue. See Ted Cruz’s questioning of Zuckerberg as an example. It’s not just data but how Facebook has such a dominant role in steering political discourse. There are more issues at play than simple data hygiene.
I highly recommend listening to Sheryl Sandberg's interview with NPR from last week. It is a great example of the manner in which people behave in order to successfully stay out of jail.
https://askubuntu.com/ seems a better match for that specific problem.
I don't think you can fix CSV. You could define a reasonable-by-today's-standards subset - I'd say, UTF-8, 0x20 delimiters and per-lexem choice of C-style identifiers or string literals.

But I need more structure. My format supports an arbitrary number of tables in a single file, some automated integrity checking, and direct conversion of text from/to runtime structures (e.g. C structs, possibly with automatically allocated backing buffers for variable-width datatypes) with only little boilerplate. So yes, that's a different format.