skysight-inc/HackerNews huggingface dataset:
This is a really simple but basic one. Market size might not be billions of dollars, but a basic learning management system along the lines of Teachable/Udemy that allows for code with built-in testing would be used overnight by a dozen code bootcamps and would pull a lot of people out of the other platforms.

Maybe it's just a feature, not a full product, but it makes any "learn to code" MOOC unusable.
How broad would a vendor's coverage need to be on day 1 to be valuable for you?
Google is really focused on GCP and suspect they will continue to be. Constantly adding new capabilities.
While it's reasonable to expect them to get it right, it is obviously going to go wrong from time to time, as with experts working any other field.
"is willing" != "has means to". And to what extent "has means to" ~= "is worthy of" is the age old debate.
Any way to get in touch?
Can you elaborate for those of us unfamiliar with topic?
> which I find is really crucial for a long flight and people are willing to pay for.

Which is why airlines sell properly reclinable seats for $8,000 a pop on long flights. It's called Business Class.
This is in the FAQ: https://news.ycombinator.com/newsfaq.html.

The cutoff is "about a year". That window varies a bit depending on the article. In this case, because the story had such a lot of attention on HN in 2017, I stretched the window; but it was a borderline call.

In general it's just fine to post older things that are interesting.
Google papers have fundamentally changed the industry.
Especially since this (single-user music management application) seems to fit SQLite's use-case perfectly:

* No external services required
* Backups, when the application isn't run, is just a copy
* Good SQL support

I'm not even sure it's possible to get to the point where SQLite is going to fall over in this context (music collection metadata managment).
> Imagine you work on Java. Do you build a DSL/Transpiler to Java so you MAYBE could change later to C#?

Why would I do such a thing? I rarely ever promote abstraction for abstraction's sake and will in almost all cases disagree with premature abstraction. There are levels and for simple CRUD things, a thin contract over your DB is a reasonable tradeoff.

> A RDBMS is even more important that the glue code. Why hide it, why think is something "easy" to trow away?

I don't think you even need glue code. You just need a well-specified contract. You don't want to throw away anything of course.

> Why think is nut to code most or all the code on it?

I don't think that. Often it can be prudent to use plsql/triggers for things. Doesn't mean the frontend of your web app needs to build the SQL in the JS though, right? Just a high level "DoThing" contract with a description of what it should do is fine.
Yeah one of the major points towards vim is pretty much every unix environment has vim installed.
> I'd argue CSV's biggest strength is that it is easy to parse correctly

Not sure I resonate with that. I work with large CSVs of varying provenance every day (I work in big data) and there's always some CSV edge case that stymies my analysis pipeline.

Timestamp parsing is extremely hard if it's not ISO-8601, as well as handling of unicode encoding, missing data, type inference, European usage of , as a decimal point, hidden characters, etc. One of the costs of almost complete freedom in input is the "interesting" possibilities people come up with to stymie your code.

The Pandas read_csv method has tons of switches to deal with all kinds of CSV parsing definitions precisely because there is no standard. Fortunately this covers 80% of the use-cases.
https://pandas.pydata.org/pandas-docs/stable/generated/panda...

Excel's CSV parsing isn't the greatest, but does a surprisingly decent job considering how ill-defined CSV is.

XML isn't really on anyone's radar in the big data world. It's an interchange format yes but is hugely inefficient for dataframe-type data.
Polar is a kind of LEO, at least if it’s low polar. The burn from LEO to Mars transfer will be about the same regardless of whether you’re coming from an equatorial orbit or a properly-aligned polar orbit, so the added delta-v cost will be the speed of Earth’s rotation at the launch point, which you don’t get to use for a polar launch.

In terms of dollars, if you’re flying on a rocket with enough excess capacity anyway (as seems to be the case here), it’s free. If you’re not, then you’ll need to cut your payload or use a bigger rocket, which would be extremely expensive.
I'm talking about business to business transactions, as in automated transactions on VANs and similar data interchanges.

The industry doesn't really use XLSX or XLS because they're unreliable to parse without Microsoft's Office libraries which require Windows licenses, and the two formats have a huge amount of overhead compared to XML formats and CSV (which costs money on VANs).

This is a space where plain text files and binary is still king. Formats like EDI are still massive. In many ways CSV and XML are the new kids on the block.

Only end users are sending Excel formatted files (and PDFs) to one another.
It's only as useless, as, say, the crew on a film, or even a highly-paid footballer. If these people didn't do their tiny part, the entertainment industry wouldn't rake in as much cash.
Or Miller: https://github.com/johnkerl/miller
The only license you need to accept to use this software is Apache 2, so I’m not sure where you’re getting the idea that there’s an Apple hardware restriction.
Sure you can, it's just that progress goes much slower. When I finally got into the 500s I was counting 1/2 and 1/4 reps as progress. For example, if I got 525 for 3-1/4 reps for one workout, and later got 3-1/2 reps, that was progress. Keep in mind for me that later could be weeks or months as I'm not naturally big.
1) Very big, because it includes the whole military intelligence complex in your definition.

2) About 20 hired specifically for this purpose. Admins, doctor and nurse managers, a few receptionists, a few ministers.

3) Not all, not even most of it. Plenty of civil, criminal and driving cases. These are very common. The patent and copyright lawyers are the minority...

4) Not in principle. In practice there are such pathologies, especially in countries without good public higher education.
In principle the education system is to supply a source of educated and intelligent workers for complex jobs, including lower levels for less demanding ones.
The trouble begins when rich and powerful segregate themselves into educational sinecures.
...or even, in most cases, Notepad.
I think Socrata (now acquired by Tyler Technologies) was maybe tackling some of these challenges?
I'm in this situation, that is I'm graduating next year and I already have a job lined up in SV.
Personally, there just isn't anything for me here. I hate winter (depression for 7 months, anyone?). Everything is far. Entertainment is sparse unless it involves 1) hockey or 2) drinking, or both. Pay is shit. Prices are shit. Selection is shit (have you tried using Amazon CA?). Food, at least where I am, is good but no seafood.

Above else, thought, in SV I just feel like I belong. On the train here, an old man snapped at me for reading a maths text with some anti-intellectual line.
Can you supply a link to an author's copy of the article -- or shoot me an email (see profile)? I don't have a subscription to those journals.
There’s a fundamental divide that most of these tools struggle with which causes the sucky behavior: you plan work at the team level but you manage scope at the project level. If you don’t have a clear correspondence between teams and projects (a team owns a project and does all the work), it will be hard to manage in all of these tools, because you’re always missing out on half the story. The process packaging like scrum or kanban merely obfuscates the fact that the team/project mismatch creates the confusion.
The big difference here is that I think a lot more people in San Francisco is a good thing, and a lot more cars on the road is a bad thing.

I never specified that N = the number of people currently living in San Francisco. The outflow of people is somewhat greater than the inflow, as children of existing residents (and often their parents, too) move out and new people move in, limited by the slow growth of homes. However, when we have so many employed homeless people in the streets, it’s pretty obvious that whatever N is, N > M.
https://sf.curbed.com/2017/5/10/15612746/sf-math-teacher-hou...

If we actually built housing in accordance with demand, then the population of San Francisco would be much, much higher, but it wouldn’t be unlimited. I wish San Francisco would get over itself; not everybody wants to live here. Patrick Wolff estimated that we would need 50% more homes in the next 20 years to make housing affordable.
http://www.sfexaminer.com/solve-affordability-crisis-bay-are...

There’s no way we can get 50% more homes with our current system, project by project, 500 units here, 1,000 units there. We need about 200,000 units in San Francisco alone. We can get there by eliminating the density limits. Just look at the zoning maps, how much of the city is restricted to single-family homes. With the more ambitious versions of SB 827, we could have achieved 50% more homes by converting half of the single-family homes into quad-plexes, as was traditional before zoning was invented to keep poor people out of nice neighborhoods.
http://sf-planning.org/zoning-maps
https://www.nytimes.com/2017/08/03/opinion/sunday/zoning-law...
Single process applications if I’m not mistaken. And somewhere in the docs it says: most writes take a few milliseconds at most, so even in a multi-process environment (say 5-10 PHP processes), it should be fine. Haven’t tried this though.
As the parent noted China has a very high debt to GDP ratio. But more importantly it's really hard to tell in China where the state ends and real capital markets begin (well, they probably don't exist outside of Hong Kong). The Communist Party certainly has a saying when it comes to Chinese banks extending loans to Chinese companies (as a part of their 5 year plans, etc). The US on the other hand has a well regulated and audited capital market. This just means that the debt in China is probably even less sustainable than it appears, which in the long term will either require a gradual depreciation of the Yuan or a constant "cooking of the books" to keep borrowers solvent. The former will cause social unrest when lots of savers see their wealth disappear and the latter just leads to an inefficient economy - with unprofitable "zombie" companies being kept alive by constant loan extensions.
What do you mean by this? As in, people hate doing marketing or they don't understand how to advertise? Plus, a mistake can cost valuable data? Where are you losing data?
They use donated money to pay people to write NPR One. If you think of this as an acquihire, is it any different?
This was the case when I graduated ~12 years ago, if anything the situation has got slightly better. When I graduated if I wanted to work in software my options were a bank or a consulting company. At least now you can get more stereotypical software jobs.
Doing some of that kind of super low level code in a C++ static initialization constructor is almost certainly a poor choice. Like setting up memory and the stack pointer. Doing hardware init there can make a lot of sense, but you have to be careful from an architectural perspective.
So how do you get your skills back?
This should be the top reply here. As a frequent flyer, I am worried about this issue every single time. But your suggestion is awesome: the suitcase containing the laptop will be kept safely behind the security line until you have passed through the bodyscanner, shoe swipe, blah blah security theatre; then you can wait there and watch it like a hawk when it comes back out of the machine.

Of course, at a seriously inefficient spot like "London" Stansted this will add about 20-30 mins to your time to clear security, thanks to understaffing and the enormous proportion of Italian/Spanish teenagers who don't pay attention to any of the warnings about removing all the cosmetics from their luggage. End result being a constant backlog of bags to be hand-checked. But that's probably better than losing your laptop. And at a well-run terminal like Heathrow 2 you're only looking at an extra five minutes to get it rechecked. I might just try this on my next flight...
Machine learning is a bigger set of techniques than deep learning. Linear regression and random forests still work fine and can get good results in many problem domains without particularly much compute hardware. In addition non-learned feature engineering can get good results in computer vision if you constrain your input images.

Do you have particular application areas or problems you want to apply ML to?
csv can also be queried by tools like q[1] if sed, grep, etc. are not quite up to the task.

I find this very useful to quickly summarize, filter, reorder/rename columns etc. in csv files.

[1] https://harelba.github.io/q/
The same is true of most of the rest of the world.

Smart computer scientists go to silicon valley unless they're tied down with commitments they can't move.

Doesn't bode well for the future of all other countries.
How to Get Google Adsense Approval Fast Within 3 Days?
By no means do the following ideas intend to imply that this is OK or that it's a problem that should exist, or that the victims deserve blame.

Ugly-up your laptop lid with stickers like a teenager. That makes it distinctive, easy to spot, just slightly less valuable, and just slightly harder to fence.

Hang onto your laptop-in-a-tub until the very last possible second before you go through screening, and then insert it on the conveyor? (I've never tried this. For all I know it may work fine or it may get you detained for 17 hours and cavity-searched by over-enthusiastic fascist bureaucrats who are immune to all explanation.)

Rise up en masse against our military-industrial overlords?

Boycott or curtail air travel? (My personal favorite, even though it's probably playing into somebody's hands.)

In case it happens anyway: Encrypt all your drives, keep daily backups, and maintain an insurance policy that covers replacement cost.
Secure internet for people who travel frequently or work from coffee shops. Filter the “free” internet connection through a TOR router and protect your network/browsing.
Duplicating my reply from here: https://news.ycombinator.com/item?id=16996428

See also the reply here: https://news.ycombinator.com/item?id=16996766

It was shown after the original proposal in 2012 that these signatures can only be observed in "non-equilibrium" systems, so we are actually supplying the pulses, which drive the ticking. The quantum basis for the robust frequency of the ticking relies on dipole-dipole interactions among the nuclear spins.

It is very difficult to understand the complex state of the system after many of these interactions, but in our papers we explore how "coherent" the interactions are by "resurrecting" the signal in what are called spin echoes [1]. Simply speaking, after many interactions the "order" of the system lies not in the nuclear spins individually, but in a complex network of interactions among the spins -- this complex order is not observable (and so the signal appears to "decay away" over time).

Using techniques developed in nuclear magnetic resonance (NMR), we are able to put this "order" back into an observable state, and watch it return. It feels like reversing time, I never get used to it!

[1] https://en.wikipedia.org/wiki/Spin_echo
I don't see how self-sufficiency matters (unless war). In a globalized economy you want to keep the high-margin business and offload low-margin business to someone else.
How to Handle Errors in SQL Server Nested Transactions
"In C an array is generally just a pointer to the first element."

No, arrays are not pointers.

"A String is generally a pointer to a series of bytes (chars) which ends in a byte value of 0 to indicate the end of the string."

No, a string is by definition "a contiguous sequence of characters terminated by and including the first null
character". It is not in any sense a pointer.

Anyone who things arrays or strings are "really" just pointers should not be writing C tutorials.

(See section 6 of the comp.lang.c FAQ, http://www.c-faq.com/ .)
FYI: SQLite doesn't have ALTER TABLE.
Someone had to keep us honest! Fun work, I would have stayed longer but I moved to another area for family reasons.
"I'm having trouble accessing your hacker news skill right now."
Primogeniture was pretty common in the middle ages in Europe. I've read arguments that it's a big reason why the Normans basically went out conquering England, the Mediterranean, etc.
Why the insults ("amazingly stupid", "can't seem to appreciate basic semantics")? Do you think they'll help convince other people reading that you're right? I'm spending my time here with them in mind, not you.

Or do you consider this to be a normal way for people to interact? I can assure you that it's not. If it is for you, then your experience is different from most of the world's, in a way that I suspect makes your life worse.

If someone is coerced using physical force to work, and forbidden to leave under that same threat, then I would call them a slave. I would use the word only in that context. You disagree. It's common for people on the Internet to disagree.

But then what word would you use to distinguish the guy on the fishing boat from the Amazon workers? How would you express that unique human evil? I saw a newspaper headline: "There are 40 million slaves worldwide, most are women and girls". I'm pretty sure that's not counting the Amazon workers. How would you rewrite that headline to fix the undercount (without changing the 40M)?

I'm still trying to understand your definition for the word "slave". I understand that you think the Amazon workers are slaves. Are all warehouse employees? Only those with the rigid, machine-enforced discipline of Amazon's warehouse? Those paid less than some threshold? Manufacturing workers, on the long lines where each worker repeats a single step over and over? Fast food workers reading their instructions off a monitor? A trader glued to his Bloomberg terminal? If these questions are "amazingly stupid", why?
The DNA of Great Marketing
The top five US public companies are Apple, Google, facebook, Microsoft, and Amazon. Three of them are HQ'd here. Netflix isn't top-5 by market cap but I wouldn't be surprised if it gets there before long, maybe displacing Microsoft.

You can certainly make that argument but it's on a whole different level now. Apple is earning 60-80 billion in profit per quarter. Many of the companies you list don't even have market caps that high.
Enterprise Security with RedMonk’s James Governor
A reading list for new engineering managers
"Explanation of needing ';' at EOL is not helping any beginner who'd be better off with the traditional "all C statements end with a ';'".]"

Almost all C statements and declarations end with a semicolon. But compound statements don't.
> Numbers speak for themselves

No, microbenchmarks never speak for themselves.
In any other context, yes, you're missing C library predecessors to main(). But it's specifically referencing embedded systems, which is why hardware initialization and memory segments are mentioned.
> To allow others to impl the data store they want and to avoid having to write blog posts like these.

But why?

For the longest time MySQL didn't support window function or CTEs. What do you do if you're using those? (I'm not sure of SQLite's window support, but I'm pretty sure it has CTEs.)

If you're using postgres, should you not use pg_trgm (trigram indexer to use an index for regex searches on text), postgis (GIS tools), pg-routing (routing tools), ossp_uuid (uuid support), better types (boolean, range, inet, geometric, &c), or features such as RETURNING just because someone doesn't like my choice of database? No, I'm going to do what makes my life easier and my application faster and more featureful.
Canadian working in SV for almost a decade here. Yeah sure, it was about the technology, job, but a lot of it has to do with lifestyle (which the article fails to mention, and can't really be measured).

Since I've been here, I've learned how to drive stick shift, sky dived, and been on way more adventure than what I could have if I stayed at home. It's about adventure and taking a risk. Not necessarily about the money, but being paid more doesn't hurt ¯\_(ツ)_/¯
> Why on earth are people downvoting this comment

all of your comments appear to be getting downvotes, probably because they look suspiciously as though you haven't even spent 45 seconds on the website, despite being in the database space yourself.
What was the project for where you wrote code that self-detected its boot location?
> Why aren't we replacing CSV with it, especially for big data applications?

Big data applications tend to use other structured binary formats like parquet and avro, which any big data tool can typically parse.
Compare and contrast: https://www.bloomberg.com/news/features/2018-04-20/h-1b-work...

( Hacker News discussion at https://news.ycombinator.com/item?id=16887739 )
Perhaps it was supposed to be "%s\n".
UBI will lead to societal degeneration. Prices of essential services and goods would skyrocket; people would lose direction and the small amount of meaning they had.

I call UBI gay-space-communism because it suggests that all human beings are capable of handling themselves outside of employment and daily routine. They are not. The market creates this drive in people, and if you remove it, people will degenerate fast.

Opioids, UBI, and free-love for everyone. Yikes.
XLSX is actually a decent format for business users. The only trouble is it has tons of limits [1], most restrictive being the limit of 1,048,576 rows by 16,384 columns.

I'm working with a client who had to split up their data into several XLSX files because they hit a row limit.

They can of course ETL the data into a real SQL database, but ETL tools for end users are still not a common thing plus not all end users have access to a SQL database.

[1] https://support.office.com/en-us/article/excel-specification...
We use sqlite in data-heavy visualization website (http://atlas.cid.harvard.edu/): We currently have an approximately 22GB sqlite file with the two largest tables going up to 150 million rows each. It still works just fine. The limiting factor seems to be the I/O throughput of the instance and disk of the server (EBS volumes).

Read-only workloads, most of which eventually get cached by the webserver, but still. Each visualization ends up pulling a large amount of data (sometimes in the 10 thousands of rows) so the network / serde overhead and the administration costs of an external database server adds up, though things have changed recently so perhaps a revisit is in order.

Where it hurts is: The query optimizer sometimes stumbles and does silly things (e.g. not always very smart about column / index selectivity statistics). The data import takes a long time (compared to e.g. postgres' COPY), so that's another pain point.
While code review isn't foolproof, this seems like it should have been fairly obvious.
Sounds like GDPR!
This is the kind of information I can only find on HN :)