skysight-inc/HackerNews huggingface dataset:
> A German company holds a very large amount of profile data and wanted to search through it. On disk storage in the 100's of gigabytes. Smart encoding of the data and a clever search strategy allowed the identification of 'candidate' records for matches with fairly high accuracy, fetching the few records that matched and checking if they really were matches sped things up two orders of magnitude over their SQL based solution.

Right. But you've still not made a compelling argument that the "smart encoding of the data" and "clever search strategy" couldn't be done with/in SQL. I'm quite sure you could probably get a magnitude or two of improvement by diving down outside of SQL -- but there's a big difference between using the schema you have, and setting up a dedicated system.

I was perhaps not clear, but I was wondering if it's not often the case with real-world data, that you could create a new, tailored schema in SQL, and do the analysis on one system -- especially if you allow for something like advanced stored procedures (perhaps in C) and alternate storage engines.

I suppose one could argue when something stops being SQL, and start becomming SQL-like. But the initial statement was that "For some problems SQL would already be way too much overhead.". The implication being (as I understood it, at least) -- that you not only have to change how the date is stored and queried, but that SQL databases are unsuitable to that that task -- and that by going away from your dbs, you can suddenly do something that you couldn't do on a system with X resources.

I'm not saying that's wrong -- I'm just asking if a) that's what you mean, and b) if you can quantify this overhead a bit more. Are using bitfields or whatnot with postgres going to be a 20% overhead, or a 20000% overhead in your scenario? Because if it's 20%, that sounds like it can be "upgraded away".
Early designs did. But CAES has begun to store the excess heat that is made during compression.

For example:
http://sigmaenergystorage.com/?page_id=447&lang=en

Advances are being made at _all_ energy storage solutions. Frankly, CAES is the most promising, now that they've figured out how to store excess heat during compression.
Batteries are also a form of over-provisioning, just a distributed one.

The real market potential for batteries is not in those parts of the West where there is cheap, reliable mains. It's those places where the grid is unreliable enough that people with money would buy generators, or places where the grid is a tiny island (Hawaii, Orkney).
sounds good to me, but why people not doing that? ssd price too high?
I mean in terms of the philosophy behind it. A programming language is a tool for thinking. And each language, fits better with some specific way of thinking about a problem. So you are mastering a programming language when you can adapt the way you think about the problem to fit the best characteristics of that language. And i feel the discussion about using Scala or Clojure should be more about that, and not so much about the amount of code needed to write a program on each one, or the fact that one has Type Inference in its nature, while in the other it is optional. Just my 2 cents ;)
Those tools do a very different type of job (archiving generated html). Rsync is just file transfer.
The community tooling around MySQL is impressive compared to what's available for Postgres. I've heard the same from people who have switched from SQL Server and miss the SQL Server Workbench suite.

My hunch is that this gap exists because MySQL is often the default DB choice for new developers, whereas Postgres tends to be used by people already familiar with database CLI tools. Once you learn how to use the command line effectively, you'll be able to work a lot faster and won't want to use a web interface again, which is to say that you will become yet another reason for phpPgAdmin to be neglected.
And unfortunately it's working. If/when I buy a new GPU... it'll be nvidia. I've owned four ATI/AMD cards and only one nvidia card (integrated in a laptop), but it seems on many big-name games I want to play these days, there's always some problem on AMD. Sure it's shady business tactics, sure it's unfair competition, sure it's sabotage of their competitor, but if AMD can't do something about it, there's no logical reason to stick with AMD.

I built a custom AMD computer to meet the recommended specs for Watch Dogs and after launch I find out that the AMD recommended specs don't provide the same performance as the recommended Intel/nvidia specs. Something like 70% of the performance is gone on AMD. I don't want to worry about politics, I just want to play my game.
Horror Stories of the Startup Castle
I'm not one of those "everything Obama does is wrong" people, but the man has been frightfully wrong on Police State issues, IMO, as well as persecution of whistle-blowers.
Shameless plug: have you checked out mnubo.com? We provide a complete data pipeline, from ingestion to visualization, for connected devices like yours. PM me at bborowin@mnubo.com if you'd like know more.
Of course caching is enabled. Reads are all coming from cache. Performance problems come on writes, which currently wait for the disk, always, even though I don't care if they do...I'd be happy with it writing to disk five minutes later, but that's not an option I've been able to find.
These tips can largely apply to anything. I work in Liferay. I wish I had half of Wordpress's community and documentation.
That said I feel sorry for people who have to work with Wordpress.
"Also, as someone who has chosen to not pursue the rat race, I often feel pushback and criticism from those who have chosen it, as if my choices directly threaten their own."

great response
But the black magic lies exactly in the handshake. You need to generate the RSA key (or more broadly, an asymmetric key) in order for the handshake to take place.

An analogy I like to use are boxes with locks. Imagine that, given a lock design, I can produce a bunch of boxes with the exact copies of the lock and keys for the lock. Symmetric crypto corresponds to everyday locks: if you have the key for the lock, you can open or close the lock as you wish. Asymmetric crypto corresponds to a special kind of lock, which comes with two keys: one kind for locking the lock (the public keys), another for unlocking (the private keys).

Now if somebody wants to communicate with me securely, the choice of lock affects the key distribution significantly. If I have just symmetric locks, for each person I want to talk to, I have to create a new box with a separate lock, and make sure they are the only ones having a key for this lock. The latter I have to in person. Meaning, if I want to talk to somebody on the Internet, I first have to meet them in person (that is, out of band), before we can communicate securely. This doesn't scale well. With asymmetric crypto, however, I can just leave a bunch of open boxes and corresponding locking (public) keys lying around (the keys for the different boxes can be same or different, doesn't matter). If Assange now wants to send me a message, he writes it on a piece of paper, puts it in one of the open boxes, and locks it with the locking key. I'm the only person with the unlocking (private) key for this lock so I'm the only one who can read the message. This is just basic asymmetric crypto.

Another thing Assange could do is, instead of leaving a piece of paper with the message, to leave a piece of paper with a fresh design of a symmetric lock. Now I can safely create my own symmetric box according to the design, put messages inside, lock them and leave them around for Assange to pick up. This is secure since he's created the design and he's the only one apart from me who can open such boxes.

This is a 10,000 foot view of how to do the switch from asymmetric to symmetric crypto, but there are many other questions that need to be addressed to get an actual protocol. Thes include "how does Assange know this public key belongs to me" (answer: certificates and certificate authorities), "how do I know the message came from Assange" (answer: digital signatures), "what purpose does this message serve" (tagging and nonces), "what kind of designs I can produce locks for" (ciphersuite negotiation) and so on. Additionally, there's all kinds of things that can go wrong when implementing this - basically, even though we know how to check the security of a high-level design of a protocol using formal methods (the so-called "symbolic verification" tools), we don't really have a systematic way of translating these designs into actual implementations while preserving the security guarantees.

Also, there are direct key-exchange protocols (such as Diffie-Hellman) which work differently than the basic idea above.
ArcGIS Online supports GeoJSON
I'd second this notion, having moved just about every year for the past 16 years, sometimes to different provinces and countries it's exhausting and can be extremely expensive. Sometimes you don't realize any savings from a move for many months or in some cases, years... and if you're unlucky, you don't realize any savings at all and in fact, hidden costs that you hadn't accounted for can end up costing you money or sinking you into a hole. So moving can also be a risk, sometimes, it's better the devil you know than the devil you don't.

Moving cheaply requires good timing, a handy amount of free labour (i.e. friends that own trucks and owe you favours or like you enough to help just out of the goodness of their own hearts), a little good luck, a keen eye on available real estate and good negotiation skills. Without all of these lining up, it's likely to cost a bit.
There is also this article (which is not about doing logic with light but using light instead of electric current for moving information from one place to another) from a week ago: http://www.extremetech.com/extreme/205713-ibm-announces-sili...
That's bullshit imho, Snowden's documents showed that GCHQ spied on American citizens & NSA spied on British citizens so that they could bypass the whole "don't spy on nationals" rule.
Your boss who wants to have metrics/results/widgets to show her boss when she wants them.

edit: it ain't a daycare.
Another step closer to the point where CPUs and GPUs will be unified.

32 SIMD registers holding 512 bits each. Ability to process 1024 bits (128 bytes, and even more in some scenarios) in a single clock cycle. Sounds more and more like a GPU to me.
Not everything that has been processed is bad. Just like taking an aspirin pill works much better than trying to get pain relief by eating white willow bark, some processed foods are concentrated goodness. Whey protein, resistant starches and partially defatted coconut flour are all absolutely amazing ingredients that are significantly healthier than the whole foods they came from.
A nice summary list. With one tiny page style quibble, in the summary table I can't tell, without a lot of cross reference, whether an X-mark or a Check-mark is the "has this feature" indicator, ((it's the Check-mark)). Probably it would be better just to leave the X-marked column blank.
Does your RDBMS's built-in caching not handle this pretty well? Just up the cache size, e.g. in PostgreSQL changing effective_cache_size https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serv...
With the world going multi-core more and more, the question that comes to my mind is - are the languages designed without concurrency in mind doomed to fail? It seems C++/C#/Java all look at concurrency/parallel programming as an afterthought - another library here, another library there, but none of them approach it as a fundamental programming necessity, on the same level as a basic loop or "if" statement.

Ultimately, it's possible to be semi-successful in designing scalable, concurrency-aware software with just after-thought libraries and language support, but is that really the way forward, long-term?
Why Entrepreneur Students Shouldn't Get A's
So are a majority of junior computer professionals in other states paid an hourly rate?
There are two issues: knowing and discussing.

In Sweden, they are separate. In US, they are unavoidably mixed.
You can read more about this project on its about page: http://www.oldnyc.org/about.html
Well, you've piqued my interest! :D
Spotify itself also has some frequently-updated curated playlists for most genres that are great for discovery.
Yeah, but most of us buying expensive phones are on-call and need a powerful wifi hot spot and a bunch of other things out of it.

I literally can't think of a company where you end up on-call and the expectation is you can respond in [at most] 15 minutes to an emergency/alert. That means a laptop + mobile wifi hotspot [phone] everywhere you go.

The cheaper phones tend to have worse reception. In my office, I used to only get Edge due to the thick walls. My Nexus 6 has no trouble getting to LTE.
i argued against this and still won $5000 against a former employer a year after i got WARNed out.
If your threat model includes intelligence agencies, set up your own HTTPS with HSTS and HPKP and don't use CloudFlare.

Wow, that was easy.
I've been thinking this too.

For example, it is very easy to write badly performing code using ORMs. And yet, ORM is often chosen for good reasons initially to give development speed (e.g. Django forms) The problem with quickly prototyped ORM based apps is that initially the performance is good enough but when the data grows the amount of queries goes through the roof. It is not the amount of data per se, but number of queries. Fixing these performance problems afterwards for small customer projects is often too expensive for the customer, but if there were a plug'n'play in-memory SQL cache/replica with disk-based writes, it would easily handle the problem for many sites.

Configuring PostgreSQL to do something like reading data from in-memory replica is likely possible, but I see that there would be value in plug'n'play configuration script/solution.
"MySQL has generally got your back in the 'less safety for more performance' arena:"

So, this was never entirely clear to me, but now that I've read a bit more about it, this might actually be exactly what I want (which is to not have the system wait to return when posting new content, and just assume it'll end up on disk eventually). The talk of not being ACID made me nervous and maybe switched off my brain. I guess it just means I don't need or want ACID in this case, all I want is a consistent database on reboot.

So, I guess maybe this does what I want, but just to be clear: In the event of data loss, the database will still be consistent, correct? i.e. we'll lose one or more comments or written pieces of data, but the transaction it was wrapped up in won't be half finished or something in the database? (I recall MySQL had issues with this kind of thing in the very distant past, but I imagine that's just bad memories at this point.)
Dude, Upgrade Your Professional Image
What supplements to stock for bodybuilding
>and performance is always a problem.

Have you proven that your performance issues are SQL related? If you configure mysql correctly and give it enough RAM, a lot of those queries are happily waiting for you in RAM, so you have a defacto RAM disk. Finding your bottleneck in a LAMP based CMS system is fairly non-trivial. Think of all the php and such that runs for every function. Its incredible how complex WP and Drupal are. Lots and lots of code runs for even the most trivial of things.

This is why we just move up one abstration layer and dump everything in Varnish, which also puts its cache in RAM. Drupal and WP will never be fast, even if mysql is. Might as well just use a transparent reverse proxy and call it a day.
For reads, yes. It's fine on reading data. The problems are writes which always wait for disk consistency to return.
Yes, http://cryptome.org/2013/11/snowden-tally.htm

Because this leak wasn't a "publish everything at once" leak. Snowden wanted the journalists to decide what to release, and that requires work, which takes time.
The doomsday vault: seeds that could save a post-apocalyptic world
Placing JSON store at equal footing with RDBMS is a little bit loaded.
what kind of job does your wife have?
The 365 million is 18650 cells, they retail for ~$8-10.

So hopefully less than $3.6 billion in cells (factory direct, new factory aimed at reducing costs). There is a bunch more of stuff in the integrated units, at the quoted $3000 price, 4.3 million of them would be ~$14 billion.

I think pumped storage is great (I've posted enthusiastically about it here in the past), but the sneaky factor is that they did a good job siting the ones they built decades ago, many of the great spots to build them are in use already.

(I edited in the $14 billion after I realized I was talking about a product with a proposed price, I had originally said something about less than $10 billion)
nice! now i can use GeoJSON A-N-Y-W-H-E-R-E!!
This is business, unfortunately.

Compared to suing your innovative competitors out of business, then buying and burying their technology like Creative [0] or suing countries for attempting to warn their citizens that cigarettes are bad for your health [1], it's almost ethical.

[0]http://en.wikipedia.org/wiki/Aureal_Semiconductor#History
[1]http://en.wikipedia.org/wiki/Philip_Morris_v._Uruguay
This is absolutely fantastic and fascinating. Always glad to see the continuing transition of libraries from physical to digital. Also, imagine the potential for a mashup between this and the current/historical Street Views available on Google Maps.
Is it acceptable behavior that spy agencies spread malwares? I do find that attitude a bit surprising as sabotage and other active attacks in peace time was considered outside the spy agencies operation scope in the past. I do want spies that spies, police and police, and tax collector that collect taxes. I do not want them however to overstep their bounds in the chase of those goals, and I object when they target innocents with military weapons.

Spy agencies should SPY, but spreading malware in the name of spying goes a step beyond spying. A line need to be drawn somewhere, and what better place than where sabotage begin?
US poll show cross-party antipathy to surveillance
Well, there is reducing consumption and then there is prioritizing consumption. I think we fall more into prioritizing consumption. We do make quite a lot more than most people in the country, say nothing about the world, and we spend it all (in different ways, including on our retirement), but how we spend it is important.

If you want to have a fun lifestyle, you have to cut your living expenses. Housing is pretty much everyone's biggest expense, which is silly because houses aren't "fun". I don't know anyone who has parties often enough to justify large living spaces. I do really like hosting parties, but I can do that in a park or a private room in a restaurant more easily than at home, and it's a hell of a lot cheaper than keeping a large dining room and living room all the time.

I also think of it as being brutally honest and pragmatic with yourself. You have to decide for yourself how you will live your life and not let other people's expectations decide for you. Want to be a literal rockstar? You can do it if you give up on the same sort of vacations your coworkers are taking. Don't want to give up on vacations? Admit to yourself you like vacations more than being a rockstar and give up on spending money on being a rockstar.

I think a lot of people distract themselves with spending money on things that they think they are supposed to want, but ultimately aren't actually what they want. For example, despite what they might say out loud, a lot of people really just want to sit at home and watch TV. That's absolutely fine. Just admit it to yourself and stop spending money on fishing equipment that you never use. Then you'd be able to sink the money into a massive home theater system instead, something you'd actually get to enjoy on a regular basis.

If you try to do both, you'll end up in over your head in debt and not getting to fish or have a big home theater.

We have friends who are trying to "do both" on travel and a large house. We have been to their place for parties twice in the last year. They've been to our place for parties twice as well. We didn't have quite as many people, but I think that also made for a better party. They don't make any more than we do, they are constantly in debt, it's ruining their marriage, and now she just got pregnant so their income is about to get cut in half. He just took a job he hates because they will need the money.

If my wife gets pregnant, our options will be to... stop booking international flights. And that's what we're planning on. We're actively planning on shifting how we enjoy ourselves to accommodate our means. Luckily, we both really like board games. I've personally put off playing games over the last few years so I could focus on travel, get the most out of it, then switch to gaming together when we settle down.

Whatever it is you do, do it completely. Don't half-ass two things, whole-ass one thing :)
this is such a great post, it should deserve to be on the top.

I see many bad comments in this topic but I believe you described an honest example of what it means to make good sacrifice and live a good life.

I am on the lookout for a new job, was searching for a remote one, and in italy it isn't as easy as outside (at least for now).
If I can't find anything suitable I'll seriously consider the part time option, and I wonder why it hasn't been taken into consideration up until now.

We are not our work, we are what we are indipendently of our work.
Work is a tool, sometimes to freedom, yes, but mostly to be part of this world, to allow us to experience the world, because that's what the money partly does.

I agree with you and with the article, we shouldn't work that much, and if we all worked less, it's also probable that even more people would have a job since we all work less.

Improving life and human interaction, living with the people we love, that's what really matters.
Nvidia technologies have been heavily used in both these games:

* Watch Dogs

* The Witcher 3

Both suffer from technical problems with the graphics, don't perform as good as they could (across all systems) and look considerably worse than in the previews.

I like Nvidia cards and drivers, but as a developer I'd stay very clear of Nvidia-exclusive technologies. I guess the business guys must like the free advertising.
I have an overclocked GTX 980 and am using it to drive three monitors, the largest of which (and the one I use for gaming) is at 1440p. The 980 has grunt, but it's still not able to push every game I want at 60FPS/1440p without settings compromises. That's partially because it only has 4GB VRAM, but mostly because the power for 1440p and especially 4K just isn't all the way there yet and many games are poorly optimized, especially past the 1080p mark. I wish more time was spent on making games run smoothly then on graphical gimmicks like better hair.
Let's be more realistic here. In any weather condition - this thing will live and die on reliability.

To get to the needed reliability(say 10x better than humans, i.e one accident per 2.5 million miles you need to drive at least 25 million miles, but considering the variety of situations this needs to be tested against , maybe more than a billion miles). To solve this google has built a simulator simulating all the roads of california(4 million miles) with a variety of situations and is lobbying to get this approved as a way for testing. And that's just for testing. You still have to solve problem!

Another data point - hyundai will be investing $70 billion in the next few years and wants to hire 7000 new engineers to work on their self driving car project.

This is not startup land.
What I don't understand about light computing is how to you use photons to do work? EM radiation is great at carrying information, but it seems that you need electricity to actually do the work of physically pushing gates and transistors around. How do you actually replace the work load with light?