skysight-inc/HackerNews huggingface dataset:
I think we're conflating topics. While the Firehose can be done with some combination of Pub-Sub/Topic/Sub-Topic Fanout, it should have little to do with the QPS for Timeline fanouts. I'd imagine the Firehose footprint is a really small part of their architecture or throughput pains. Timeline is a per-user join on multiple graphs and with paging.
'migrating to Go' implies that Go has the same advantages as Python, which is not really true.
rvm uses https, which is far better than plain http.
LOL at their research. Ksh is not even metioned (or only show in "plural"?).

Also, what's so bad about K$? damn you guys. my country uses two letters before the $ and it's fine.

What's next, going to push a new character set standard?
Figuratively, Twitter have switched from doing a design rooted in Databases 101 (OLTP) to a design more rooted in Databases 102 (OLAP).

That is, they moved processing from query time to write time. And that's a perfectly legitimate strategy; it's the basis of data warehousing.

OLTP is about write-time speed. It's great for stuff like credit card transactions, where you only really care about the tally at the end of the banking day. The dominant mode of operation is writing.

OLAP is about read-time speed. You do a bunch of upfront processing to turn your data into something that can be queried quickly and flexibly.

One thing that's problematic about the teaching of database technology is that the read/write balance isn't explicitly taught. Your head is filled with chapter and verse of relational algebra, which is essential for good OLTP design. But the problems of querying large datasets is usually left for a different course, if it's treated at all.
Basically Microsoft screwed up with the license for version 2, and it's legal to redistribute and use them, at least as an end user. They changed things for later versions. There's a good amount of coverage of the legal status on Wikipedia.

https://en.wikipedia.org/wiki/Core_fonts_for_the_Web
Wimbledon causes social media surge
This is great for us. We understand these tools and can use them. But most people don't. So if all geeks switch to the things on this list, we've left most of society just as susceptible as before.

Other, possibly better, solutions:

1) If you work for one of the companies listed as "proprietary", you can do the most. Stand up and say you care in company meetings. Tell managers and executives that it's worth finding better ways to secure, anonymize, or not collect information in the first place. Even if it comes at the cost of profitably or usability.

2) Authors of lists like these: Instead of saying all commercial software is lousy, compare them to each other! Make having secure, private software an actual selling point that people can understand.

3) Developers, designers: make beautiful, usable software that is secure and anonymous by default. Don't have privacy as your ONLY selling point. We can only win if we're private and amazing.
Right - and there are already much subtler devices like Memoto (http://memoto.com/) that have been created for the purpose of continuously logging photos. The obvious progression for Memoto, I'd think, is video.
Google will track your IP and probably store the searches you made alongside it. These can easily be linked back to your google accounts.

Duckduckgo claim not to log your searches.
The Internet Began in a Hut
The Sound of Your Voice Speaks Volumes
I see your perspective regarding quality content in the sense that inquisitive founders/geeks may not be "qualified" to weigh in with your standard of "quality".

However, for some "inquisitive minds", discussion (flawed as it may be) yields insight and can create enough interest to further investigate a topic via "quality" sources. Of course there is erred content on HN, but it still provides an introduction for someone to determine whether or not they want to dig deeper into a subject from more credible perspectives.
Surprised Arch Linux [1] isn't listed. It's probably one of the most secure distros by limiting the installed packages to a bare minimum. Combine that with App Armour (or SELinux designed by the NSA) with a firewall and basic network monitoring to protect against rootkits. Plus always-on VPN, dm-crypted harddrive, noscript etc.

NSA also released SEAndroid [2] which hardens Android significantly. It's included preinstalled w/ Samsung S4. Although still not very popular and I'm sure not heavily code-reviewed.

[1] https://wiki.archlinux.org/index.php/The_Arch_Way

[2] http://selinuxproject.org/page/SEAndroid
I disagree – customer support doesn't start with the person on the phone with the customer, it ends there. When a company designs a process where the person on the phone isn't empowered to effect change that person is guaranteed to be useless.

That this was happening to Google, and that it occurred exactly when we tried to _pay them money_ definitely makes the situation hilarious.
Probably their mobile clients. After I read the article perhaps Twitter felt that having control of the clients reduces variability in stress to their system. Like bad actors accessing the firehose suboptimally.
This comment just made me realize how right on the original vision of justin.tv was. It was probably 10 years ahead of its time. Maybe PG is a genius at finding founders who see the future.
This Digital Telescope Will Blow Your Mind
>> it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers

Why not break the load up among a farm of servers? 5 minutes to deliver a single message? It's too bad multicast can't be made to work for this use case.

At least analyze to see if there's a pattern of geographic concentration of her followers and optimize for where their datacenters are.

Use peer to peer, let the clients help distribute the messages.
Juxtaposition as an operator is already used by function invocation, e.g.

f "Hello"

If f were a string, how would I know I wanted to concat it instead of calling?
Yes, much more write-dominant. Apps which are built on top of the feeds can create issues, though. E.g., An obscure ticker pasted into a chat room with 500 people instantly starts monitoring in all their windows -- not just the static price (equiv to a RT), but the live feed. That would be as if A RT'd B and suddenly all of A's followers added B to their timeline. The degree to which that happens depends on how many features like that are integrated into the app.
Nice resource, thanks for posting. It would be really cool to have some kind of ratings and reviews for each service/app listed. Maybe an official review/rating and then user contributed.
Anyone able to decipher for the tldr / too long couldn't grok group? Have been following / been interested by the vostok story for a while now.
water goes to the lowest point since it weighs more than fuel. then if it's not drained properly it gets sucked into the engine at the start of the flight. would be odd to discover water contamination after a long-haul flight.

I kind of assumed the FOHE was something that used external air, it wasn't water in the oil or the fuel. you descend through humid and coolish air ie not summer heat, water condenses and freezes as it's sucked through something from the pressure differential, or from actual icing conditions. but I could be wrong.
Artur, you should only be seeing the list of notes that you created, would you mind sending me a screenshot so I can see what exactly you are referring to?
I really do not understand the thought process behind this page:

Why is chromium not listed as a free alternative to Chrome/IE/Safari?

What components of DDG are partly proprietary and which are not? (not a criticism of DDG just this page) What is a "free search engine" anyway?

Why are Firefox and Thunderbird listed alongside Iceweasel and Icedove?

How do you list OpenNIC if they have not adopted an official privacy/anonymization policy?

Speaking of official privacy policies; I see that you tried to load /analytics/piwik.js. Where is the privacy policy for prism-break?
AT&T Privacy Policy Change Risks and the De-Anonymization of Data
He's using a different meaning of the word counterpoint than the one you're thinking of.
Rust Promises Hydrogen Power Boost
even if you aren't there are social services

Where do you think those social services come from?
> None of the proprietary browsers will track you.

Can you elaborate a bit on this, how do you know they won't? My default assumption is that anything I can't see the source code of and compile myself is compromised.
In what way is compressing an array of integers "less general" than a general compression algorithm? Is that not exactly what a normal compression algorithm does, or at least can be construed as doing, with, say, 8-bit integers?
"Steve Jobs Passed Away", 4,271 points -> https://news.ycombinator.com/item?id=3078128

If you're interested in exploring, play around with the URL parameters for the API. Here's the one which gets you the right answer to this specific question with the minimum amount of work:

http://api.thriftdb.com/api.hnsearch.com/items/_search?q=&so...
Well a fencepost/off-by-one error is hardly unique to bad programmers. There are certainly instances of poor crypto implementation, but I don't think this particular example is worthy of the appellation of poor programming.
Wow, thanks for this post Amy.
Eventually? Although I've not tried them myself, a quick search of the internet pulls up numerous "spy" cameras built into all sorts of things, including eye glasses, that are virtually undetectable.

One example I found:

http://www.homesecuritystore.com/p-2319-dvr290-hidden-camera...
Bike should look like a bike. I wouldn't want a system where things could impersonate other things.
Don't intend to come off annoyed, although this isn't really worth anyone clicking on to read. In answer to all of the above, ask your recruiter.
I think you're implying that the FISA court's keeping the basis for its legal authority secret is analogous to any independent federal agency making statutes in accordance with its congressionally granted authority. I think the critical difference is that in the run of the mill federal agency case, the source of authority to make rules is clear and a citizen's legal recourse if he or she disagrees is also clear. In this case, one can't sue the federal government for violating the constitution because one doesn't know what's its stated basis for constitutionality is.

It's hard not to conclude that the FISA legal reasoning is secret because it sucks -- it's weak and at best unorthodox. In other words, most people qualified to have an opinion would think its not sound -- i.e., not really legal -- i.e., it's an unlawful power grab. That's one way a democracy might turn into to something entirely different. Really!

Is there the rule of law in the U.S. or not? Are FISA judges subject to the rule of law or do they perhaps garner their authority from a divine source like the Kings of England? Or maybe the chief justice's left testicle? Or is it righty? We can't really tell. And we can't really tell whether we live in a lawful state or not -- because, in this case, it's a secret. Severe coughing fit followed by a mournful sigh.

It seems like if one's answer to the question "what is your legal authority?" is "it's a secret" then you really don't have any legal authority and are deriving your authority from some other source, whether you'd like to admit it or not.
I'm here for the startups and investment money. I don't want to consult anymore.
I remember asking a question about scaling on SO and getting a response about how my data was small, because it could fit on an SSD (for that one small component)

I miss not having real scaling issues sometimes. They make my head hurt.
Seems like the lack of links in that column makes it pretty obvious
I typically use Clojure serverside and JS or ClojureScript client-side with a PostgreSQL database backing persistence.
This reminds me of the IBM RANDU PRNG on which I gave an undergrad presentation (w/ visualization). RANDU was legendarily bad; all you had to do was plot triples in a 3-D volume to see that all of the generated points would fall within one of 15 (Edit: 15, not 11) planes.
Should have called it ZIPTasticUSAPI..

{"error":"Zip Code not found!"}

Also, whats wrong with geonames or google geocoding apis? Not slim enough? :D
Don't worry the housing prices will drop off a cliff after the up and coming massive earthquake finally hits San Francisco. It's guaranteed to happen, it's just a matter of when. You'll have a whole new set of problems but a lot of people are going to haul ass out of there and rent will drop. And the subsequent effects will fix a lot of the real estate problems. First, whatever the earthquake doesn't destroy, the fires or building demolitions will. In the ashes or rubble where the old low-rise "historic" buildings stood, new buildings will finally be built and you can bet your money that they're going to be high density. NIMBY (not in my back yard) residents will have so many other matters to attend to that they won't be so strict about keeping San Francisco looking the same as they remember it as kids so progress will be easier.

With destruction and death, comes renewal and new life. Fitting for the startup capital of the world actually.

I wish everyone the best of luck surviving and may the death toll be as minimal as possible.
Is this a continuous 47 square miles, though? San Francisco is always a little difficult to compare to other cities that are much larger metro regions. If you were to replace the lower density parts of SF with the dense urban areas in Oakland and Berkeley, you might end up with something much higher than what you sectioned off from Chicago - and as a region, that comparison might be more accurate.
> Several of these suggestions seem somewhat disingenuous - e.g. many of them to be about free software more than actual concerns about tracking [..] None of the proprietary browsers will track you.

No, these issues are very much related. It is the very nature of proprietary software that you cannot inspect and modify it, so you cannot know if it will track you or not, and cannot fix things if you are.

(Inspecting outgoing traffic is helpful, but unless you monitor all activity all the time, and make the effort to actually understand every single bit that is transmitted, you can't be certain.)

Furthermore, some of these browsers explicitly do track you. For example, Internet Explorer, Chrome and Safari provide ways to sync your bookmarks, all of which track you - and some of them encourage you to do this, for example if you do not log in to Chrome it says "you're missing out". (Firefox also has a sync service, but it is encrypted on the client, so the server cannot read the information, and you can't be tracked.)
Off-by-one errors are very common (and they have a long history). My best guess is "an honest mistake."

As for the "endemic blind spots," any crypto software needs professional reviews and a good deal of time before it can be used in critical situations.
>>Finally, many of these risks are reduced by actually being a part of the community. Don't want a factory built 0.5 miles from your home? Go to the planning meetings, talk to your city planning staff/council members. Join the HOA board. Being a homeowner means that you have an incentive to invest in your community and make it the community you want it to be. If you just disconnect and let others make decisions for you, yes you can be screwed. However, if you participate, get to know your neighbors, and become a part of the community, you can make it everything you want it to be, and more.

I think you're grasping at straws here and it shows. All of the things you listed are incredible time-sucks. If you have ever been to an HOA meeting, you will know exactly what I'm talking about.

And did you just suggest going to city planning meetings to prevent the building of a nearby factory? I'm sorry, but do you even understand how these things work?
You can always run Wireshark to see if/when they phone home.
> I'm pretty militant anti-home ownership. This is partly to combat the omnipresent, uniquely American, societally-negative idea that you really ought to dream of owning a home some day.

But, what if I really want to own a home? I'm a homebody. I want a place to call my own, and I want to make it comfortable for me. I enjoy entertaining guests, cooking for them, and making everybody comfortable -- when I move back to Boston, I'll invite you over. :)

> The only reason home ownership appears to be a better idea, is because people ALWAYS fail to look at it in the risk-adjusted sense.

It's not strictly an investment. What if I derive pleasure and comfort from living in a place that I can customize as I see fit? Every time I rent an apartment, there are things that I hate and drive me crazy: carpet, electric range, no outdoor space, awkward floor plan, etc.

> If someone gave advice to an unsophisticated median american to go out and buy tons of stock options, or short sell stock on margin, they'd be rightly chastized. It's unethical.

Agreed. I'm a fellow MIT alum who graduated in 2005. I know what I'm doing -- and what I want to do is own a home... and in the Boston area, to boot. I wouldn't advocate pushing the "dream" on other people, but I know what I'm doing; why chastise me?

You say that your "militant anti-home ownership" view is "partly to combat the omnipresent, uniquely American, societally-negative idea that you really ought to dream of owning a home some day". Do you find the fact that I want to own one repulsive?

I'll be back in Boston soon. I'll take you out for a beer (or tea, or any alternative beverage, if you don't drink), if you want to talk.
> It's our indifference that gets us into a lot of trouble. Time and time again.

Actually, I think the inverse is true. If apathy wasn't the defining human characteristic, we'd have had WW3 through WW9 by now, and people would be killing each other in the streets with hammers.

"Fuck it" is a survival trait.
With normal software you can load up on the unit and integration tests to make yourself more confident with your software. When the concern is with the integrity of a cryptographic system, things are not quite so simple. You can write tests, sure, but your overall confidence afterwards is going to be much different.
Sociologically: there is a surprisingly large contingent of people who believe that if a company makes a claim, it's the God's honest Truth. The OP may not necessarily fall into this camp.

Technically: if the browsers were somehow phoning home, even if the data were highly fuzzed, I'm sure there would be guys like tpatcek who would manage to detail, if not the content of the tracking, at least the amount of data sent and the targets. I don't recall there being such a scandal in recent memory.
Yes. A pro friend of mine is fond of saying: budget 10x to design, 1x to implementation, and 10x to verification.
I was thinking the same myself - couldn't it work just as well on a random byte stream by just passing in every 4 bytes as a 32-bit integer?
I actually agree with what you're saying, I was more responding to people I know (and/or people that I see post here from time to time) that put SF on a pedestal, and seem to have convinced themselves that the downsides to SF are actually upsides.

I always ride the public transport when I go to SF, it's a great convenience and much less of a pain than driving up there and around there. I wasn't saying that public transport in SF is a bad thing (quite the opposite), but rather that it's often the only choice.

Like, even when it would be way, way more convenient for any number of reasons to just drive to wherever it is that you're going, various factors (parking, traffic, whatever) make it just too much of a pain. Even just owning a car when living in the city can be too expensive and more trouble than it's worth. I do see this as a downside to living up there (or just visiting), even though I would still ride the public transport most of the time anyway.

I know lots of people that live in SF and don't have a car (or in some cases even a valid driver's license), and they will swear up and down that having a car is totally pointless and why would anybody ever do that and what the hell is wrong with you anyway, do you enjoy filling up your tank with evil every week? :) But they spend 100% of their free time wandering around SF. It's not as feasible for them to go on day trips or weekend trips up and down the coast, or get out in the wilderness (at least, out beyond the reach of the bus system), so they just don't. They just hang around SF all the time, which is a rather wonderful place to be stuck in, but they are stuck there nonetheless. I love going on long drives and exploring all the remote corners of the Bay Area and beyond, and I couldn't imagine life any other way.
Ask HN: Where to find a mentor in my city (or any)?
I live in Los Angeles and would love to meet someone who I could get advice/motivation from in moving forward career wise. I've been in this business two and a half years but seem to be the only one at my company that gets excited like a kid at learning new programming technologies/concepts. The meetups in my neighborhood that I've attended tend to attract very new programmers so not a lot of learning there (although i really enjoy helping them out). Other than "co-workers" and "meetups" I'm not sure of the best place to search. Thanks!
I can only comment in the context of playing videogames with people in South America, but you may find the information helpful.

I've been playing a lot of Counterstrike lately, and apparently the game is wildly popular in Brazil. The average player from Brazil appears to have 200-300ms of latency, and when playing with them, the game suffers intermittent lag spikes.

This doesn't really give you an idea of the availability, but it should give you an idea of what your connectivity is going to be like when you are trying to interact with services in the U.S.
See: http://lemire.me/blog/archives/2012/02/08/effective-compress...

You can use it for text but it won't work as well. It's designed for numbers that are created by some kind of real process.

It compresses by looking for the largest and smallest number and only storing the difference from the smallest number, and only as many bits as needed for the largest. (Good for situations where most of your data lies inside a range.)

It stores the difference between successive numbers. (Good for measuring a process that varies slowly.)

Also see: http://en.wikipedia.org/wiki/Golomb_coding which works on the principle that small numbers show up more than large.

None of that makes sense for text which doesn't have those patterns.
I love the picture. What a political agitator. He was a good little left wing agitator and was fatally shocked to learn that when you're not serving the agenda you can get bit.
Funny, I always considered my emotions to be a feature, not a bug. Life would be pretty boring if nothing ever gave me joy, sadness, relief, anger, gratitude, anxiety, laughter, sorrow, or peace.
Software Recognition Technology Is Amazing, but Not Amazing Enough
Although proprietary software may be easier for a government to compel to be modified to add tracking, it still runs the risk of being noticed in most reasonable cases, and there is in fact no evidence that any Western government is doing any such thing. It does increase the chance that you are being tracked due to incompetence, but I don't think this is particularly likely for such well-known software.

True, there is a difference with the bookmark sync - I do not think it is valid to discount a browser entirely based on this.
If you're going to continue using Google Mail, it's a dumb idea to deliberately switch away from Chrome. The connection between Gmail and Chrome is among the more carefully guarded TLS connections on the Internet.