skysight-inc/HackerNews huggingface dataset:
Ask HN: How to handle 50GB of transaction data each day? (200GB during peak)
Recently I switched jobs and joined some old employers and friends on a new endeavour, an advertisement platform. So far things have been very good, so good in fact that we're quickly growing out of our current infrastructure. Last christmas everything went crazy and the, then only developer, rewrote a bunch of things to make it scale better.

Currently we write away every single transaction to txt files and once every X minutes gather those and start inserting them into 2 Mysql shards. After that we run a bunch of group by queries to generate cross table reports. Currently our CMS is very limited by this, you can only view data relations in the predefined tables and we would like to have (a lot) more flexibility here.

Perhaps the biggest issue of all is that both I and the other developer do not have experience with handeling this much data. We both have worked on systems which handle a fair bit of data, but not nearly as much as this. We've been looking at MapReduce for MongoDB but already run into problems when we try to insert a weeks worth of data, which is about 350gb, because we only get so many IOPS at AWS. If we want more IOPS, it gets really expensive really fast. We're also not entirely sure yet if MongoDB and MapReduce is the way to go.

What I'm looking for is a few pointers to help us get headed in the right direction. Which databases should we be looking at? Should we be looking in an entirely different direction? Are there some (recent?) blog posts about this which can help us out?
Signing code can only do so much.

What Apple possesses is the somewhat unique ability to design a system that is actually secure by burning the key into the secure enclave and not allowing it to be updated. The only way someone would be able to get to it then is by attacking the physical hardware itself (which I'm sure an NSA-level attacker could do), but it would render this entire thing moot, as even Apple wouldn't be able to unlock the phone if it wanted.

I say unique because they can bake security into the actual hardware design, and tightly control how the entire thing works, which android & windows simply can't do. In order to trust your OS (and in turn, your signed software), you have to trust your hardware first. The security of the entire system falls apart if you can't trust your hardware.
Delphi was the most productive dev environment I ever worked in. It was sad to watch it slowly dying all these years...
What Is Opt in Page? How to Create Opt in Page in Most Effective Ways
Some people and sometimes I catch myself doing it send a sentence in multiple messages since the message doesn't cost anything compared to a sms where you would definitely try to fit as many characters as possible into a single message.

However I'm not sure how common this is and if it could be one of the reasons for the high amount of messages per user.
The founders aim to keep it free forever, should they want to earn money they'll add some non critical features so people who want to pay will keep the app running
15B Telegrams Delivered Daily
> on the (unmentioned) condition that they make those sites essentially unusable over Tor

Most people don't see it as a condition like this. It's more like your website will be served through the CF network instead of your normal server. This has a lot of effects: increased security (DDOS and others), increased speed (distributed CDN, caching all your static content), (non js) analytics, free SSL (their certs, this was before letsencrypt) etc. A part of that incresed security is blocking traffic from "suspicious" sources, like TOR. Which your average small business with a website has no problems at all with.
Apart from the obvious problems (more pixels = more GPU/CPU power needed), the more dense the screen, the less effective is the backlight: With a denser matrix, less light passes through the tinier cells, and you have to crank up raw backlight strength to compensate. And that is absolutely murderous for an outdoor phone that has to have a backlight as bright as possible, so you can read it in the sun.
The mere existence of getaddrinfo is a symptom of the disease. Just look at it. It lacks a clear interface. It's designed for calling code which is like, "Gee, I don't know exactly what I've got. Maybe it's a local IPv6 address, maybe it's a remote hostname. But I might want to bind it. Or maybe connect to it. I'll get back a list of these things which might be useful. Most likely I'll choose the first one or maybe write an ad-hoc filter to pick the one I actually want." Systems programming doesn't have to be this crappy.

I get it, the idea is to replace this core low-level library with something you could maybe run C programs on top of. But damn, that is a crappy interface (which is specified by both an ISO standard and an IETF RFC so I must be an idiot, right?).
I think Apple in general and Tim Cook in specific are highly moral people: I think they are truly looking at security as a way to make the world a better place; in discussions I have had with employees at Apple, they truly do attempt to build systems where even they don't have access to your information... only at the same time, but somehow from the other side of their mouth, when questioned about their ability to do things that are evil when they do have that ability, they just say "well, we'd never do that", and refuse to discuss scenarios where "we" is difficult to define (as it only takes a finite number of disgruntled employees to decide to do something bad) or they are forced by external parties (such as the United States government). I'm honestly kind of glad to see that latter scenario actually play out, and particularly to see it play out so publicly, so we can see how they react as "well, we'd never do that" even has the possibility of turning into "fuck, we were forced to do that".
In Gaza, Bicycles Are a Battleground for Women Who Dare to Ride
He's not in charge of, but still holds at least $12bn of MS stock, on which a lot of his wealth is leveraged and predicated, which would certainly give him a vested interest in defending microsoft's interests, which are the government's interests.

You may well be right however that the circle he runs in has influenced his stance on this too, however he has historically taken a pro-government anti-knowledge stance - see http://www.rollingstone.com/culture/news/bill-gates-the-roll... for instance in which he trots out the "Snowden is a traitor who needs to come home for a fair trial" rhetoric.
Type 1 diabetic here. I remember having the flu before I got the diabetes, and later on there's been some research that if you have the genes and get the flu when you're a child, there's a big chance of losing your insulin production. Would love to have some sources for this though, because it's been a long time since I heard about this.

I was lucky though, born in Finland with the best type 1 diabetes study and completely free insulins and medical care.
"Since July 2007, PayPal has operated across the European Union as a Luxembourg-based bank." -- wikipedia
If this were a normal door lock that someone possessed the master key to, the FBI would have just requested access to the master key temporarily. The FBI is not operating outside the law here: they have a court order. While in some ways it would be more horrific, I'd almost rather see them ask the more analogous question "we would like a copy of your master key" to see how Apple responds, as they'd no longer be able to pretend like it would take them a lot of time to build what the FBI wants: the FBI would just be asking for literally 512 bytes of data that Apple has that stands between them and their goal.
A hands-on guide to learning User Experience Design
Yeah, sometimes it is used more in an "IRC" way than in a "SMS on steroids" way.

I think that the web and desktop clients just push this probability. Having a computer keyboard instead of a phone to write triggers a different "state of mind" in the user, I think. At least it's like that for me.
How do you think of its business model? Keep it free to get a big user base then sell it to huge enterprise? Like Whatsapp and Facebook?
Goldman Sachs, HSBC Back Cameron Push to Keep Britain in the EU
>In particular, freezing the syntax for as long as the proposed timeframe for Dotty to be delivered would kill any language.

Because Scala doesn't have enough syntax already?
It's the closest we have to a user friendly messenger that is not entirely closed. I call that a step in the right direction. No reason to throw out the baby with rhe bathwater.
I don't use Telegram because of secure messaging, I use it because is clearly superior to WhatsApp and Facebook Messenger, and so many people.
I actually agree with you. Apple should just make it it so that Apple CAN NOT do anything.

> If Apple had the phone's PIN stored in their database, do you think the government should be able to compel them to hand it over? And why is that any different?

Well, I think, they should not, but I don't claim its different.

Apple wants to have it both ways, build it so that they have full control and they can be compelled by nobody to give up that control. I would rather they build a system, where nobody is in control at all.
It's not just what FBI orders, it's the unprecedented legal basis for the order, All Writs which simply states that they can order whatever they want. ("Why?" "Just because!")

Which is very bad if it is accepted this time.
You're more then welcome to submit one yourself.
I'm using it because it's really cross-platform, works reliably on Linux, Windows Phone, web (these three are the most problematic when it comes to messengers). Also, it's extremely lightweight and fast.
Oh, I see now. Then it is multiplied by the device number as well. But now I understand it and it seems impressive anyway.
When you write an application, it's fine to pull whatever dependency you feel like.

When you write a library, you need to think very carefully about the deps you bring because you'll be imposing it to all your users.
Wow. Just wow. I keep thinking there is some huge cavet missing from the story, but if there isnt, that is massivly disconnected from reality.

Call and message info, okay maybe, but full access? There is no way you can say thats fine.

I have said it in an earlier comment and ill say it again, i think they are inly doing this, on this case, to make a precedent, and only on this case, as few people will want it to be on record, that they are 'siding' with terrorists.

Imho, its bullshit, all the way through.
It says it's pushes which would be past fan-out for groups.
I don't know - perhaps loose certificates have a strong correlation with the amount of rouge ;)
In an interview with Rolling Stone's Jeff Goodell, the Microsoft co-founder and richest man in the world did not mince words when asked if he considered whether Snowden is a hero or a traitor.

"I think he broke the law, so I certainly wouldn't characterize him as a hero," Gates said. "If he wanted to raise the issues and stay in the country and engage in civil disobedience or something of that kind, or if he had been careful in terms of what he had released, then it would fit more of the model of 'OK, I'm really trying to improve things.' You won't find much admiration from me."

Gates said that there "has to be a debate" about government snooping, but indicated that some aspects of government surveillance are best left a secret.

Microsoft has seen quite a bit of controversy regarding its alleged cooperation with the NSA. Last July, the Guardian reported that Microsoft had aided both the NSA and FBI in accessing user data, including providing video and audio conversations from Skype, Microsoft's video chat service. A Der Spiegel report in December also found that the NSA would use fake Windows error messages to spy on people.

Ref: http://www.huffingtonpost.com/entry/bill-gates-snowden_n_496...

http://www.rollingstone.com/culture/news/bill-gates-the-roll...
Yes, this sounds more like a optics company.
On a side note: The FBI didn't/couldn't even properly investigate 9/11 so why do they still dare to use this wording? ("to help them open a single phone that may hold crucial evidence to help explain the most deadly terrorist attack on U.S. soil since Sept. 11, 2001") What came out of PENTTBOM? One single court case against Moussaoui, who was not directly involved in 9/11 at all, but that's it. Hundreds of arrests which led to nothing but illegitime long-term hold ups under military law, but no due law.

If Apple can help them to extract data from those phones, fine. But Apple apparently built secure phones without bypass, so they are out of luck and it makes no sense to come up with fantasy warrants without any technical solution.
What happened to "just one phone"
Instant answers are open sourced. :)

I'm not affiliated with DuckDuckGo, but I did develop a couple of them, like WordPress cheat sheet[0] and nmap cheat sheet[1]

[0] https://duckduckgo.com/?q=wordpress+cheat+sheet&t=canonical&...

[1] https://duckduckgo.com/?q=nmap+cheat+sheet&t=canonical&ia=ch...

When an instant answer is developed by some member of the community, you'll notice a small (i) icon on the right side of the instant answer. Once you click on it, you'll see the details about who developed it.
And I have to mention David Intersimone (a.k.a DavidI) [1] - Delphi evangelist for last 30 years. Respect!

[1] https://twitter.com/davidi99
Play nice - How the internet is trying to design out toxic behaviour
Hi,

Till now I haven't heard anything like this. It will save lots of time as we don't have check for errors and also writing speed will increase. This would be a great innovation.
It's one of the few - if not the only - native, reliable and fast among all major platforms. It just works.

Sure, there's the whole security debacle behind, but all in all, as an instant messenger, it works wonderfully.
I wonder how much of these are from Iran? I know that they are insanely popular in Iran. If I wanna guess, I would say more than 10% if not 20%!
Typesafe is now 'Lightbend' – claims it still loves Scala, drops Java tool
Why SAP Wants to Train and Hire Nearly 700 Adults with Autism
Great, patent attorney with uMich undergrad in computer engineering, well connected in Santa Barbara. Available after 10am. Email @ Wayclever@gmail and let's discuss.

Kenneth Stein
I think there are a lot of conflating issues in this discussion.

Firmware signing and how updates are delivered are one thing. I would argue that having only one possible adversary is preferable to everyone being able to create firmware that runs on your device. If there's a practical and secure approach that would allow users to install only firmware updates they approve of, I'd be all for that[1]. In the end - please correct me if I'm wrong - this would require a user-generated key or passphrase of some sort, and then we're back at a brute-force problem and the question of how secure is that passphrase and how are rate-limits enforced.

The iPhone's disc encryption, however, does not rely on this so-called master key. That's why I think calling this a backdoor isn't a fair assessment. It's entirely reliant on the complexity of your passphrase. The iPhone's security architecture, including the firmware signing and in newer versions the secure enclave, make attacks against this significantly harder (or next to impossible, if the secure enclave firmware is actually read-only ... something that definitely needs to be clarified). Compare this to your typical desktop full-disk encryption, where you usually have no countermeasures whatsoever against this kind of thing.

[1]: Speaking as a developer. I'm not qualified to answer this for sure, but my gut feeling is that such a feature in the hands of typical end-users might actually be a bad thing for security.
You're really missing the point. Yes the only relevant code that is going to be run on that device will code signed by Apple. I don't think FBI will have direct access to this "key".

The FBI wants Apple to create "malicious" code/update/software version that would allow for multiple decryption attempts among other things. Apple CAN comply with these requests, probably easily. However, by doing so they will destroy trust in Apple signed code and set a precedent.

It doesn't matter WHO has the key because Apple will be acting in proxy.
> Obviously "unfettered access to the device" is really useful to steal user keys while the device is unlocked, but it's not so helpful after the fact.

Exactly: the ability for Apple to send a specific user a different firmware update than they send everyone else is extremely brutal and there is absolutely no way the user (no matter how intelligent) could even tell that they were being targeted as the only person who has even remotely powerful access to the firmware being loaded is Apple themselves.

> This late in the game, how can that possibly give Apple user filesystem keys? Those require the PIN or password.

You just brute force this. On the iPhone 4 it took minutes to brute force a 4- digit PIN code, and clearly it wouldn't be a challenge to brute force a 6- digit PIN code (this is still less than a day). If the user has a password, it might take a while (depending on how good it is), but it is still a guaranteed attack. You can quibble with me on the definition of "unfettered", but I maintain that "will take (maybe) some time but almost no effort to get a 100% success rate, and which will complete almost certainly before the statute of limitations expires on the crime" is not usefully "fettered".
Hi,

De Caprio will win this oscar award as he deserve it the most. He is an excellent actor.
Agreed, I'm extremely glad this is being done in such a public fashion. It's important to have transparency about these issues.
Thanks. This is exactly what I needed.
Indian Juice Company Raw Pressery Soaks Up $4.5M in New Funding Led by Sequoia
> That's what you're buying with all this "Entrepreneuring". The ability to replace your old income with a new one that doesn't necessarily require you to work full time for it.

Time is the limited resource, not money. You can always get more money, but you can never get more time.

Many (most?) people forget that.
I like your comment. But before you can load the buffering icon, you have run out of mobile data.

I think most users do not own a 4K monitor at home or in their companies. I am not sure if 4K recording/ decoding is useful in this period.
Honestly no. A lot of the complex Scala you see is because Scala doesn't have syntax for something and so you have to put it together out of other things. Type lambdas and Miles Sabin's unboxed union types trick are prime examples of cases where adding some syntax would make a lot of real-world Scala a lot more readable. Likewise ADT case classes where you don't really want the class to be its own type - a little syntax would save a lot of repetitive "smart constructors".

Also there are cases where it's not more syntax that's needed, but different syntax. E.g. the way irrefutable unpacks in for/yields desugar to a (with)filter call is just dumb.
Drax: pre-tax profits fall 64% to £59m in 2015
It's also missing the important point that rewrites don't happen just by talking about them and making grandiose statements.

I counted about two people in the entire thread that have actually written code to support this much desired rewrite. The rest are chitchatting.
I think Telegram does really well on both functionality and UX, but is really brought down by their closed-source policy. I'm not sure what the motivations behind that are.
I don't think you are responding to my post? At least I can't see which part of my post you comment, that is, what I am missing.

The issue here is if All Writs is a good legal basis for the precedent of "change your product."
Would be nice to have a desktop client and pidgin integration, both of which Telegram has (although the latter is an unofficial plugin).
We need to handle data in a similar level as you mention and also use plain text files as only reliable medium to store data. A recent blog: http://nunobrito1981.blogspot.de/2016/02/how-big-was-triplec...

My advice is to step away from AWS (because of price as you noted). Bare metal servers are the best startup friend for large data in regards to performance and storage. This way you avoid virtualized CPU or distributed file systems that are more of a bottleneck than advantage.

Look for GorillaServers at https://www.gorillaservers.com/

You get 40Tb storage with 8~16 cores per server, along with 30Tb of bandwidth included for roughly 200 USD/month.

This should remove the IOPS limitation and provide enough working space to transform the data. Hope this helps.
The problem is threefold.

Firstly, as the letter notes, the majority of people using TypeSafe/LightBend's services are in fact "traditional" Java shops. This suggests that - in the enterprise at least - there's limited Scala penetration.

Secondly, the move to a more generic name (LightBend) is one more step away from their core competency. The name TypeSafe was chosen to highlight a particular aspect of the platform they felt was important. Although they likely still do, their customers don't appear to appreciate it (or at least in the same way it was meant, as an advert for Scala).

Finally the bifurcation of technical resources between ongoing Scala development and Dotty as a research project are likely to come up short in both regards. We are on the cusp of Java 9 being released within the year and Scala's big backwards incompatible step forwards with Java 8 still hasn't happened.

At the least you'd have to be a gambling person with a penchant for long odds to continue with Scala at this point.
Gold Hedging Returns as Harmony, Acacia Lock in Profit Margins
I think if you use eval, all bets are off anyway.

But I think that in ES6 without eval you can't refer to the current scope in a dynamic way; which means that the engine can resolve bindings at compile time without worrying about people fiddling with scopes behind the scenes, which should simplify optimisation a lot.
Shouldn't it be called patent extension racket?
Robinhood's disruption in online stock trading
There's a fundamental difference between physical property and intellectual "property": an owner of the former can conceivably hire guards to protect or restrict access to that property, since physical objects are generally localized phenomena.

But there is no conceivable way to enforce things like copyright or patents without something at least as powerful as a government, since they are supposed to be binding on every single person, everywhere, all the time.

IMO the concept of "intellectual property" is at best orthogonal to that of a "free market".
How this city used Social Media during Flood
The first case is clear. The works are created by you, and the involvement of a tool of your own making has no effect on copyright. If you make a line-art generator program and print out 10 million sheets of paper filled with random scribbles, they are all copyrighted works by you, just not very interesting ones.

The second case depends entirely on how you license the generator program. If you sell the generator and don't make an explicit claim over produced works, they are the copyright of the person who clicked the button. The act of creation doesn't have to be more complicated than that (after all taking a photo also involves just clicking a button).
Explosion from hydrogen? Not a problem at all - hydrogen is MUCH MUCH safer than gasoline.

The main reason for that is that it doesn't stick around - it's lighter than air, so all the fuel rises instantly, and so does the bulk of the fire, which stays with the fuel.

It makes hydrogen fires rather safe as fires go - they are out within seconds.

For example the hydrogen fire in the Hindenburg was out within about 10 to 15 seconds. The casualties came from burning fabric and other things (like crashing).

Hydrogen is not a good fuel for other reasons (see my other post), but not because of safety.