skysight-inc/HackerNews huggingface dataset:
Kafka is certainly better than RabbitMQ in some respects. (In others, it's disappointing: It's practically useless if you're not running on the JVM, as clients for languages such as Go, Ruby and Node aren't up to date with the "smart" Java client. It's also clearly more low-level and designed for large installations, and less friendly to small ones.)

The problem with storing indexing state outside the database — using a queue, for example — is transactionally protecting the gap between the database and the queue. Bottled Water is cool in that it can actually bridge that gap safely, as I understand it, since PostgreSQL will keep the decoded stream until you've been able to propagate it to Kafka. On the other hand, if you have the stream, do you need Kafka? Can't you just push it directly to ElasticSearch?

For us, this issue — this and standardizing on an elegant cross-language RPC — is probably the main architectural challenge we're facing right now in our microservice development. We have tons of microservices with private data stores that need good search and also internal synchronization between microservices, which is coincidentally the exact same problem space: You have service A with its complex data model, and then a service B that wants to subscribe to updates so that it can correlate its data with that of A. It's a complicated problem that requires a simple solution.

I am not sure if I want to hit my production database with huge read loads on every reindex of a secondary data store.

Hopefully a full reindex shouldn't happen that often, though. And a full reindex would require a full scan of your production database (not the transaction log) anyway, since you don't want to keep the entire change log around forever (and can't, since the log only starts at the point when you started running this system).
I had the opposite experience. DotPeek created much better code. using statements were decompiled as usings, JustDecompile produced try/finally blocks.
A "library" that wants to have a process of some sort continuously running encounters the problem that the more cross-language generic you try to make the library, the less possible it is to work out a threading model that can be used by everyone, and the less language-generic you make it, the smaller your target audience is, which has non-linear affects on the usage and development resources it can attract. Languages that include a "blessed" runtime model like Node or Go can easily ship such "daemonized libraries", but then they are generally impossible to bind to for anybody else. Using POSIX threads, by contrast, will lock a lot of other languages out that have very sophisticated runtimes, be very difficult to use in others, and, even in C, when you get right down to it, there's no guarantee that they'll play nicely with the rest of the C program.

Distributing a service invokes the sort of lowest-common-denominator solution to this problem, which is "OS process". Everyone can talk to an OS process.

(This is an explanation, not advocacy or celebration.)
Tropology performance – PostgreSQL vs. Neo4j
Open Frameworks Face Swap App: Achieve the Alias of Your Dreams
Fuck Yes.
How to Build an 8-Figure Business Using Podcasts
The article underscore the point that the complexities of our planet are intricate, and particular.

Nowadays people seem to think that life must be common throughout the universe, given the number of stars and planets, and etc. It seems to me that the opposite could be the case, that life is extraordinarily rare.

As far as I know, we don't really know the likelihood of life emerging in our own planetary conditions: given a billion earth like planets with roughly equivalent atomic makeup, distance from sun, etc (assuming these are common, which fact we don't know either), how often will life emerge? We can guess and say "all the time!", but it could be an infinitesimal small amount.

Even moreso for different conditions than earth. Commonly people say "given a liquid like water, and certain other factors", but we don't know if life will emerge in any other conditions except our own. We have no proof whatsoever.

We also don't know how likely it is that life reaches our own point, of being capable of pondering life around other stars, or capable of imagining travel to them. There may be many, many extremely low probability events that must occur, in the billions of years between life's genesis and our own existence.

In fact, there's a good likelihood that our own dreams of interstellar travel will end in our extinction prior to our ever making the attempt. If that happens, it will prove my point.

To me this general optimism towards life on other planets is vaguely anthrocentric, like the belief that god is an old man, or that the universe revolves around the sun.

As such it results in a tendency to underestimate our own fragility, and to pay appropriate attention to the challenges of maintaining our own continued existence on earth.
Awesome :]
Not to mention, it's beautiful. Not everything needs to have a practical use.
It's also what the IETF says. Don't belittle those who are trying to fix the problems you're frustrated with but can't be bothered to understand.
I'm wondering, is it possible to circumvent the surveillance, still enjoy a good internet experience, while not devoting too much resources?

I suppose it starts with a VPN service. I use VPNs occasionally, but never in a systematic way. How convenient are they in term of reliability and performance? then, what about gmail? are there other email providers that offer better privacy guarantees and offer a comparable service?
You could also associate a key (as in, letters, or even symbols for the programmer crowd) with each note,

would be a nice touch-type learning software
There are already existing providers for this kind of service. Ex:http://kapowsoftware.com/

I am not sure, what is different in your approach?
I remember the conlanging talk from Sai at one of the CCCs. Really awesome stuff.
https://www.youtube.com/watch?v=PEKcsto7Mjk
It's more a case of dealing with the legacy problem of node.js never having upgraded V8 as often as they should have.
A lawyer would probably be a better person to answer this question, but if I had to venture a guess:

1. Identifying products that the source company is the sole distribute of

2. Logs that are able to identify the crawler and what it accessed.
A Gallery of Interesting IPython Notebooks
> 3... Or this mansion in Alamo Square for $6,400,000.

Grew up in a 7+7 city house in Zimbabwe (5ha property). When we left it was sold for less than US$100000 (current day value). Given that perspective I'd have to venture a guess to the true meaning of the article: property values vary wildly depending on location to extreme scales.

9+9 is a mansion by modern standards? Bloody hell.
It's always instructive to see how people failed and how they eventually realized why.

In this case I would have told them up front this is too much to do to start with. To get big wins eventually you have to start with small ones.
>I've been looking to purchase some land just outside the Bay Area to build a small cabin...

You can do this in the Bay Area itself, though in general the closer you want to be to a supermarket, bank, etc. the more you'll pay. If you're west of Palo Alto, Los Altos, Mountain View, Cupertino and cross the 280, you encounter rural areas very quickly. And if you head far enough into the hills, prices begin to fall rapidly relative to flatland prices.

Near Boulder Creek, for instance, you can get 2 acres for $52K that's three miles from a shopping area: https://www.redfin.com/CA/San-Lorenzo-Valley/0-HARMON-GULCH-...

That's 23 miles from Cupertino via route 17 and smaller roads. Of course it helps if you enjoy living in the middle of the woods!

I've posted about this concept on HN before:
https://news.ycombinator.com/item?id=9437492
Please excuse the self promotion but if you like this topic I wrote a related post "Insidious Bugs or: How I Stopped Worrying and Learned to Love Exceptions"
http://verysimple.com/2009/04/03/insidious-bugs/

It does seem like a lot of people are afraid to let a user see an error, but if you're not extremely careful then attempts to shield the user can mask bugs.
Sort of right.

I am not looking for a solution that tracks my whole life. Instead I would like to focus on health with Segment.com-like integrations into other services (https://segment.com/integrations) to always know what's going on with me.
Major fan here.

I can't believe this is not getting enough vote to rise to the top. This guy is unstoppable with amount of tips and the effort it goes into illustrating them with animated gifs.

*Must-follow for frontend developers
Tales from Direct Sales Email Outreach
Lisp has had this feature for... never mind.
This is a bad idea. Concerns about Docker's security (or lack thereof) aside, containers provide "good enough" separation of concerns to apps. They are not security hardened and designed to be exposed to vulnerabilities, particularly when by design a honeypot is to tempt attacks (automated or otherwise).

A virtual machine (or better) is better suited to this task.
I don't think that it has to be hippie at all. There would be some shared areas, but people would live at their own homes. The company and the farm would be shared, but that's the idea.

Guess that the whole twist would be to not only depend on agriculture for providing income and food.

Also, I wouldn't make it about living isolated from society, without electricity, cars or mobile phones. It would be a much more relaxed, simple life, but every individual or family would put their own limits, respecting their comfort zones.

And yes, there would be life in community, maybe sharing some meals or having kids play together, but as I said at the beginning, families would have their own houses or appartments, so they could also here decide how far they want to go.

The cool thing would be having lots of like-minded people (= engineers) living and working close together, which could develop into some nice side projects and ideas.
I thought it might be cool to do something like that by buying up a bunch of foreclosed land in Detroit.
You are right, and yet, when the salary had been established it should be possible to talk about trading somebody's salary (money) for equity. And compare that deal with the deal that the investors investing money get. As it is possible to take that 30k/year salary and invest is elsewhere, on the terms that investors investing money get (convertible debt, valuation cap, etc).

Thinking about it as money allows one to make a prudent investment decision.

I'm actually surprised that YC people haven't streamlined that process of trading early-employee salaries for convertible debt. Current situation with early-employees equity really screws up a lot of early employees and in my opinion poisons the startup atmosphere in the Bay Area quite a bit.
How to create different shapes using the clip-path CSS property
I'd say you were just too early, everyone will benefit from them having pushed these ideas further.
https://github.com/yjwen/org-reveal/ is also pretty nice for exporting from org-mode to a shiny HTML5 presentation
Jifbox: a GIF photobooth for the web
Is there anything wrong with him benefiting from it? Mike is a real guy, with a real bootstrapped company, who gives out tons of advice at his podcast (link in another comment here).

I bought his MicroConf/Podcast partner Rob's book a while back, and it was well worth the money I spent on it, and I continue to advocate it for people interested in doing bootstrapped startups. I haven't read Mike's book yet, but from looking at the ToC, it looks to be of similar high quality.
You can invite me to the design project https://www.upwork.com/o/profiles/users/_~015159f894bb70a226...
Ouch for Netflix and others as it looks like movies will cost about $18 to watch.

"Services for bandwidth-heavy applications like streaming movies and large downloads will be available for purchase on our Fly-Fi® Plus plan at $9/hour"

http://www.jetblue.com/flying-on-jetblue/wifi/
It sounds like this is mostly stuff that appeared in newspapers in the 1800s, so the term of any possible protection they had has long expired. Unpublished writings would be treated differently, but even they would be out of copyright if first published today.
Thanks for the mostly transparent insight into your experience.

How does one enter a deal to pay back investors with a shuttered operation? Does that mean make them whole or just pay something back? It seems like someone is holding onto a bag - THE bag?
Show HN: Q-directives: A faster directive system for Angular.js
That seemed to be what they did. Only a pod cast radio instead of a headphone amp.
Erdös Writing Like a Programmer
Afterward, the entire region was remodeled on American style government representation, modern central banking, market economies and so on.

No, not really. The dominant form of government in Europe is the parliamentary republic (several at the backdrop of constitutional/ceremonial monarchies), compared to the more unique federalist model of the United States where there is a strict separation between the union and localities, and the President being the head executive as opposed to a Prime Minister.

Europe was also quicker to apply mixed market principles and has generally been higher on the dirigisme aspect (even though the US is hardly some paragon of non-interventionism itself).

Central banking was initially a European (British, I think?) invention.
while I think the topic looks interesting - i'm certainly among the target audience - its a bit too heavy in sales pitch.

First, i'm not giving you my email address just so I can get the table of contents and a free chapter. I know that "converts" better but I think given the target audience, its tasteless.

Second, there's really no "content" in this page that gives me any kind of inclination that this book is going to be good. They vouch for the author and the sources, but not for what the book contains. Maybe their advice is run of the mill and can be found anywhere? or maybe its so unique and interesting they're afraid to give any of it away? I couldn't say which.

Ultimately, this feels more like a get rich scheme than someone having something legitimate to offer. Maybe this could all change with a revision to the website. I'd be more compelled to take a look at this if this was simply up on amazon with tons of great reviews.
> It wasn't even like Sky and Skype are directly competing with one another

Not yet, but as the article says, Skype were seeking to register their name as a trademark in a number of fields where Sky DO operate - e.g. Products, services, TV.
One category has a typo: 'entertaiment'
It's really easy to say this is why someone failed and this is what they should have done instead but take a moment to walk in their shoes. Do you really think it never occurred to them? Every decision has incredible amount of weight to it and only they had to the bear the consequences of the those decisions.
Yeah, you should generate an always-increasing sequence number in the source database.

Timestamps cannot be used to "select ... where T > ?", since at the time of polling another record may come in with the same timestamp as the newest one, and this will skip records; "where T >= ?" won't work either unless you can de-duplicate, and that requires local state.
> some of the non-French speaking neighbourhoods are quite dangerous

there are no such thing as "non-French speaking neighborhoods" in France.
A History of US Secret Human Experimentation
What smartest people do on weekend is what everyone will do in ten years
If your business hinges on "playing the VC game", do you really have a business?
x86 CPUs have always been forgiving of unaligned access. So while modern x86 CPUs might have resolved this performance issue, if you ever want to run your software on ARM, for example, you're still going to have to align everything.
Thanks! Will fix it!
POP!
> This post talks about the use of OS low level atomic functions

This is a pet peeve of mine, to call that an "OS" feature. In all recent CPUs I know of, atomic ops are not a privileged operation, and there is absolutely nothing for the operating system to manage in a traditional sense. You don't trap into the kernel and have it compare-and-swap, you just, um, compare and swap.

Maybe your OS provides a convenient C API, but it is not "OS" functionality. It's just instructions on your CPU. You could just as well write them inline. In many common uses, that's what ends up happening - the atomic ops are put inline with the rest of your code.
Police CAN obtain cell-phone location records without warrant
As far as I understand, the majority of Node core contributors thought it necessary to move Node.js to io.js, and are now active there. io.js is basically the new Node.js, barring the name. It's less a fork, and more a rename, to get control back from Joyent who could have threatened some of the core team over the trademark.
VPN will make you a target. That's the whole point of recording at the ISP level, the traffic is still quite individualized.
Free Guided Meditation Course
Great advice! Would you build the MVP first before pitching to the data owners? My prototype only indexed (one time scrape) 10 sites and still relies on a fair bit of imagination from the business owner. I'm thinking an MVP would have to index at least 100 or so sites before being at all useful.
I saw an interesting talk by Professor Richard Wrangham (disclosure: a personal friend) of the Harvard Human Evolutionary Biology department that addressed part of this once. He pointed out the open question of whether substantial intelligence was an expected likely fallout of life evolving or a rare occurrence that happened via a bunch of low probability events. He concluded that it was more likely to be a reasonably common occurrence, based on claiming that the intelligence of the most intelligent creature on Earth through the fossil record has increased monotonically, even though they have not shared a common evolutionary line. Meaning that at any given time the most intelligent organism might be a primate or fish or whatever on distinct evolutionary paths, but that the peak intelligence across the planet keeps increasing.
Plagarism/copyright infringement can be proven through the use of 'trap streets' - wait for the site to scrape the fake data off your site, which could only have come from you, since you made it up.

http://en.wikipedia.org/wiki/Trap_street
Pretty good resource here:

https://www.privacytools.io/
Your point is smart and very well-taken. You don't want to push people in a direction they don't really want to go. They are liable regret it later.

A person, especially a young person, can become convinced that they are OK with a path they don't much care for, for a few years, maybe a decade. But eventually fate will come knocking, and they will walk away or get depressed. You don't want to be the cause of those lost years.
Or just use std::atomic and other std::mutex in Objective-C++. Under Objective-C world, no memory semantics are well-defined, and all these are hacks on pile of other hacks.
Awesome resignation letter from a CEO to the company board
How did Wattage secure a deal to return its investors money? Presumably they had spent it. Did they arrange a loan on terms that would give the original investors liquidation preferences?
Last Fall I was in Minsk, Belarus. I was wondering why streets are so wide. It turned out that all (for all practical purposes) buildings were destroyed by Nazis about before their departure. On our way, the only building that survived (was older than 1944) was concentration camp building.

Minsk was rebuilt from ground up, really.

I don't know the real percentage of destroyed buildings in Minsk, but I guess it is about same or even bigger.
I'm sorry, but this makes no sense.

given all the human and computer driven computation to create simulations on a screen resulting in something 1/100,000,000th as complex as a sqft of rainforest

This is an absurd comparison. Besides being apples to oranges, besides being super arbitrary (where did you get 1 square foot?), besides being completely undefined (what kinds of simulations are you talking about?).. it doesn't make any sense.

Let's follow your logic a little bit.
* A butterfly in the rainforest creates a chrysalis. That chrysalis is not real because it's 1/100,000,000th as complex as a square foot of rainforest. It's a noble and passionate endeavor, but it doesn't even approach the richness and complexity of reality outside that chrysalis.

Does that make sense to you? Do you agree with it?

Let's use your logic to try to compare apples to apples. Let's compare a square mile of rainforest to a square mile of New York City. Is one more real than the other? Since you seem to think complexity has something to do with realness (an idea that seems silly and arbitrary, but let's go with it), is one more complex than the other? Hard to say. The rainforest almost certainly has more individual living things. But is that how we define complexity? The city is chock full of human brains, and it's hard to argue with the fact that the human brain is one of the most complex things on the planet. The city also seems low in entropy compared to the jungle. The very fact that a city exists, distinct from the jungle, seems to imply lower entropy. If you let the city go without tending for a long time (and if its location was right), it would become jungle. You might argue that that's because there's no one keeping the complexity high. And since high complexity means more real (I still don't get it, but I'm just going with your "logic"), I have shown the city is more real than the jungle.

Does that make sense? Not really? It's definitely not rigorous. It's not based on sound principles. The logic is faulty in all kinds of ways. It makes leaps of logic and uses poorly defined concepts. Does that stand out more when I do it compared to when you're doing it? I urge you to not be "that guy" or "that girl" that uses that kind of logic.
A Victory for Innovation
Show HN: Unit-test your readme's code examples
These types of article always lead me to start pondering the strong and weak anthropic principles again. At which point I have to give up on getting any more work done as my mind goes chasing itself down the rabbit hole.
But that's not how they marketed themselves from the get go. It was more about enabling the end user to come up with stuff (I remember reading they were basically targeting other software developers who had a hardware itch they wanted scratched). I think that was _specifically_ the leap too many.
Tips for Customizing Your IPython and Pandas Display – Practical Business Python
Kafka does indeed have a good design. But it doesn't solve the potential transaciton gap between your store and the queue.

For example, if you commit a transaction but you're unable to reach the Kafka queue (because you crash, you're SIGTERMed, or there's heavy load causing a network blip, or any other number of reasons), you'll lose updates. You can't very well write to Kafka before you commit, because it's not visible yet outside the transaction.

The only way is to use a transaction log in the same database, in a way that lets the log be read after the commit is done. Logical streaming would let you do this (Bottled Water [1], as someone else here mentioned, does this with Kafka) in a safe way. It's conceptually identical to storing a transaction log table, but wouldn't require as much custom code, and you'd get incremental updates for free.

[1] http://blog.confluent.io/2015/04/23/bottled-water-real-time-...
How sorry I am for MapR that Michael has left them .
He is one of the best architects I've met .
The professionalization of the peer-to-peer space is getting incredibly interesting. Hotels must be realizing that hosts are starting to offer places that are essentially like hotels... only with much more diversity, many more locations, more unique features. Tools like this are definitely driving some interesting changes in the space.