skysight-inc/HackerNews huggingface dataset:
Is a shorter solution possible?

Is decompiling SWF and/or de-obfuscating Javascript really necessary?

Years ago, I approached these sites like the author in the blog post. I spent hours reading about Flash and figuring out the schemes websites used. I decompiled swf. I also tried rtmpdump, livestreamer and other all-in-one solutions. I used early youtube downloaders (clive, etc) and later quvi. I also used /dev/bpf to capture HTTP requests. I tried it all.

I can relate to what the author is trying to do. I never settle for "streaming". Download only.

However I found over the years either the websites have made things easier or I was simply trying too hard. Or maybe I just am not interested in the type of video that is served in this way. For some reason it takes much less effort now.

To make sure I am not imagining this, I decided to try one of the author's examples.

I chose projectfreetv.so

1. I followed a link on the main page for some TV show I have never seen.

2. I read the HTML.

3. I noticed /watch/?aff_id= URL's.

4. I chose the first one, 493165, and followed it.

5. I read the HTML.

6. I noticed a URL pointing to a website that serves TV shows and followed it.

7. I read the HTML.

8. I noticed a URL pointing to an MP4 file.

9. I downloaded the file.

I assume I got the TV episode on offer. The screen size is a little small but the file was about 89MB and appeared to be the full episode.

Total time: less than 10min

What did I use? sed, netcat, less and ftp for the download. The entire process is a one-liner.

If anyone has some more examples to try, please list them. I would welcome the challenge.
The Go FAQ addresses this: http://golang.org/doc/faq#garbage_collection

Upon re-reading, it's distressing to hear that the GC latency has been raised from "not significant" to "low."
Part of it is also things getting so good in the world (most people in America don't have to worry about shelter or food) that people start to invent problems or find challenges to solve.
> No, because there aren't enough people who are willing to pay $3000.

There are. There are far more than enough, because...

> Rent controls push the average rent down temporarily (until supply drops to regain equilibrium)

... there isn't enough supply to meet demand. There are literally _not_ _enough_ _apartments_ for the people who want to come into the city. There hasn't been enough supply to meet demand in SF for at least ten years. What little is being built will come nowhere close to meeting this year's influx, let alone next year's.

In almost any other city, I would agree that rent controls are obscene. Here in SF, rent controls don't go far enough. Units built before 1970-mumble should not reset to "market rate" upon vacancy.
How about a puzzle game about Forth, written in Forth, running on a Forth machine? https://github.com/JohnEarnest/Mako/tree/master/games/Warrio...

Obligatory Javascript implementation of said Forth VM: http://johnearnest.github.io/Mako.js/?rom=Warrior
Where does that peanut allergy thing come from suddenly? When I was a kid no one had any allergies and now every other kid seems to due from a peanut.
http://www.hammerspoon.org/ allowed me to write a small i3 replacement, it's not as complete, but it's mine
An update on CSV download scripts
You may enjoy this: https://chrome.google.com/webstore/detail/the-great-suspende...
Not sure why you don't want to think that it could be a false memory; those are pretty common, particularly when you're dealing with a very young child and a very old memory (also seems to be common when dealing with dreams). Of course there are other possible explanations as well. Another user user points out it could have been a groggy child being carried. It could also be a groggy child that either saw for a few moments or heard people talk about the square pizza, and then incorporated that into their dream.
> Now you add 1024 bits to the pool.

A further complication - you don't actually add 1024 bits to the pool. Instead, you add N bits to the pool (N > 1024), estimate its entropy as 1024, and increase the overall entropy estimate by that much.

Estimates are always pessimistic, because it's better to be cautious and under-estimate the entropy than the alternative. So you might have 2048 bits or more of real entropy, but that can still be "depleted" by a request for 1024 bits.

The paper that describes Fortuna [0] goes into greater detail on this. Yarrow, the CSPRNG that preceded Fortuna, attempted entropy estimation. Fortuna rejects entropy estimation, and tries to build a CSPRNG that is secure without needing estimates:

"...making any kind of estimate of the amount of entropy is extremely difficult, if not impossible. It depends heavily on how much the attacker knows or can know, but that information is not available to the developers during the design phase. This is Yarrow’s main problem. It tries to measure the entropy of a source using an entropy estimator, and such an estimator is impossible to get right for all situations."

"Fortuna solves the problem of having to define entropy estimators by getting rid of them."

0: https://www.schneier.com/fortuna.pdf
This makes intuitive sense to me but doesn't match up with what I've seen anecdotally. I lived in some very wealthy suburbs of NYC, and nearly every family was a two-earner family. My wife and I also homeschool, and have met a bunch of other homeschoolers both in CT and now in CO. By your logic, you'd expect the primary earners in the homeschooling families to earn more than in the average family, but in my experience the reverse is true.

You're right that there's a level of income at which one parent staying home is just not a plausible choice. But it's a lot lower than I'd expected -- I've met some homeschooling families that are doing extremely well by their children at a substantially lower income level than I'd thought feasible. I think the issue here in most cases is less a problem of economics than of cultural values. (Before anyone reads anything pernicious into that, all I mean is that I think American culture overvalues possessions and professional accomplishments and undervalues family in specific and relationships in general.)
As for performance challenges "sequential synchronous random writes" doesn't make sense to me, what exactly do you mean?

LMDB is read-optimized; for reads nothing else can touch it. For writes, there are other engines that may be faster some of the time.

For sequential writes LMDB cheats. The cheat is described in my LMDB Internals presentations. Nothing else can touch its sequential write speed.

For random writes - LMDB generally loses to other designs when records are smaller than 1/2 a DB page. For larger, LMDB wins, and the margin grows as record sizes grow.
I benefit from rent control.

I rabidly support every new apartment development. Trinity Towers (across the street from the main branch of the SF Library) should be a model for how these things work:

* Remove rent controlled tenants. They will be displaced for 6->12 months.

* Tear down old building.

* Build Phase 1 of new, far denser building, as quickly as possible.

* Move rent-controlled tenants back into their new units at the same rate they had when they were displaced. Those units are covered by rent control forever. The rest of the building is at "market rate".

* Build the rest of the building at your leisure.

The only thing I might add to the plan is city-sponsored housing to put the displaced residents.
This is an absurd editorial. The cited arguments against rent control assume that rent control is just about capping rents. Rent control is about much more than that: it often is about capping rent for long time residents. If your grandmother has lived in her apartment for 65 years, she should not be tossed out just because a 19 year IT worker can pay more. Rent control takes in many factors, and is not just about capping rents in a region. However, I don't expect The Economist or it's reader demographic to care much about those affected by the rabid efforts to dismantle what protects many, just so that people can better profit on their properties.
You can easily set up billing alerts in cloudwatch to do this. I have done so. I agree by default it should prompt you to do so during the aws on boarding process.
Actually, the preferred and recommended way to get randomness on modern kernels is to use the new getrandom(2) system call, with the flags argument set to zero.

http://man7.org/linux/man-pages/man2/getrandom.2.html

With the flags set to zero, it works like the getentropy(2) system call in OpenBSD. In fact, code that uses getentropy(buf, buflen) can be trivially ported to Linux as getrandom(buf, buflen, 0).
Not a single sentence of mine has a subject other than "Berkeley DB" or "it". I'm completely baffled how you read anything categorical from my post.

"More likely" is not documentation, when you need guarantees that certain pages are not evicted in favor of others. (If my memory of the source code serves me, these levels map to internal values that have some nonlinear relationship with some LRU cache parameter, with the highest priority effectively locking pages into cache, except for certain cases which I forget.)

Also, the powers-of-two phrase is a recent addition: http://web.stanford.edu/class/cs276a/projects/docs/berkeleyd...
Universities are largely to blame for this. Having a 3.8 GPA versus a 3.9 can dramatically reduce your chances of getting into a top university. Being in the top 12% of your graduating class versus the top 8% can hurt your chances of getting in. It's stupid, because these are statistically meaningless differences, yet it matters. In an environment like that, it's rational for parents to get involved.
It may well might be more local than "all of MI" (plausibly only A2?).

It was admittedly second hand, but when I lived out there a friend (with local parents) noted that their property taxes went up even though their house value had plummeted, because the value was still above the assessed value. I remember being mildly surprised because I had also not heard of such a system outside California.
If you'd ever tasted poverty, you might think better about the pursuit of money.
Not sure why it isn't listed in the overview, but this is based on SQLite (hence the name). HVent looked into the details yet but it looks pretty fun
Very true, but one of the things that was clear in the article was that while the contents of the secret societies were secret, their existence was not.
I fail to see what that has to do with a modern justice system.

Crime can be personal, but law and justice are statistical, bureaucratic instruments that target societies, not individuals. They only produce as an accidental external factor occasional catharsis for the victim.
Bitcoin simulator click game
This housing pressure has been -with the possible exception of a blip during 2008- near-constant for at least the past ten years. How long must a trend persist before it is no longer short-term? :)
Who Has It Better, Law or Culinary School Grads?
If an 80 year old pensioner has lived in their San Francisco apartment for 50 years, but can no longer afford their rent because it has skyrocketed, what do you think should happen?
Django MultiHostMiddleware
You have a better chance at betting on black in a casino than getting ahead trading options.
YouTube gaming live (404)
Is the joke that this is awful advice?

(Except for saving the entropy pool between reboots, which can be useful, afaik.)
I don't think it's absurd. There are definitely some hard policy questions. However, that said, Krugman, an economist many populists like to cite agrees with their position. Rent control is short-term good, long-term bad.

The solution is one of two things, more housing units, or fewer residents in a given area --look at any town/city where there is a pronounced long-term population decline, do you see a similar, "minimum rent" control? No, but you do see rent go down and typically the place suffers many ills.

All it comes down to is everyone wants in when the going is good and everyone wants out when the going gets bad.
"Sequential synchronous random writes" means that you issue writes with random keys in sequence, waiting for each to be synchronously written to durable storage before writing the next. This is an important metric in certain domains, and it is generally bound by the underlying hardware regardless of database architecture (though certain designs can make it much worse).

How does LMDB outperform log-structured databases when writing page-sized objects with random keys when the working set is larger than RAM? I'm genuinely curious. Any other Btree-based database will perform at disk read seek rates, at best.
"Invent problems" seems to be a strange way to word things. It seems to me that with some basic necessities taken care of we have other problems to address?
I think this says more about the current state of AAA games rather than Unreal Engine. Batman and Mortal Kombat have years of engine and gameplay work and diverged from vanilla UE3 a long time ago. It's easier for those studios to upgrade their renderer and port to modern consoles than it is to throw out and rewrite years of custom work. Other surviving AAA franchises use in-house publisher engines or long running franchise specific engines.

ARK (http://www.playark.com/) is a solid success, but it's a mid-tier game. Fable Legends, Gears of War 4, Street Fighter V, Lawbreakers, Tekken 7 and a handful of other games can probably be considered AAA. Given where the games business is right now I think most commercial UE4 titles will be mid-tier indie titles from ex-AAA developers, and Epic has tailored their business model and engine to reflect that reality.
Not trading options, setting up stock options for employees.
Book - Refactoring for Software Design Smells, with Foreword by Grady Booch
In addition to the cyclic references problem, rc has a high constant overhead (memory traffic from constant manipulation of rc fields) and a problem with pauses (cascading deallocations). Also it precludes efficient shared memory concurrent access of managed objects.
> Then, yes, your body changes, adapts and becomes more efficient, because you're changing things, and it's getting lighter so be definition it needs less calories.

This doesn't require any "adjustment" on the body's part, it is a natural outcome of having less mass to maintain. I am speaking of something different, of the possibility of the body sensing starvation and shifting into kind of a high efficiency sustainment mode, perhaps lower metabolism, perhaps more efficient extraction of calories from food. Does this actually happen, I don't know, but I believe from personal experience and reading forums that there is a phenomenon of hitting a weight plateau that is not explained by the simplistic calories in, calories out theory. I don't know why this idea seems so offensive to people.

> I wouldn't go there. I mean, if you want to change the rate of weight loss, just eat absolutely zero calories for 6 months. I grantee you'll lose a massive amount of weight.

Why do conversations on certain topics have to degrade to nonsense if someone doesn't agree with your point of view?

> It's interesting to me you're still viewing this as "weight loss", and something to be maximized and made more efficient

Some people, me for example, can diet for a period, drop <x> pounds, and then I'm good for years with paying very little attention to diet - not everyone has it that easy though. But while I am dieting, I prefer to optimize it so it's over as quickly as possible.

The idea of optimizing things is a fairly common theme on this site, but again, there seems to be something about the topic of dieting that changes the way people think.
I would start with rpi-buildroot and build up from there. If you have low latency audio questions feel free to email me.

https://github.com/gamaral/rpi-buildroot
Personally? I think the state should underwrite those folks in housing set aside for retirees. Since they are retired, there is no need for them to be close to jobs --they do need to be close to medical and shopping facilities, etc, or provide those services via alternative means (door delivery, personal visits, etc.)
Opportunity cost is the loss of potential gains from other choices when you make a decision. It is not the theoretical amount you're pissed off about "losing" but for the existence of a law you don't happen to like.

People who owned property with a certificate of occupancy prior to June 13, 1979 (i.e. a rent-controlled property in SF) can genuinely grouse that they bought an asset with a different expectation of return than what they got. Everyone else bought into a system knowing the rules of the game. It's no more "opportunity cost" than someone who buys a suburban property with land-use restrictions complaining that the laws prevent him from using his land to start a refinery.

If you buy a rent-controlled building today, you know that you're buying a building with a defined revenue stream. It is not "opportunity cost", except in the sense that you chose to buy the building in the first place.
150k job. Bite their hand off.
Thanks, these look quite interesting! I'll read them when I get time.
I can answer the questions on refactoring and design smells. The recent ACM Computing Reviews book review for the book is here: http://t.co/A6FfjZjJd0
Are there text versions of the Clinton Emails?
The State Department released 7,000 of the Clinton emails the other day.

https://foia.state.gov/Search/results.aspx?searchText=*&beginDate=&endDate=&publishedBeginDate=20150801&publishedEndDate=&caseNumber=

Sadly, they've posted PDFs of printed copies of the email.

Are there text versions available? I thought it would be interesting to play with them, doing simple term frequency, map out relationships, etc.
Most of the parents I knew at my son's (private, not inexpensive) schools had two parents working. It was sometimes the case that one worked less (as a substitute teacher part time, maybe). But the one-earner families tended to be outliers.
Happy personal news
Unicorn? Une corne?

Ceci n'est pas une corne.

Sometimes a horn is just a cigar.
> You're not trapped anywhere.

I am. Over the past five years "my" apartment building has changed hands several times. The current landlord is a large company who actively attempts to remove long-term tenants, frequently neglects legally-required upkeep and maintenance and actively attempts to suppress tenants' attempts to organize.

In a reasonable market, this landlord would not be permitted to act as a slumlord; his rooms would remain empty, and his reputation would catch up with him. He can only behave this way because city wages don't track city rents.

The city's failure to build housing to meet demand has trapped a not-insubstantial slice of its population in substandard conditions.

> ...you're forcing someone else to pick up your tab...

Last I checked, the cost of durable goods and competent labor hadn't more tripled over the past five years. This is to say that the cost of renting a room in a building in SF has far outstripped the actual cost of running that same building in SF over the past five years.

I'm pretty sure that it's not me that's running up a tab.

It's abundantly clear that there's something rotten in the market, and that it's not rent control. :)
Then don't call these “unlimited”, because otherwise it's a pure lie. Unlimited is unlimited.
Great read. Helped me understand real world scenarios and usecases! Also the book is well structured. Really Cool..

- Difficult design concepts and smells explained in a very easy manner
- Examples are quite helpful to understand the concepts better
- The explanation flow in each chapter is consistent for easy understanding
- The Anecdotes are quite interesting :)
You have no idea how much I agree with you.

http://qz.com/476722/be-careful-your-love-of-science-looks-a...

I submitted that article awhile back, never made it anywhere. The incredibly unyielding trust in scientific measurements on HN is amazing to me. I agree that science has proven itself an amazing tool of mankind, but I also believe the true nature of the universe, spacetime and the human mind is beyond the reach of human science... or rather, when that truth presents itself to us we'll have to throw out so many things we thought were constants that the result will have us giving up on science all together. In the same way modern scientists look at history's voodoo & whichcraft... some future scholars will laugh at the concept of science.
There are literally not enough apartments for the people who want to come into the city

"Want to come into the city" isn't a yes/no proposition. How much are those people willing to pay to come into the city?

I'm sure there are plenty of people who are living in SF right now who are not willing to pay $3000 to live there. Lucky for them, they don't have to, because their rent is controlled at a lower price than that.

In a free rental market, you would see some current SF residents leave because they're not willing to pay market rates, but the market rate would drop as a result. In other words, long time residents would move out to make space for the people who want to move in and are willing to pay for the privilege.
An easy hack is to use rngd with /dev/urandom as an "entropy source" (risky if you are going to use /dev/random for anything that REQUIRES true entropy, but otherwise should be safe as the article suggests).
I live in the US and have two middle school aged kids. I don't actually know the legal age when children can be left "home alone" so I looked it up. It turns out there is no law and social services advice is to "consider your child's maturity" etc. A quick flip through the web page for free range kids seems to indicate that most US states have no firm age. Therefore it looks like it is not true as stated.

fwiw around here (small-town western US) you see kids as young as 4-5 making their own way to/from school -- either walking or biking, without adults, and the typical parent will leave their kids home alone when they're 10-12, depending on the kid and other factors.
Last time I tried using gpg on a VM it failed to work (literally would not do anything) because it blocked on /dev/random. Would you say that the gpg people should be outsourcing their work to someone else?
s/tripple/triple/g
If $3000 is the market-clearing price for people chasing 5% of SF's housing stock, I can guarantee you that the market-clearing price would be less than $3000 if 100% of SF's housing stock needed to be cleared.
Why didn't we get /dev/getrandom?
OK, for your sequential synchronous random write case, LMDB is fair-to-middling. In fully-synchronous mode (which is the default) it takes 2 fsyncs per transaction, so at least 2 IOPS per write. Many other DBs (including SQLite, Berkeley, and Kyoto) take even more than 2 fsyncs per transaction, and are slower than LMDB. Some others take only 1, and are much faster in this workload.

LMDB outperforms all other engines on larger-than-RAM DBs with page-sized objects because its zero-copy architecture doesn't waste any RAM. For other DBs you must dedicate some fraction of RAM to write buffers and page caches in order to obtain their maximum performance. This means the largest working set size they can keep in RAM is much smaller than the actual physical RAM. With LMDB there are no separate write buffers or caches, all of available RAM can be utilized for your working set. This is precisely the scenario tested here http://symas.com/mdb/hyperdex/

I'll note that in that test, LMDB performed at exactly the disk read seek rate; HyperLevelDB's LSM performed at much worse than that rate.
> in which case application developers may prefer to implement a DRBG using /dev/random as a conditioned seed.

Wait, what? dev/urandom is a DRBG seeded the same way dev/random is. This doesn't seem to make sense.
One point of a direct syscall is that you don't have to open any files. That protects you from file descriptor exhaustion attacks.