skysight-inc/HackerNews huggingface dataset:
It would be very good to have one or a few solid legal decisions relating to scraping. Today, as far as I know, the entire segment is still in limbo.

It is important to understand that, legal or not, in the US anyone can sue you for anything. In another comment people were discussing the legality of following or ignoring "robots.txt". I tend to be pragmatic about this stuff. If you fabricate your own legal interpretation and end-up being sued by LinkedIn, it could end-up costing you $250K.

When facing large corporations law firms often ask for sizable retainers ($100K+) and proof of cash-on-hand to go beyond that, $250K total not being an unusual number. They don't do this out of greed. They do it because litigation at those levels can be very expensive. If you only have $100K you could find yourself burning through all of it quickly. If you don't have more cash to continue you'll lose the lawsuit and the $100K you spent will be burned for no reason at all. In other words, the law firm is protecting you by asking you to have enough cash to litigate.

A few years ago we started to develop a very extensive product based on obtaining data from Amazon. Some of the data is available through their API and other data had to be scraped in various forms. The product is extremely valuable yet the issues pertaining to scraping made me decide to put it on hold. Even if you can make millions a year the prospect facing a monster like Amazon in a lawsuit, as improbable as it might be, is scary enough to go look for other pastures.
An example in the article was over 37TB, and another was 205TB containing "hundreds of thousands of uncompressed InnoDB tables".

Maybe that's a system that could have been better served by something like Cassandra? (Depending, of course, on access patterns, complexity of queries, etc. etc.)
Part of giphy's problem is that its search is strictly worse than using google and filtering by "animated".
Hmm... interesting.

I'm not sure how practical the "to law enforcement" stop is here, for two reasons. First, lots of these trackers are based in places where their data collection-and-sales are legal - the EU makes decent efforts to pursue foreign actors, but it's a serious limitation nonetheless. Second, it's not clear how often illegal action can be demonstrated. The authors of the DefCon presentation here only identified one of the ten extensions they suspected because the rest couldn't be identified beyond reasonable doubt; with data being gathered by so many actors identifying one is hard.

Even so, producing a bank of highly-suspicious extensions could be truly valuable. Law enforcement (sometimes) has the resources to pursue these things beyond a DefCon presentation - honeypotting might make it possible to advance suspicion to proof by providing fake data to specific trackers. And publicizing high-likelihood suspicions might produce publicity against bad actors and moves to transparency and guarantees by better actors.

At any rate I'm interested by the idea.
Thanks for the mention j_s! This is a really interesting thread and a long article by Cliff. I'm working my way through it now.

Looks like the "current salary" question came up. Any other topics I should jump on for this thread? (Soooo many comments, it's taking a long time to get through them.)

Cheers!
Twin Peaks with ARKit and Mapbox Unity SDK
It would be interesting to put a TOS into your scraper by the pragma or useragent field that says if you return data to the scraper you accept the TOS.
The patch is not easily visible, because it consists of tiny pieces almost invisible to the naked eye.[7] Most of its contents are suspended beneath the surface of the ocean,[8] and the relatively low density of the plastic debris is, according to one scientific study, 5.1 kilograms per square kilometer of ocean area (5.1 mg/m2).[9]

https://en.wikipedia.org/wiki/Great_Pacific_garbage_patch
Which the JOE folks make quite clear. Why would anyone assume this was not for complete novices new to programming in general? It seems odd to me.

I find it a very good start for someone new to programming with zero experience and wishing to start with elixir.

There are hardly any books that approach programming elixir from a complete lack of programming knowledge POV. I'm happy to see this!
Submit Your Infographics to Infozyum for Free
Fish antibiotics are human antibiotics.
And they said it was a myth just like the Bermuda Triangle...
I don't think it will be regulated any time soon.

However: If or when it is regulated, it is likely that existing software development standards will be adapted to ensure the presence of a safety case, adequate testing and so on and so forth.

The vast majority of software projects don't have to comply with quality standards like these, so there is a dearth of open source software to help deal with e.g. requirements specifications and traceability.

I'm taking a bit of a long term punt with my side project and am trying to implement a set of integrated open source tools tailored to the development of artificial intelligence applications (esp. machine vision), but with support for aerospace-style quality requirements such as (the truly excellent) SMC-S-012.

High Integrity Artificial Intelligence:-
https://github.com/wtpayne/hiai.git

It is still early days ... but I'm chipping away at it gradually...
The regulatory agency on Nuclear power is stiflingly draconian.

That's a feature. It's why Harrisburg, PA is still populated.
Do you have any more info on these RO storage systems? What springs to mind is... tape?!
If we selfishly look up the food chain, how does this affect fish we eat? What are the human health impacts of eating fish that ate micro-particles of plastic?
Truffle Framework – Your Ethereum Swiss Army Knife
Nice, Leonid Afremov is a great choice of input art.
>Certainly, with this report, you are nowhere close to having presented evidence for the extraordinary claim that "hardcore Islamic immigration" has created an epidemic of "honor killings".

Notice how the word "epidemic" (and thus the rendering of my claim as "extraordinary") is yours.

I only said that honor killings "have been re-introduced as a practice" in Europe [1] due to fundamental muslim immigrants, which is neither extraordinary nor up to argument.

Seems like the word "sketchy" has also been thrown around a lot aiming to delegitimize any claim that such things can and do happen nowadays in Europe and based of specific, religious and regional based, cultural values.

For all the sketchiness in favor of over-reporting such things that you mention, there's as much an underreporting out of fear of looking as culturally insensitive/racist/etc, and out of lack of access (of social workers, police, etc) in those communities.

And this is not some fringe right-wing publication I've linked to, this is an official EU briefing report.

And it makes total sense that the difference between "an honor crime and domestic violence is cultural intent", since by definition a honor killing is a cultural act, based on the idea that one's status/standing in the community has been compromised because of the behavior of its daughter, children, etc. Domestic abuse is a more widespread umbrella category. You can have domestic abuse without honor crimes. Or you can practice various kinds of domestic abuse AND honor crimes (which is often the case in families that would consider honor crimes acceptable). Domestic violence is indeed highly prevalent across Europe and the US, but it is also highly prevalent across subcultures that consider "honor crimes" an acceptable practice.

[1] This also shows that I have no problem to admit that those were once a common enough practice by christian Europeans too. It has been on the wane all through the 20th century, though, before being reignited by newly arrived populations.
Depends on what you mean by it.... people do definitely use it to imply a sedentary night watching tv, although others use it ironically to indicate a booty call.
Oh no... I commend your effort, but this is totally ill-conceived. What's even more disheartening is the number of people who looked at this and also thought it was a good idea.

There is just so much wrong here... let's start with the basics. If this is truly a FSM then why on earth are you using a transaction table (state is a value)? An accumulating snapshot table (state is a field), besides being EXACTLY for this purpose, will be far more efficient in almost every way. Your "analytic" queries would be simple select statements, and your trigger (shudder) could be reduced to simple constraints. The sheer amount of over-engineering here is staggering.

Lastly, what is the purpose of putting the transition logic in the database? It's simply redundant upon actual implementation. Somewhere, somehow, another program has to actually carry out your actions (paying/shipping/cancelling) on the order and make a call to your database with the appropriate event inputs. So why not just put the entire state machine with the rest of your business logic? As many have pointed out, this is where it should be anyway.
This is important, we need to keep watching this closely because it might deter future big companies to manufacture in the USA.

Think about Apple. This is one more thing to worry about on top of higher salaries than China.
They're not cheaper. They're identical. Antibiotics for humans are cheaper because they are more commonly distributed. Amoxicillin is $9.

You need a prescription due to government regulations. The FDA wants to prevent antibiotic resistance from developing in humans.
Again, you are thinking in terms of propellant carried up from Earth.

Thrust available to a spacecraft using a solar sail is a function of distance to the sun. The sail is a fixed cost, but it may be presumed to degrade over time. It has no propellant that costs additional money to launch.

If you go in, the ever decreasing efficiency of the sail is offset by the greater available energy from the solar wind, and you may still be able to complete the mission (more slowly) with a damaged sail. Your available thrust increases with every kilometer closer to the sun.

If you go out, the decreasing efficiency of the sail compounds with the lesser total energy available to the sail, and if the sail is damaged, you may never reach escape velocity at all. You will never have more thrust than the instant the sail deploys.
It's better than calling it an island which I would definitely assume means "visible".
There's a huge subsidy in that the Federal Government subsidizes the disaster insurance.
APNG support landed in Chrome 59, released in June!
Would you be ok if dropping a value were to just mark it as 'deleted' from the catalogs, and no new values could be assigned? The reason enums have these weird restrictions is that they can appear in indexes, even after a value has been deleted (or its creation rolled back). If we don't know how to compare them after deletion, the index can't be correctly traversed anymore...
The plastic absorbs various pollutants. Fish eat the microplastic, and thus the pollutant. Bigger fish eat small fish, and we eat those bigger fish.
Wow http://elixirbridge.org seems really interesting! I'm following the official tutorial right now to get the basic down and will go through the book Programming Phoenix
> by someone with no experience in the language

What evidence do you have?
You might want to read the linked article, because that's not what it says:
http://www.businessinsider.com/elon-musk-tesla-factory-injur...

"Musk then said he wants direct reports about every injury, without exception, going forward. He also plans to spend time on the assembly line."
Good question. I'm working remotely myself and can't relocate easily because my wife is a doctor and it's very difficult for her to move between medical systems. I was lucky to join this team when this wasn't considered a problem.

That being said, I'm in Shanghai frequently to sync with my colleagues and it's an amazing city. I've found pretty much most conveniences available to me in Berlin and to some extend even better. There is an active expat community, and a fascinating vibe.

My manager (from SF, USA) has been there for over a year now and has enjoyed the experience a lot and extended his stay. YMMV, but I'm fairly well traveled and would say that it's a pretty great spot if you're willing to emerge in a foreign language and culture.
With the current state of the art, I'm more worried about generative methods being used to make the fake news problem infinitely worse, because now we can make convincing fake images, video, and sound bites.

That is, I'm much more worried about deliberate misuse of algorithms and data, which can be done now, or in the very near future, and which regulation may not really help with.

We should be less concerned about the singularity, and more concerned about the first time machine-generated audio or video clip is used to "prove" misbehavior, or exonerate someone who should be in jail. With the quality that we're starting to see [1], those who can through a few hundred thousand $$ at the right people are going to be the first to exploit it. Will we know when it happens? Has it happened already?

[1]: https://boingboing.net/2017/07/17/fake-obama-speech-is-the-b...
>People set world records without ever using a strange shoe.

People set world records without using shoes even. The Vibram is less strange and more a compromise between foot protection and flexibility. In other words: gloves aren't strange, mittens are.
Also here in the Netherlands something similar happened. A company with nuclear waste went bankrupt and it became a expensive problem.
The entry you linked to agrees with my definition:

"Strictly, a plethora is not just an abundance of something, it is an excessive amount."
Gif is far more useful as a media type rather than file format; and more than that, i don't think most people realize or care that they're actually watching some kind of video-sans-audio.
Where is the memory card physically stored? Is that something that a hacker could easily gain access to without being noticed?
If the top result is advertised but still decent content and I still have the option to shuffle, I don't think that would deter me.
Ha! A few gb ram and a few tb disk is real cheap in the cloud.

If your dataset fits in ram, and when it doesn't you have to buy more boxes with more ram rather than just buying more disk, then I think you are making a very uneconomical trade off.
Server time doesn't pay employees.
Let's cut a crap, shell we? We can't solve climate change problem which might be acute for last 20 years, yet we believe somehow that we are able to solve storage issue of nuclear waste for next 100 000. Egyptian pyramids are ONLY 5000 years old! And we are not 100% sure what is written in there.

How do you warn next generation after 10 000 years, that some particular site is dangerous/radiaoctive? How do we keep something safe for 100 000 years? Is our Earth look same after 20 000 years, 50 000 years, 70 000 years? Will there be new volcano or shift of tectonic plates? Ice age? How do you keep such waste safe?

Even as of today, there is no final storage solution for spent nuclear fuel. There is one know being built in Finland, and it is just for waste produces in Finland. BTW, there is very nice movie about it: Into Eternity. You should look it!
Let's cut a crap, shell we? We can't solve climate change problem which might be acute for last 20 years, yet we believe somehow that we are able to solve storage issue of nuclear waste for next 100 000. Egyptian pyramids are ONLY 5000 years old! And we are not 100% sure what is written in there.

How do you warn next generation after 10 000 years, that some particular site is dangerous/radiaoctive? How do we keep something safe for 100 000 years? Is our Earth look same after 20 000 years, 50 000 years, 70 000 years? Will there be new volcano or shift of tectonic plates? Ice age? How do you keep such waste safe? And what is the true cost of nuclear energy if you account 100 000 years storage of it?

Even as of today, there is no final storage solution for spent nuclear fuel. There is one know being built in Finland, and it is just for waste produces in Finland. BTW, there is very nice movie about it: Into Eternity. You should look it!
it's a separate service, but available at a discounted price for prime subscribers.
Anecdotally, yes – most doctor's visits seem to end in steroid or antibiotic prescriptions. I've known of some doctors that prescribe antibiotics for things like colds just to give the patient a placebo.
You don't need Draconian regulation to have effective regulation.
No they don't. But people desire them even when they're not necessary. Which is why they resort to taking "animal" (actually human with animal branding) antibiotics.
This is very very bad idea because this bacteria can also consume the not discarded plastics (unless this only happens under certain controlled conditions).
So already your tech is obsolete by the time you start building.

Cutting edge tech tends to be unproven and unreliable tech. When you can lose an entire city if things go wrong, worrying about having the latest tech is the wrong priority.
I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service.

Every service has "terms of service" which are the conditions that you are allowed to access the service and what you may do with the service once being granted access. For example, if you start pouring toxic waste into your sewer, you will find that the city will both disconnect you from the service and they will fine you for violating the terms of service you nominally agreed to when being hooked up.

In LinkedIn's case, they allow you to access their service, with HTTP, to render a page in a browser for viewing of that page. Full stop. Any other use of the data you acquire over HTTP, or any other method of acquiring said data over HTTP is disallowed by the terms of service.

Not only does LinkedIn have a legal right to stop scraping after the fact, they have literally centuries of common law in support of that position.
Facebook acquires Ozlo to build out its personal assistant
Would libel/slander laws cover making fake videos of a real person saying things they didn't? Even if they did, I don't know if they would cover creating fake news if it didn't say anything untrue about a specific person.
SoundCloud problem was that it tried to chase and copy competitors instead of being unique--but it has truly unique user experience which it completely ignores (probably need to write about it a blogpost). By following competition you don't win. You don't get a leadership position. You just lose slower.

As a result, SoundCloud faced a notorious problem of wholesale transfer pricing [1] power. And it didn't have enough money to win or even participate in that battle.

[1] Wholesale transfer pricing = the bargaining power of company A that supplies a unique product XYZ to Company B which may enable company A to take the profits of company B by increasing the wholesale price of XYZ.
Scraping would be forbidden via TOS; theoretically it would be the users giving access to linked in (who actually is bound by the terms) who would be liable, not linked in.
Just because present state is 99.7579% uptime (for the month) doesn't mean it will always be so.

You back up your data, why shouldn't you backup your github data?
puuuh, somebody should explain the concept of user-agent groups to them (would heavily simplify their robots.txt)
Actually, this is a list of neural network simulators, so I was surprised to see even Theano, which isn't designed to represent a biological neural network at all.
Voting needs something called "homomorphic encryption", which allows simple arithmetic to be performed on encrypted data without decrypting it.
Wow they're going to Digg themselves into oblivion, pun definitely intended!
> People set world records without ever using a strange shoe.

Zola Budd was pretty fast with no shoes. http://www.garycohenrunning.com/images/ZolaBarefoot.jpg
Yes, the idea behind a blockchain is a merkle tree of events, metadata and salt that hashes with a predetermined number of 0's (the difficulty) as a prefix.

If you wanted to make your event stream more tamper-proof you could also introduce a merkle tree-like row which hashes the current event, the previous hash and a secret key. Any tampering of the data would need this key and would need to recompute future events.

It's still not perfect because with the key this is easy to do.
That makes sense. Will reach out to the interviewer to get a more detailed feedback. We also share the full code + questions as part of the full evaluation. Would you be interested in interviewing the candidate as a function of this evaluation? Is the interviewer identity important?
If ever you decide you want to migrate away from Postgres, you'll need to rewrite all the business logic into some other out of process languag

You'll rewrite your front end 20 times in 20 different "frameworks", and your app 5 times in 5 different languages, for everytime you actually change databases
I find in general it's helpful to prepare for all interactions. Salary negotiation especially. When you get on a call to talk about salary with a recruiter you should already know pretty much how that call is going to go. If someone catches you off guard with, "Hey your demo shit the bed and ruined the client meeting" just ask them to schedule a meeting to talk about it. Always feel you like you can 'reject the premise' that you need to have a conversation on someone else's terms.
Germany's manufacturing companies have massive union presence in everything they do and they're some of the most successful in the planet.

Let's stop with this ridiculous demonization of unions. American corporations have chosen a confrontational model with unions and staff. It doesn't have to be this way. There are better models around if the owners of capital are willing to cooperate with workers.
I just rewrote the template completely. It's not that hard. So I do not have to deal with standard templates issues.
3 Ways to the Do the STARMAN Remake
For software engineers it would be something vaguer like showing that you've succeeded on a project with sufficient complexity. But I don't know what the equivalent would be for factory workers.
The weird thing to me is that PayPal accounts are wholly separate from Venmo accounts. I already had my PayPal account linked to my bank, and I had to do this tedious process all over again when I connected Venmo to my bank. Pretty lame considering it's the same company.
Epic win on the weight conversion :-)
Owners of plants are responsible for the first $12b. 3 mile cleanup was well under this. So I'm going to have to disagree. But I'll give a caveat, it is a sample size of 1. A major accident would cost much more, but they are extremely unlikely.
Titan Xp Gets New Drivers, Will Accelerate Professional Apps Like Maya
I do not think so. Too many blogs are there that provide Google with data and show Google Ads. It's a valuable asset for Google.
Why not use plastic?
Today we're announcing the availability of this API across all Android devices running Google Play services 11.0 and up.

Of course, Google had to lock up peer to peer communication by tying it to their online store. It's also unencrypted and unauthenticated, so you don't want to use this for home control.
Can second this opinion, and I was using it back in 2011 when it was a brand new experimental thing from Nokia (which at the time owned Qt). My first reaction was "omg I want to be able to use QML on the web"
I'm a Tesla shareholder (of little significance in the grand scheme of things, but I bought some) and this wouldn't put me off. In fact, I think they should have a union.