skysight-inc/HackerNews huggingface dataset:
"First of all, if you don't know the encoding, then you don't know the encoding"

True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't.

I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether.

And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'.

About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).
> Google could take full control of Android, by not releasing future versions to open source and changing licensing terms. The logic: Android is too big to fail, that OEMs would have no other choice but align their customized platforms with the broader Android ecosystem.

There is yet one more option for Google: instead of moving towards more copyright, move towards copyleft. At the very least, requiring derivative source code to be published would allow the various Android derivatives to build off the work from each other, and allow work to be reintegrated back into the core. Taking it even further, prevention of hardware restrictions on software modification (GPLv3) would encourage modifying and swapping Android derivatives.

Again, Google has to worry about adoption if they make their license more restrictive (and GPLv3 is definitely more restrictive for the makers of derivative works). But if they continue to release excellent software, and there continues to be a lack of viable alternatives, they might get away with it.
Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8.

Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle...
I think you are on to something here. I have worked with many very smart men and women who couldn't produce. I have know doubt that they were smarter than me (more knowledgable, quicker to answers, generally quicker to understand techniques and maths). Despite this, they would spend absurd amounts of time on fairly simple stuff. There was some need to "do it the right way" that led to effective paralysis. I'm not trying to diminish the importance of doing "it" right, but sometimes it is easier to just iterate than get the best solution from first priciples, particularly when we don't know the "right" way to begin with!

Upon talking to those folks, while the never stated it, I came to the conclusion that they were afraid of being wrong, or failing. I do my best to lead by example and fail spectacularly on tasks, only to learn and come up with a good solution when I can, but I am not sure how to communicate that failure on an iteration of 3 can be a huge positive. Even communicating that fact explicitly doesn't seem to mitigate that fear.

I suspect it comes from schooling, where incorrect answers are punished (with bad grades, peer judgement, teacher judgement, parent comments, etc).
My feeling is that your diagnosis is spot-on. But I don't know enough about organizational psychology to know whether such a thing is really possible.

I suppose that logically if one of the google founders had a similar approach to dealing with privacy and data collection, naturally they would hire and attract people with similar attitudes.

I've seen from google employees a general sense of emotional detachment and even contempt of their own customers that I haven't seen in other successful tech companies... it's as if they're not really dealing with fellow human beings.

It's no wonder that they haven't been successful with creating meaningful social services. On an institutional level, they just don't get or even respect how normal people think. Too bad for them...
http://www.utf8everywhere.org/#faq.asians
Again, that is not what I said. What I meant was more along the lines of "You did a great work and I like both the drawing and its meaning. I would love to see that on a T-shirt, and if I were to make that T-shirt it would certainly give you credit for this."

I wouldn't feel like wearing a sweater like that for personal preferences, but I would like this idea to spread. You would definitely have a point if she designed and produced that as a T-shirt and now somebody else started making poor replicas, but hers is probably a unique specimen.
I tend to phrase this as intelligence by attitude.

You get people who are bright, but have virtually no curiousity and who stay within their specialism of knowledge, often due to setting a high value on others views, so not wanting to look stupid.

Conversely you also get creative and determined slow people. And they will thrash the merely bright in almost any problem that contains significant depths.
I clicked on the "0 comments" link, but all I get is the RailsCasts description in isolation, with no obvious way to submit a comment.
Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
Some good suggestions here.

I've considered this issue a lot. Having lost many friends and family members to cancers of various types, and counting cancer survivors among my friends and acquaintances, I am very keenly interested in curing cancer.

Right now, being so deeply invested in my own business, I would wish there were things I could do to simply fund cancer research. But nothing is simple, and there are a lot of problems with cancer research funding: http://www.nytimes.com/2009/06/28/health/research/28cancer.h...
The comment on that piece that most struck me was from the breast cancer survivor who eagerly participated in a research study, given that she knew which carcinogens she'd been exposed to in her lifetime, but found that she was asked useless questions using research money that could've been well-used elsewhere. #28 on this page: http://community.nytimes.com/comments/www.nytimes.com/2009/0...

You will quickly find when you research cancer that acquiring a decade's worth of specialized knowledge isn't going to get you to your goal. It's much better to have a working knowledge of statistics and the ability to read medical papers, something you can do with far less than a MD or PhD.
Markus Kuhn's web page has a lot of useful UTF-8 info and valuable links (e.g. samples of UTF-8 corner cases that people often miss).

http://www.cl.cam.ac.uk/~mgk25/unicode.html
Why you should be careful about 'optimizations'
When I wrote the above comment, I was ballparking it to be around 100nm, which I think is closer to µm than nm.

Looking a bit closer at it, I would say that the bands you see on the feet and body seems like the "precision" of the system. These are around 4 pixels wide. The 20µm ruler is 84 pixels wide, giving us 20µm / 84pixels ≈ 0.25µm/pixels.

So the band is 4 pixels * 0.25µm/pixels = 1µm. (Assuming my premises are right)
Hmmmm...are you logged in?
I wish Google would add some of its apps to the Amazon App Store so I can have a nice gmail reader on my Kindle Fire.
I don't think informal channels, and conferences, which are infrequent, and really expensive to travel to, are enough. Before the 1600s, science was largely done by wealthy people who had large enough houses to have a laboratory in. Scientific results weren't publicly shared; at best they were shared between the experimenter and a few of his/her friends, who communicated by private letters.

In the late 1600s, the first journal was founded, which meant that it became the norm for all scientific results to be shared publicly. This era coincided with the birth of the Scientific Revolution, which was an incredible flourishing of scientific thinking, that formed the basis of modern science.

I think that with a much more connected scientific community, that operated more as a global brain, rather than relatively disconnected nodes, scientific progress could double. So if cancer would normally be cured in, x years, I can see that coming down to 1/2x years, with an accelerated science, and, given the length of x, that shortening is likely to be a matter of years.
I am running Xubuntu now on my laptop and I can tell you that it comes recommended for me. Its very light on system resources and extremely fast. I hate UNITY and not a fan of GNOME 3. Do try it out.
I think Genius is the ability to take two different ideas, merge them, and come up with a new idea: invention through prediction. Children are amazing at this until it is torn out of them via the education system, peers and adults (in general).
Yes! I have been meaning to write something like this for years.

There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files.

Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history
This is why I really like Udacity's classes. Even though I have used Python in the past, starting from scratch and building a small search engine is awesome, and bits and pieces picked up along the way are really helping my comprehension of the language.
Looks like I'm not the only one working on this. A couple of days ago I forked a Lispy language in Go called Kakapo. Yesterday I added goroutines, channels and macros.

https://github.com/ijt/kakapo
I find it telling that the only data point given in this article is the purchase of instagram and a few other debatable acquisitions. The rest is conspiracy theories against VCs and speculation.
Vivaldi Tablet - A Tablet on top of GPU/Linux, Mer and Plasma Active
There have been a lot of articles recently about how this or that corporation pays their taxes. They are always shallow analyses, just info-tainment.

There is a broader question of how the US should structure corporate taxes, but demonizing individual corporations for how they follow the crazy rules is a silly thing. Perhaps useful to further the conversation, but not really a reflection on ethics.
DNS in its current form is broken as well. It wasn't designed to be used by the current internet. Furthermore, a particular ISP can decide to use its DNS servers as a mechanism for censorship.

I think this is a little naive. Let's say you want to fix this, what could you do? Well, you could sign the responses; that would prevent hijacking, but it wouldn't stop the ISP from returning SERVFAIL, since the server could really be unavailable.

Well, then you could use another DNS server, and encrypt the queries so that the ISP has no idea what's being queried (and therefore can't do selective censoring). But now you've replaced one single point of censorship for another.

You could then send queries to multiple DNS servers simultaneously, so that no single server can censor.

Now think of all the latency added by that system, and which is useless because the ISP can wait for the DNS to resolve and then drop the HTTP request to that host.

You could make the ISP a dumb pipe by using a VPN, but now you've gone back to replacing an evil for another.

Eventually, to really fix the problem you'd need a Tor like system with anonymity, dynamic routing and hash-based addresses, but that's extremely inefficient and complete overblown for vast, vast majority of cases unless you're in a dictatorship.
A million guys walk in to a Silicon Valley bar. None of them buy anything. The bar is declared a rousing success.(Ref:http://www.quora.com/What-are-the-best-_____-walked-into-a-b...)
It's nice to see someone approaching this question with actual data, and not gawking at a handful of acquisitions that are arbitrarily deemed "excessive".
I am always looking for ideas for software that I could work on that seems more useful than cat pictures, and this gets pretty high up in my estimation. Most of my minimal NLP expertise is being used up in developing software for foreign language instruction in the near future, but I'm definitely bookmarking this just in case I can ever take it up later.
Dr. Robert Lustig explains: How Hunger and Hormones Cause Obesity
In Wikipedia though it says that "Gould’s notes included possible applications for a laser, such as spectrometry, interferometry, radar, and nuclear fusion. He continued developing the idea, and filed a patent application in April 1959."
Losing control of android could be a boon to Google, they could focus on core development more and the android ecosystem would be even more scalable.
> True. But as you said, you have to know if it's BE or LE on UTF16.

Yes, with UTF-16 you need to know not just the encoding, but also the endianness. That makes UTF-16 worse, not better.

UTF-16 is really the worst of all possible worlds: tons of wasted space with all the complexities of variable encoding without fixed endianness.
I don't much like narrative-centric journalism either, but that's basically how journalism works, mostly because that's what audiences repeatedly show they want. Imo, fundamentally most of the issues worth writing about have to do with systems more than individual actors. But the only way to analyze a system in a way that people will read is to weave a narrative through it.

I personally prefer more systems-centric journalism myself, which focuses on how economics/law/etc. interacts rather than tracing some particular person/company/story, but it rarely appears outside of specialist journals. Even in the Valley, journalists trace how the startup/technology ecosystem works by weaving tales of specific founders and startups that they hope illustrate broader principles, instead of describing those broader principles directly.
Ali Taran ile Ayşe Özyılmazelin Boşanma Nedeni
I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness?

In particular, what should I watch out for to make an ASCII parser UTF-8 clean?
Oxytocin: the trust molecule
> No it would not be a good idea ...

Did you notice I specifically said "my situations"? I wasn't talking about your situations, and it appears that the existing mechanism suits your needs. It does not suit mine and has been so problematic that I had to abandon use of submodules.
Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything on the Web), UTF-8 is actually smaller than UTF-16 even for CJK text.

Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice.

> Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors.

Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like.

So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.
No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.
As long as you don’t care about errors, converting to UTF-8 is quite fast. Just use native calls:

var decode = function (bytes) {
return decodeURIComponent(escape(bytes));
}
var encode = function (string) {
return unescape(encodeURIComponent(string));
}

(Definitely test it before wailing about benchmarks. My guess is that whatever else you’re doing is likely much slower.)

If you do care about errors, or especially if you need to deal w/ UTF-8 streams that might be chopped mid-character, use something like https://github.com/gameclosure/js.io/blob/master/packages/st...
Grails Authentication with Shiro
Arduino 100% of the way. When you're good with that, you might want to move on to ARM.
What will ‘rm -rf /’ actually do to your Linux/BSD machine?
I sympathise, but doing bitwise operations on floating point values without care is just asking for it
I personally cannot explain the problem. I think it was some sort of rule governing what bar-licensed lawyers can and cannot do or what an organization of lawyers can do. It's one of those rules that is like Rule #1 of the Screen Actors Guild. The person to ask about it is Marcel Leonardi.

Marcel is one of the few lawyers in Brazil genuinely proactive with respect to internet rights in Brazil. A few years ago he spent some time interning at the EFF in the US via a grant given to him by Google. He is now part of Google's general council in Brazil and AFAIK still works largely on issues of internet rights.

The following is his first article on the his experience at the EFF and the possibility of founding such an organization in Brazil. However this article was written before he encountered these legal hurdles. I don't know if he wrote specifically about the legal hurdles or if we simply discussed them in person over lunch.

http://leonardi.adv.br/2010/01/eff-minha-experiencia-e-as-po...

If you want to know more or want to get involved, I'd contact him, since he can better explain the legal hurdles to creating an analogous organization in Brazil.
Show HN: A Pintrest clone for code snippets (using gist)
Wow, Japan, where they worship white people.

How adventurous and original of you. Did you think of that one all on your own?
Ask HN: Books for the Beginning Hacker
What books would you all recommend? I want to find a good book that covers the basics from vocabulary to the difference between languages, etc.
This made an interesting connection between two discussion threads I've noticed on HN recently: (1) is there a bubble?, and (2) why is there so little innovation in SV?

I think the article made this claim:

Innovation is risky business but hyping a social CRUD app is less so. Hence a disproportionate amount of the money and (VC manufactured) media is focusing on the latter. And these less/non-innovative companies can only match the returns of a successful innovation based business in a bubble/greater-fool environment. So both the lack of innovation and the bubble are being caused by these financial firms.

This seems plausible to me (as an outsider). I would say more that we are seeing this as a direct result of the money looking for returns post real-estate bubble burst. Smart money doesn't seem to chase bubbles, but create them.

The lack of innovation becomes simply the opportunity cost of an investment bubble.

What's unfortunate is that I bet there are 10's maybe 100's of people like Elon Musk looking to get funding for their ideas but are being over looked by these same firms.
You're confusing things. Strings cannot be utf-8 any more than you can be your signature.

"strings" are abstract data structures. They are lists of characters. Not bytes, not integers, but characters. Often, we use the Unicode character set as the set of allowable characters. There are other character sets.

Internally, strings often represent characters as integers. When using the Unicode character set, strings then use the Unicode encoding to integers (a table mapping characters to unique numbers). Sometimes we use other character sets and encodings.

Unfortunately, integers are abstract. You can't store them in a file or transmit them over a network until you pick a concrete representation as bytes. How many bits per integer? Big or little endian? Etc. That's where UTF-8 comes into play.

UTF-8 is a merely a compressed data format used to represent a sequence of integers as a sequence of bytes - a way that happens to have some properties that make it convenient for representing strings.

UTF-8 is not Unicode.

UTF-8 can also be used for other types of numerical data. As a silly example, suppose you had a list of ages of houses. Many houses are less than 100 years old. A few are more than 300 years old. An efficient serialization of that data would be to represent the ages as integers and then utf-8 encode your list of integers.

Some true statements: A character set is a set of characters. Characters are not integers or bytes. A mapping from characters to integers is an encoding. Unicode is a standard that defines a character set and an encoding to integers. Mapping integers to bytes is confusingly also called encoding. UTF-8 is an encoding from integers to bytes. Unicode defines a set of characters and an encoding of characters to integers. UTF-8 is an encoding of integers to bytes. UTF-8 is not Unicode.
Show HN: Tabminder, a Chrome extension to (gently) prevent distracted browsing
I've been teaching myself to program over the past year, and HN has been a great resource. I started with Python, but wanted to learn some Javascript, too, and settled on writing a browser extension as a first project. I wrote this simple Chrome extension to bug me when I browse distracting sites and keep one tab from turning into ten.

I'm a big fan of tools that help manage the total time I waste online like Leechblock and StayFocusd, but I wanted a tool to remind me to spend my daily distraction quotas wisely. This extension prevents quick visits to HN and other sites from sucking up too much time by providing a gentle reminder every few minutes. Open a distracting tab, and it starts counting down from your preset time limit. When time's up, it will prompt you to close the tab or restart the timer.

This is my first browser extension (and the first real thing I've written in Javascript). I'm sure you'll find some bugs, but I hope you also find it useful. It feels great to have the ability to build the tools I want myself, instead of hoping someone else will!
Face paint to beat those pesky face detection algorithms
Me too. It's fast, but not extremely fast actually.

Previously, I use arch linux and love it. But not recomended for beginners.