skysight-inc/HackerNews huggingface dataset:
This usually depends heavily on your language/framework. Coming from a ruby/rails background, I can recommend devise.
There is something you can do about it, but it's expensive and different from what you are used to. Safer languages is a good place to start.

Input validation can be made to be machine-verified, and anit-patterns can be learned and avoided. Static analysis tools can raise red and yellow flags.

Finally, frameworks are being built for handling security critical things like account validation, crypto key handling, an related tasks that are important to get right.

As the space continues to evolve, things like this will become less common, and someone who is aware of the trade and is diligent will be less likely to make mistakes.
All the problems I've solved seem about equally hard, since I put my full effort into solving them :) I tend to go into new subfields where I don't have a background, so the hardness is probably just proportional to the length of time I spent on the project.

The problems I failed to solve seem are the ones that seem the hardest, of course. I tried to write a cluster manager / distributed OS by myself, starting almost from scratch, and that was too much. I spent upwards of 4 years on it, and had some success, but I'm starting to move on.

In particular, I learned that having a reasonable amount of security with reasonable amount of development effort in a distributed system is still an unsolved problem. It's basically a bottomless pit of work.
All of my machines are linux, so I like to use ansible for it. I can sync my dotfile git repos, configure the OS how I need it, automate syncing vim plugins, get all of the PPAs I need. This doesn't quite solve the problem of half-finished work on one machine being available on a different machine (unless git counts), but at least your systems are now set up identically everywhere you can. One option is to work remoted into a VM somewhere in the cloud very close to you (to minimize latency), I've heard of people doing that.
Depending on which kind of "hard" you mean. Some problems are straightforward but very involved to fix, while others are incredibly difficult to investigate but easy to fix once found.

My first significant contribution to FOSS was to port OpenOffice.org to work without the then-proprietary Java, so that it could go into Debian main (and other distros with similar requirements). At the time, OO.o took 8 hours to build, or 3 hours with the wonders of ccache, and I was hacking on the build system itself, so incremental builds were often broken. (And the first thing OO.o built was its own implementation of make.) So over the course of a month or so, I would hack on it, rebuild to see it get a bit further, and repeat until it finally built without error. The net result was dozens of patches submitted and merged into Debian and ooo-build, and the 1.1.0-2 changelog entry listed here, which made it all worth it: http://metadata.ftp-master.debian.org/changelogs/main/libr/l... ('The "Wohoo-we-are-going-to-main" release')

The most challenging problems were two different mysterious crashes in BITS (biosbits.org), a Python environment running at the firmware level. Because of the environment, a crash means a sudden unexplained reboot, with no diagnostic information.

First, I was trying to debug a crash in the initial CPU bringup code, which brought the CPU from 16-bit real mode to 32-bit mode. After extensive investigation, including assembly output of characters to the serial port to indicate how far the code got, and hand-comparison of disassembled code with the original, it finally turned out to be a bug in the GNU assembler, mis-assembling an expression with a forward-referenced symbol when in .intel_syntax mode. The forward reference ended up becoming an unresolved relocation (with a 0 placeholder) instead of the intended compile-time constant, resulting in a wild pointer. It was one of the rare instances where the bug really was in the toolchain, combined with an environment that makes debugging a challenge.

The other such bug, in the 64-bit version of the same environment, involved GCC compiling struct assignments into SSE instructions that assume aligned addresses, and GRUB not actually aligning its stack for SSE because it never actually used SSE itself and didn't happen to use struct assignments. Debugging that one involved a quick hack of a general-protection-fault handler that hex-dumped the bytes of code around the instruction pointer, searching for those bytes in the compiled code, and matching that back up with the disassembly and source code.

Most recently, I debugged a race condition in a build system, where disk image manipulation (done by syslinux and mtools) was failing to obtain an flock file lock. The kernel doesn't actually have any way to find out who holds the lock, so I ended up instrumenting the flock syscall to print the conflicting lock holder. Turns out that udev took a file lock on the loopback device as soon as it showed up.
A special graph dissection algorithm. It took me 3 months. >.<
Yes, that is part of it. I pretty much abandoned the thought of someone being wrong and someone being right. We all have this bias, that we are more intelligent then others. While on average, we are just average.
yep. when I stumbled on this I was all yeyy. Then I saw the status and was all "sigh". That led me to custom URL protocols which turned out to be a deadend
If I (somehow) got hired at Valve, I would dedicate myself to fixing Steam's deeply shitty UI. Just off the top of my head, trailer videos on Steam don't buffer more than a few seconds ahead unless you watch the entire video, and it is literally impossible to return to the exact beginning of a video without first switching away from it (you can only navigate by clicking--not dragging--the progress slider, and it lacks the resolution to go back earlier than the first second or two). And of course it doesn't recognize universal-standard keyboard commands, like Space to play/pause and arrows to navigate. They only recently let you play/pause by clicking on the video instead of on the tiny button.

Trailers for pre-Greenlight games are actually better, because they're forced to use YouTube since Valve isn't willing to invest streaming video bandwidth in them yet. It's wretched.
Very interesting. I don't suppose the hosting-platform is going to be released as open source too? Granted, self-hosting an image that also work with the pharo-cloud shouldn't be too hard -- but it's always nice to have the option to easily move hosting "in-house" if that is needed. Would be nice to just be able to update the post-url and tokens, and keep using the same tooling for that use-case.
This is precisely the sort of error that Fuzz testing should have uncovered.
Debugging can be hard sometimes, too. Here is a Quora topic about it:

http://www.quora.com/Whats-the-hardest-bug-youve-debugged

My favourite answers:

Crash Bandicoot: http://www.quora.com/Whats-the-hardest-bug-youve-debugged/an...

Flash Player: http://www.quora.com/Whats-the-hardest-bug-youve-debugged/an...

500-miles email: http://www.ibiblio.org/harris/500milemail.html
I suspect that they are a lot easier when you are primed with the idea that they are trick questions.
http://waiter.com has made a few cool megabucks in SV.

The original was http://waitersonwheels.com, which I used in the 1990's. It started as a catalog-based service, which waiter.com knocked off and improved.

It's not a defensible business model unless there's some exclusivity agreements with restaurants on the menu. (too tasty to resist.)
Genuinly curious: I thought you can't represent ellipses with bezier curves, at least accurately... What did you do?
but it would not be accurate, i was thinking if there was a way that the attendees could actually check it and mark themselves as present to the event or place without having to be counted by someone, all the organiser has to do is sit and watch people check in.
It was challenging in every way imaginable. There was no existing algorithm that could deliver the accuracy and robustness we required. It had to run within the limited power budget of a phone. And it had to be done quickly before the hardware became uncompetitive.

At the time it was given to me it was a rough demo with no clear path forward to shipping. We had no metrics to tell how good it was, how good it had to be, or whether we were even making progress. We had no team of computer vision experts to work on core algorithms. We had no idea if the problem was solvable at any amount of power consumption. There were more than a few people within the company who thought it couldn't be done.

I want to be very clear about credit. I put this as the hardest thing I have ever done but I was only the manager in charge of the project. While I built the team and owned the problem, I did not write the code or design the algorithms. I had incredible people who did outstanding engineering work and researchers who advanced the boundaries of computer vision. It was a privilege to work with them and I am proud of them.
Register allocator.
It's clear the problem involves some PHP sites developed with git and instead of using a specific www directory inside the project the server points to the root folder of the project thus exposing .git (and the rest). Classic dumb error by PHP developers. I have hard time believing one would be able to expose the .git folder in a Rails,Spring or Django application since the public folder isn't the root folder of the project.

I wish servers would be configured so they don't server ^\..+$ files by default. I wish servers would behave as secure as possible then it's up to the developer to whitelist features rather than the other way around.
Swaping ssd with OS and data.
Wow, that sounds really cool!
What parts did you use (how much was "from scratch")?
Any blog post or article about this? I'd love to read more.
Hardest technical problem would have to be my PhD. I worked out how to apply alias analysis to PHP (previously people had figured out how to do it for the easy subset of PHP - I extended it to the entire language of PHP 5.3 or so). [1]

Along the way I'm pretty sure I also figured out how to build SSA form such that you have your alias analysis results available to be used at SSA construction and therefore redundant computation [2]. I never got to chase that down but it was really interesting.

[1] http://paulbiggar.com/research/#phd-dissertation, esp chapter 6.

[2] http://paulbiggar.com/research/#fit-2009
I was part of a team that designed and implemented a P2P UDP based video streaming protocol, that receives chunks of a stream simultaneously from up to hundreds of peers. It wasn't a "big problem" per se - I worked on seemingly harder problems before and since - but this one was really hard to get right, it was a very non deterministic beast that was extremely hard to test. I remember lots of times that I secretly felt like we would never get this thing working as well as we wanted, but in the end we did.

I left this company long ago but they appear to be going strong still. http://www.giraffic.com/ . I'm sure they improved on that work a lot since then.
Three Mysterious Philanthropists Fund Fourth-Largest U.S. Charity
Clearly, when I think about the hardest thing I ever coded, I have the following story in mind.

Back in 2002 I was writing a floppy disk driver for the little OS we were writing with a friend. It turned out finding anything else than very sparse documentation was really hard, plus for some unknown reason the floppy drive behavior seemed to be of non-deterministic nature. Maybe the fact that I was 15 didn't help.

At some point, after many nights spent on debugging it, it just worked. I still don't know why. I never changed any line of the code after that moment, by fear of breaking it.
One thing I like about keeping reset tokens in the database is that you basically get a free audit log (if implemented properly).

Some other disadvantages were pointed out in the previous discussion[1].

[1]: https://news.ycombinator.com/item?id=9053830
You'd use a data(base) modeling tool which creates the DDL script - including comments - for you.
thank you! This definitely seems to open up a path that I can look into. As for custom protocol handlers, their experience is pretty messy. But well, try and use what you've got eh.
Technology firm owner shares $27M to employees after sale
It's still not cross platform. Changes made to the Makefile under one OS might break easily under the other OS's.
While I've had some weird technical problems (I've worn a number of hats across Network/System Admin, to both front and back end web Development) the hardest is always the non-technical issues.

A few years ago I was contracting for a company that had a Native American Casino as a client. They wanted to build a gamified app/site to engage their customers more.

The single hardest problem was trying to look at the situation from the players point of view. Gambling like this (slot machines) is inherently an illogical thing to do - they know they're never going to make back the money they put in, but they walk away with a smile night after night.

Trying to rationalise (so we could understand their goals and what they might want out of an app/site targeted at them as players) proved impossible for basically everyone on the team.
Similar stories where company owners are being generous and/or equal to their employees are becoming not too remotely uncommon.

Not long ago something similar had happened: http://www.independent.co.uk/news/world/americas/seattle-ceo...
EmDrive independent test
Probably yes, but the point is that if you've seen similar problems before, the fact that they are counterintuitive trick questions is part of your pattern-match, and so you jump to a problem-solving strategy that manipulates the data you're provided in a way that's accurate rather than one that's not.

It's the same way you build expertise in any domain: proceed with your initial hypothesis, but then once experience proves you wrong, build a new, more accurate mental model. That new model will give right answers in a greater percentage of cases, but inevitably it's wrong too, and so you repeat the process.
I usually find architecture problems to be the hardest to solve. The hardest one I've had to deal with is taking a legacy web application ~3 million lines of code and giving it some form of architecture so the product can have a sustainable future. Some of the issues included inline CSS styles, Core logic written in linear Classic ASP, ASP Web Forms written in a linear fashion and so on. As you can guess, what made it hard was trying to solve these issues without breaking anything, this is an obvious example of why automated testing and code quality are so important.
Read the material, skip the code, write your own code in Ruby, that's what I've been doing in Clojure.
A simulator gave different results depending on the length of the filename of the executable under test. Only when I ran it on two separate but identical machines with different host names (I used the hostname in the file path to keep a record of where things were run) did I discover the cause of the differences in results I was seeing, which was due to the way that syscalls were handled by the simulator.
What is startup365?
All I found about them was:
Startup365 is an American TV show based in Detroit, Michigan. The first series started filming in 2010. The TV format was created to help startups from around the world spread their name and create awareness of new and exciting technological innovations. The program serves to give exposure to young companies, entrepreneurs and their ideas. It has facilitated links between investors and developers. The program proved a big hit in hi-tech circles and enjoys a particularly strong following among computer science students and those with a keen interest in new technological innovations.
The first show was filmed on September 23rd 2010 in Detroit, Michigan, hosted by Cooper Washington. Carl Dennison is production manager and Lisa Delaney serves as executive producer. Both been with the show since the very start. Other key senior film crew includes Josh Silverman as a director and Emma Waterson (started in the 2nd season) as senior tech adviser.
STARTUP365 well known for its annual competition to discover the best new startups from around the world.
Have a look at: https://github.com/StackExchange/blackbox
Oh, that makes more sense. Sorry I got confused by the ascii tree.
> Turns out their user creation code used /dev/random, and the system was running out of entropy and so the code was blocking. Moving the mouse or typing on the keyboard would add entropy, thus cause the tests to resume.

Funny how this goes completely against the typical operant conditioning a user undergoes when working with computers. Usually if your software hangs up, you want to touch nothing and let it finish. But in this case it's actually additional user activity that's needed.
I tried running a home server with data, running a home server and forwarding X from there over ssh, limiting my needs to CLI (tmux, irssi, vim, etc) and running everything on a remote server, using built-in sync for various apps (like notes)...

None of it ever worked well so I bought a nice leather messenger bag and learned to like carrying my laptop around and using the small (13") screen most of the time.
> Java

You'd be surprised - see JavaCard.
Laymen tend to underestimate the very vast body of research which supports current physical models. "New physics" would imply that something has been fundamentally wrong with physics for a long time, and that is unlikely.
Yeah, strcmp() exists for a reason!
I still think Snapchat is a dumb idea. Maybe well executed? I tried myself to start using it, but everytime I install it, I'll uninstall after few minutes..
I have a MBP, desktop (Xubuntu), netbook I rarely use (Xubuntu) and Android phone + personal webspace (just use it for a simple wordpress and owncloud+mail accounts). My setup is pretty consumer/not very sophisticated. I recently removed my last Windows VM but will probably create a new one soonish because there's the occasional windows only thing I need to do. That VM is just kept up to date and has antivirus etc. but no synching with the other machines (shared folder with the host)

- GitHub/GitLab

- owncloud with personal "webspace" for data, especially important for Android (+ a photobucket account for images for random funny forum posts)

- Zotero for Firefox (with said owncloud) for my academic work. This is worth its weight in gold. I prefer to work with Latex but you are often forced to use Word/LO and the plugins are amazing (instant citation/bibliography; snapswitch to a different format).

- Thunderbird (and Outlook on OSX, meh but needed/convenient for work would prefer 100% TB)

- Firefox sync (not really using it anymore but probably should)

- KeePass to store my passwords across systems

The overlap between the machines isn't huge and I mostly just keep them up to date. The stuff I use on multiple machines is on the same version if possible but not automated in any way.
My personal backup strategy is pretty horrible (I just randomly put important stuff on external storage), I need to up my game there.
Not even keeping my configuration files etc. on github as of now (planned for the near future)
tl;dr: Need better backups, need scripted os sync

Note: I don't use my phone all that much for stuff I can do on other machines (don't mobile browse a lot). It's mostly an IM/call platform with occasional data use.
> So, in your using TIOBE as a baseline, are you suggesting that Java should be the language of choice for embedded platforms?

Many do by way of Java ME[1], though that would not be my first choice. As I said regarding your premise of a business case:

> > From a business perspective it makes a lot of sense to use languages that allow for as broad a hiring scope as possible with the least amount of cross/re-training.

> By your own argument, then, the most sensible languages to choose from are those which have been used in embedded systems for decades. Assembler, C, sometimes C++, and Forth.

The vast, vast, majority of embedded systems are written in one of the aforementioned four languages (with a nod to Java ME since I do not have metrics regarding its production use in embedded systems). There exists a huge software world outside of the browser.

> From their home page: "It is important to note that the TIOBE index is not about ... the language in which most lines of code have been written."

Reviewing my earlier comment revealed that I forgot to include the referenced link. It has been updated to help disambiguate that post.

From it, there is a definition[2] of the "Ratings" I quoted:

The ratings are calculated by counting hits of
the most popular search engines. The search query
that is used is

+"<language> programming"
The number of hits determines the ratings of a
language.

...

The ratings are calculated with the following formula:

((hits(PL,SE1)/hits(SE1) + ... + hits(PL,SEn)/hits(SEn))/n
where n is the number of search engines used.

By this definition, their "Ratings" are not based on LoC.

> MOST software development these days has a web component... not all, but most... that means JavaScript...

Giving the largest room for expression, I'll interpret "a web component" as software having some form of networking capacity. With that caveat, sure, network communication is very prevalent.

However, assuming this implies JavaScript is a straw man argument[3]. Much network communication exists outside the browser as well.

> Node was developed precisely because JS makes a good domain language for interfacing with lower-level libraries.

I disagree with this, but respect your right to have that (and any other) opinion. In an attempt to keep from going completely off topic, I won't go into this anymore in this thread :-).

1 - http://www.oracle.com/technetwork/java/embedded/javame/index...

2 - http://www.tiobe.com/index.php/content/paperinfo/tpci/progra...

3 - https://en.wikipedia.org/wiki/Straw_man
If you're talking only about coding (and not other life challenges), the hardest problem I've solved so far has been figuring out how to build https://recent.io/ with my co-founder. Recommendation engine, fetchers, iOS app, Android app, etc.

There's a very big difference between concept and working code. :)
The article mentions “capitol crimes”. Do they need smarter spell checkers or have I missed something?
I got tired of carrying a laptop around so I get a smaller laptop.

You sound like you would do well with that idea, but maybe it takes the form of an external (e.g. USB) storage device that you store virtual machines on but don't forget about backups. Perhaps you have a script snapshot and rsync diffs between your storage and your machine so that each machine's local storage can also be your backup.
Dawn of the Super Athlete
Of course interpreter always eats up more cycles (so, more power) than a native code. Does not affect much peak consumption, but drains battery integrally.

Anyway, there is no point at all (zero, full stop) in using a dynamic language in embedded, where all your code is fixed and rarely updated, with no runtime metaprogramming (eval and such).

And, sorry, but there are far better uses for the L2 cache than keeping any fat interpreter there. I would have accepted a cache locality argument for a direct threaded code (and that's the reason why Forth rocks more than C in embedded), but not the Lua kind of interpreter.
> I build my software targeting Debian Stable + Backports. At FOSDEM I noticed that some people consider it uncool. I was perplexed.

I am also perplexed... what are these other people doing?
Finding the Green's function for a nasty differential-difference equation. Great stuff, but I feel sooo embarrassed thinking back on how I attacked this kind of problem back then.
Espruino – JavaScript for Microcontrollers