skysight-inc/HackerNews huggingface dataset:
I attempted to automated my course registration process a few years ago. The registration software had a fixed limit of the number of pages you request before you get locked out.

When you are locked out, a vague message appears about "attempts exceeded." Not knowing what to do next, I went to the registration office. They quickly realized what had happened and threatened me with disciplinary action because automatically accessing the registration system violated their vague interpretation of the school conduct guidelines (I also received a free lecture about how 'spiders' are border line illegal).

After being further threatened with a "last day, last hour" registration timeslot until the sun burns out, I put on my best "I'm sorry" face and handed over a letter of apology and the 30 lines of code as they requested. They even had the courtesy to make snarky remarks about spending more time doing homework and less time trying to break their systems.

WWW::Mechanize -- so easy it's dangerous.
> OK, let's agree that this test requires that candidates rave about your shop, stores, plants, or restaurants. But what else do they say? Is it insightful, helpful, specific, and accurate? Does the candidate see problems, suggest solutions?

That's tricky. You want someone who will make your organization better, but you probably don't want someone who is going to launch every new relationship with criticism.

And given the way criticism is dealt with in many (most?) cultures, a lot of people will be trying to show they are the right kind of candidate by not criticizing. So I wonder whether this approach is going to rule out lots of good people.
Nope. I jumped in. But it's as generally related to the topic as your reply, which was about as direct a reaction to tc's post as mine is to yours.
Fravia died on Sunday, 3rd May 2009
++ for your app
Make a deal with them. It doesn't sound like they object to ClassHunt per se, rather they don't like the additional load that ClassHunt causes.

So, what do they want?
1. Reduced load and a more stable class registration system

What do you want?
1. For ClassHunt to be used across campus (and not get banned!)

What can you do to satisfy both these goals? One idea would be to pitch ClassHunt as a load-reducing service. By acting as a proxy to the class registration system you can smooth out the load. Have them actively promote ClassHunt on the class registration page and explain to them how more people using ClassHunt will result is less load for them, not more.

It will also make the situation more flexible because they only have to deal with one person (you) rather than relying on the collective behavior of the whole of campus.

Anyhow, just a thought.
.. and I am sad. A great man has passed.
Are they at all open to compromise? Where I work we have a school for special needs kids and I had a problem with getting data from the individual school districts. So what I did was to write a short program myself that did what I needed and asked them if they'd agree to run it on their servers. There was some initial hesitation but after I bought them lunch and used the opportunity to walk them through the code line by line they were willing to install it.

I'd imagine a similar system could work for you. It's a public database so you could write something that ran on a client machine (so it doesn't compromise their security) and since it's internal it wouldn't eat into their bandwidth (which is the only realistic problem I could see your program causing, one query every 15 minutes is not going to take that much processor power). Have it send you short text messages when there's a change and you're set.
Check out NTP and the stratum clocking levels:

http://en.wikipedia.org/wiki/Network_Time_Protocol

Basically, the atomic clocks themselves are considered Stratum 0 and the computers attached to them Stratum 1.
How often does this info change? Is there any reason why you must poll it so frequently?
LOL. Yeah that's the worst case option. But I'd rather not get in a cat-and-mouse game with 'em where I've to find a new IP every hr because they blocked the current one.
Yes, what I am doing is monitoring EVERY class on campus for its availability, which changes by the minute.

I can get all the classes by department, there are ~130 depts so ~130 page requests.
Definitely not. If your boxes are pegged at 100%, you have no spare capacity!

The real question is, are you CPU constrained at peak capacity? And in the case of dynamic web apps, the answer is almost always yes. So CPU improvements are a huge deal. Double your CPU efficiency, double your server efficiency.
Installous/Crackulous/etc seem like they're pretty popular. I know semi-casual users using them.
Great ideas! We'll see how receptive they are. From the legalese letter I received today it doesn't seem to encouraging.

At the same time, the contacts I spoke with at the IT dept. few days ago could not stop praising the website.
Could Wolfram Alpha Sway Google Regulators?
I think this is the right idea but it would require a lot more than he's doing now. Product reviews, support if things go wrong, added features to match what the school aleady has and so on.

I'm not saying this is a bad idea. Just that zaidf should tread carefully and not get into an agreement like this unless he's willing to put the work in to go all the way.
It's not the nicest thing in the world, but his university (UNC, as far as I can tell) has 30,000 students. 130 requests/min is the equivalent of having 0.4% of the students on at one time.

It obviously creates a visible imprint, but it shouldn't be a show-stopper for the school.
Well, you already know what the worst case scenario is (you get cut off). Anything you can negotiate will be better than that.
This might help:
http://thwart-ipa-cracks.blogspot.com/2008/11/detection.html
nginx. love it. no plans to switch. we have a crazy tricked out setup.

apache for running mod perl but that is being replaced within a year.
One other piece of advice. If the IT dept seems to love your site try pitching a compromise to them and let them pitch it to their boss. The pitch will be worth more coming from an emplooyee anyway and giving the IT dept a chance to play hero makes it worthwhile for them (which in turn makes them advocate even harder for you)
Increasing public spending when private spending goes down in a recession is kind of what Keynesian economics is all about. Higher taxes may be necessary in the future, but the idea is that private spending returns faster than it would otherwise, thereby increasing government revenue. Cutting public spending would reduce private incomes, which would exacerbate the recession and lower government revenue even further.

If the idea was to start a discussion about whether the Keynesian response to recessions is a good idea, I don't think that's going to be worthwhile unless there are some economics Ph.D.s around. It's an interesting topic, but something about the internet makes people overly confident in their uninformed opinions.
http://www.google.com/search?q=Fertility+vs.+per+capita+GDP+...

You can click on the first link - your answers are there.

Notice that the two examples you gave belong to a fairly narrow field - statistics, something that seems to be the trickiest to collect in more or less structured form. But try it yourself, you'll see how easy it is to find virtually anything nowadays. In most cases there's a good chance someone in the world thought about something you are interested in and has put it out on the Net already. If you are the first to ask a question, then I doubt WA will give you useful results either.
Well, this is my new startup I was planning to pursue over summer. The launch at UNC was meant to be a model. And my newer ideas are not really dependent on this one feature but on other ideas to make school life simpler. So yes, I will be able to dedicate resources and time if it comes to that.
Like icey said, you know the worst case scenario. They've played their hand.

But reading the letter, their only concrete objection is the fact that you're using "excessive resources." That's their pain point, so make them see that your service helps that problem, not makes it worse.
Storage
One of the best ways to do this is to visit localselfstorage.com. Here, you will get access to the most comprehensive directory of local storage service provider.
Yeah and like I said, it didn't cause a problem for over 3 weeks. And I am still not convinced it was the culprit in the last few days.

From what I've experienced and from friends who went here five years ago, their system goes down/has issues every year during end of semester. This year they seem to have a scapegoat to blame it all on...except it doesn't help to admit that they are still having issues even after blocking me.
Taxes deprive the private sector of money. The net effect of not cutting programs when tax revenue is down is that you have to raise taxes even further to make up the gap increasing depression on the very area you are trying to help grow.

California is facing a 23 billion dollar budget deficit and their bond ratings are junk. It's going to be interesting to see what they do.
Yes, getting an internal champion when you're trying to pitch an organization is very, very important. Sometimes it's very difficult, but in this case it appears that most of the staff actually like the product!
Idea is good, very original... I registered and am still waiting for the activation email (being more than 10min).
Sell them a license to run the software on their servers and query the database directly to reduce load.
Except that they would only have to deal with 1 hit/min instead of 130, and possibly on a static file instead of a one generated on the fly.
Mental Math Linked to Eye Motion, Brain Scans Show
The results for "Fertility vs. per capita GDP for continents" is significantly less useful though.

http://www.google.com/search?hl=en&safe=off&q=Fertil...
That query on Freebase, ready to compose and consume as JSON: http://tinyurl.com/on79u3
This is pretty neat to see. But this is probably the wrong metric to use to measure performance of a spell checker. In the real world you don't want to spell check writing that is 0.5 spelling mistakes per word, and many of these misspellings are not representative of how people actually misspell words. (Who would actually type "mist ache" for "mistake"?) Also, this may just reflect the fact that you have your thresholds for tagging a misspelling set lower. For good metrics you would want to have a large corpus of real documents and measure both the false positive and false negative rates.

(To be fair, the author probably knows this stuff already, and is just having fun.)
The results are kind of bogus in a lot of cases, though. For Safari, for instance, there are multiple "correct" versions (one for 10.5 users, another for 10.4 users), so uptake of a new version can't be expected to ever reach 100%. Same goes for Firefox, where some users are still using the 2.x branch: they obviously won't get a 3.x update.

Really, Chrome is at an advantage here by only having a single version. Comparing other browsers' update cycles to it is hardly fair.
"It doesn't sound like they object to ClassHunt per se, rather they don't like the additional load that ClassHunt causes."

Maybe I'm just a pessimist, but when I hear about something like this I assume they just don't like the service for one reason or another, and try to use a [usually poor] technical excuse for shutting it down.
It's not partisan differences or the system holding things back, California has a Democrat majority with the votes (24 out of 40) to push things through. Voters there are polling at a 1/3 approval rating for raising taxes. The majority just doesn't want to be the ones responsible for cutting programs or raising taxes.

http://en.wikipedia.org/wiki/California_State_Senate

In addition, their bond ratings are pretty low, they have the lowest rating of any state. The net result of that is that borrowing money becomes much more expensive and the cycle escalates.

http://articles.latimes.com/2009/feb/04/local/me-budget4

You can't run programs with deficits forever, eventually you have to raise taxes or cut funding. Neither are popular positions to have but they are necessary.
Article: "itch Rosenberg, VP of Marketing at KIVA, says that human workers are giving name tags and other identifying marks to their favorite robots. At some companies, “the robots” send you a birthday card each year. We have a hard time, it seems, avoiding anthropomorphizing and adopting robots as pets. And that’s okay. They may not be cute, but blue-collar bots are a working man’s best friend."

Well, if it's a choice between anthropomorphizing and demonizing, and given human psychology it probably is precisely that choice, anthropomorphizing is the radically better choice. Cultures that demonize robotics will shortly find that supremely maladaptive.

I've sort of worried about the demonizing undercurrent of American culture, but perhaps it will be shown merely a Hollywood-induced fad once the rubber hits the road and we all continue to deal with ever-more robots.
The title is hilarious. I read it initially as

"[Some] Hackers wanted [the] Scholarships available to coders to save journalism and democracy"
The $34'130 seems to based on the GDP per capita of canada or the USA. So 10'000 at the bell curve mid point of wages would cost 9.2 billion. Recruits in the military do get similar wages even though they work in pretty horrible conditions for example.
Useful Tools - New Web 2.0 Blog (Pretty Cool)
Sell your product to them or set a deal up where you integrate into their system. It sounds as their is a demand for your service! Take advantage of it and either be the business man yourself or find someone you know who can sell!
Was Twitter’s May 8 Downtime on Purpose?
True, because neither is the question itself very useful, and thus nobody cared to build an answer for it.
Sans-hardware (not supported until 3.0), there are a lot of apps for the iPhone that do credit card processing, including the one that's on Apple TV ad right now: http://innerfence.com
The idea is that the people who no longer offer any services that can't be performed by robots are economically useless. The only form of wealth would be ownership of resources, not labor. If that happens, the owners of resources can do whatever they want to the ex-workers.
that was his point
1. What stops the engineers from becoming obsolete? We're just further down the line.

2. Only a relatively small percentage of the population has the skills for creative scientific and technical work such as programming and engineering. (Let's say 20%.) Once robots can do all of the work of individuals 90IQ and below, what do you do with these people?

http://www.dailykos.com/storyonly/2009/5/8/02946/35811 has an example of a channenged man being fired from McDonalds.

"God, I'm so fucking angry. My brother doesn't care about the pay. He just wants the chance to be out there with people - to feel the same sort of satisfaction that all of us working stiffs experience from knowing we've contributed, however marginally, to the human machine. The kind of satisfaction that only a job can provide.

He's upstairs crying right now."

(Note: I am not suggesting that anyone is owed a job, here, but rather that there is a dignity involved with being able to contribute to society in some fashion, and that we could see instability with a lack of outlets for individuals.)

We already see the problems in Detroit with the lack of jobs and ability to get out. While some of these people could be retrained into other fields, many cannot. We have to figure out how to keep them integrated into society, or present an alternative path.

3. Perhaps we see the example from Elf Sternberg's utopia world -- he presumed that in societies with no need for human production that people would create what they enjoyed and that the system would move to a gift culture. Perhaps. Even with power, however, we're still looking at some way to ration resources...
Devil's advocate: Without knowing the details, I would observe that the load generated by searches can be very different. Searching for all 100-level Physics classes may be an optimized code path (or DB query) that occurs in a fraction of a second, where as "Give me all classes in a department starting with P" might be slow, or "Give me all courses taught by Prof. Bill" might go down an unindexed code path. They might never have noticed if normal users only fired it only once per hour, but if you happen to be using more of those queries than a normal student because it's more convenient for you, then you might actually be hammering their system without meaning to.

Or, put another way, "queries per second" is a terrible metric for "how much load I'm subjecting their system to". :)

I can give you one bound you might want to instrument up to check their contention about load: How quickly they answer. If they answer in 1ms (minus ping time), then there's a sharp bound on how much of their resources you can be taking up. (I'm going to assume bandwidth isn't much of a concern here, based on the numbers.) If you see some queries taking longer, perhaps even into the seconds, then they may have a point. It could just be general load, or it could be your thing specifically.
There is already a large and growing creative class. There is also a sizable and growing leisure class that doesn't need to work full time to make ends meet.

I think the myth here is that there is a tiny cabal of people that would control everything. They wouldn't be tiny: they would be public companies with many, many thousands of shareholders.
I'm the author: There are two types of spelling errors, real word and non-real word errors.

A non-real word error is what most spellcheckers look for. They check if a word is in a dictionary and if it isn't they generate a set of potential words from that error and then rack and stack them. AtD's non-real word spelling corrector uses a dictionary gleaned from the intersection of many wordlists and a corpus of text. The rating of each correction is handled by neural networks. It's all fun and I have carefully measured, tweaked, and remeasured this part of the technology. I also reproduced an experiment conducted by some Polish researchers in 2005 and used their data to find out how I compare to Office 2003. I learned my dictionary needs more work but my accuracy is on par with Office here.

The real word errors are just as serious but somehow ignored by most spelling correctors. A real word error is where you use a word that is in the dictionary but really meant some other word. You can imagine why this is difficult. I use a statistical model to try to pick the best word from pre-made confusion sets. While I was impressed with the accuracy of this I had to bias heavily against false positives making it less than what I'd like to see. I'm currently working to have the grammar checker take over or augment some words where this technology does poorly. That said, this poem is all about the real word detection and correction and I was quite delighted to see how well my code did against it anyways (even after I bias heavily against false positives).

So in short, yeah I'm just having fun.
Where did that 1hit/min come from? Does the FOIA have guidelines about that?

(I ask for information, not as criticism.)
MIT students win grant to study hot air
I meant regular in the the industrial manufacturing sense. There are, at least, a few different kinds of boxes - and they are made of a warpable material. That means you need to perceive the environment, no just assume you know it. That makes it hard.
"Founder and CEO Mick Mountz experienced the challenges of existing material handling systems firsthand while working at online grocer Webvan."

I wonder how many other dotcom flameouts went on to rethink the expansiveness of their plans, and started solving individual problems that caused their failure. Clearly, webvan in 2009 would stand a better chance than webvan 1999.
How Microsoft will shake up the security industry
Relationships Matter – A Lot
If that's the case my advice would be to start putting the effort in now to write a whole registration system. Because even if you convince this school to go along with you there's a good chance the majority of schools won't be so permitting (they are after all beuracracies)

Better to dive in now and have a solution for them rather than jump in with the hope that every school will be willing to put up with you.
There's days that I wish that people like Ray Kurzweil are correct about the Singularity. When that day comes, what we cherish the most will no longer be lost unwillingly.
Security Risks on Your Macintosh
Not sure how well thought out this decision is considering that SHA-1 has resisted preimage attacks for almost a decade longer than the SHA-2 family (since it is that much older).

Re-keying hundreds of developers as well as the entire Debian infrastructure seems like a pretty extreme reaction to a problem that doesn't actually exist.
I don't know, I could imagine them being somewhat serious about the license/number of connections violation.

Speaking from direct experience, I wouldn't underestimate the amount of bureaucracy internally that would come down on this like a ton of bricks. Like others have mentioned, his best bet is to get someone to champion his cause internally. It doesn't even necessarily have to be someone in their ITS department - it could be a faculty member, or even someone in administration.

Hell, you might be able to get a job out of this.
The plow (and other technological advances) did make the farmer (relatively) obsolete. The great majority of farmers have gone out of business in the last hundred years or so. That's why there are so few farmers now.

The former farmers found jobs because plows can't work at factories, call centers, KFC, etc. Sufficiently smart robots could. That's why analogies with previous technological advances don't work.

A more relevant comparison would be with horses and machines that replaced them pulling plows, hauling stuff, etc. Where are all the horses, mules and oxen now? Were they freed from unnecessary drudgery to specialize in labor where they had a competitive advantage, or did they suffer severe declines in number, surviving essentially as pets and museum pieces?
Why discount the effort before you try? The worst that happens is they don't buy the pitch, and he's back where he is now.
I have to imagine that Dorsey knows this which makes me wonder if he has something up his sleeve, unless he thinks that his reputation in the business will help carry the project. This is, after all, the tech industry; we shouldn't underestimate the viral.
University Makes iPods, iPhones Mandatory for Freshmen
Manage your Netflix queue via Twitter
The Million Dollar Programming Prize
They seem to have taken down the URL in the submission but you can still find it here: http://www.pressdemocrat.com/article/20090508/ARTICLES/90507...
The Kindle DX: Is this the thing Apple is supposed to be working on?
I would be very surprised if there were a tractable preimage attack against either algorithm in the next hundred years.
Macs with 3G: More connectivity, more problems
StorageMart
Secret shopping or mystery shopping is the only means of looking into the performance of your business, especially your employees. There are elaborate feedback forms that businesses of all kinds use to get feedback by customers.
Isn't a credit card app basically B2B? I don't see how a reputation in consumer internet is going to help.

I think he has something more up his sleeve.
Mapping Ports and PIDs
I think what he means is if they make a custom page, they can provide ALL classes data for ALL depts on one page...where as right now I am having to query 130 different pages to get class data from 130 departments.