skysight-inc/HackerNews huggingface dataset:
TOPIC: CUSTOM DATABASE STRUCTURE IN RAM?

ig1 said: "I've worked on a number of high volume systems (million+ client interactions/minute), and you don't want a conventional database. Either use a custom data-structure to keep it in memory (even if it's across multiple machines) or if you really want to use a database use one thats designed for that kind of usage (think tickerplant databases, kx, etc.)"

my reply: Others have suggested similar alternatives to using a traditional database program, and right now I'm tempted to learn more about doing this -- unless someone can tell me why I shouldn't?

By the way, I've never heard of ticker plant db's before but I'll look into them if others think they may provide an even better soution than a custom coded C application. For those of you who have used ticker plans db's, what do you think of their use in my situation?
Topic: MY EMAIL ADDRESS, TOP OF THE PAGE

lallysingh said: "If you're doing something game-ish, talk to me privately. Scalability of video games is my phd topic."

my reply: It certainly seems like I'm doing something 'game-ish' although it's not likely to be used by the typical gamer. I've posted my email at the top of the page so please send me an email and we can discuss the details privately. Others can email me privately as well to discuss this or other topics, thanks.
Topic: ONE OR MORE SERVERS FOR THE DATA?

ig1 said: "Figure out how to partition your data/algorithms so you can split it across multiple machines..."

but lallysingh said: "A single dell box could handle your load if you wrote the whole thing in C on a modern quad-core desktop."

my reply: If I'm not mistaken you guys have a difference of opinion here regarding the ability of the data to be processed on a single machine. Can you expand on the reasons for your apparent difference of opinions?
Topic: LOAD TESTING DURING DEVELOPMENT

owkaye (that's me) said: "This is not a system I can grow into, it must be capable of this performance from the very beginning."

then ericb said: "I would suggest you load test extensively and make load testing a part of your development process from the get-go. Initially, I would load test to evaluate approaches and estimate hardware needs.

and lallysingh said: "Build a load simulator. A raw mode that just sends over the handful of bytes, and a cooked mode that bothers to printf' a GET request."

my reply: I don't know how to do this (or C coding or javascript coding) personally but thanks for pointing out these technical needs and more. Your suggestions will be put to good use when I have a complete concept of the basic requirements, and I can "fill in the blanks" by hiring programmers and buying equipment as I go along.
Topic: GENERAL PLAN OF ACTION

lallysingh said: "So, for advice:

1. Ignore database bullshit. You don't need it, it won't help. If you want a DB for other purposes, fine. A snapshot process writing to your DB is fine, just don't put it in the critical path.

2. Build a load simulator. A raw mode that just sends over the handful of bytes, and a cooked mode that bothers to printf' a GET request.

3. Start with a reasonable prototype, and work your way to something performant. Hell, you can probably do it in java if you don't mind buying a few more CPU cores.

4. Integrate as you need for the rest of your requirements. For example, have another box serve the rest of your webapp, and dedicate a stripped down apache box with a custom module for this stuff.

In essence, I'm telling you to treat it as a very smallish HPC problem, instead of some sort of nightmare webapp problem. It fits better, and suddenly you have lots of people/knowledge/COTS equipment available to you."

my reply: Thanks for your outline of the best way to approach this project. I don't understand parts of what you said here but I think I get the general idea. Do others have any suggestions to add to this, or to change it?
NOTICE ! THIS THREAD IS BEING MOVED !!!

I'M GOING TO START A NEW THREAD TO CARRY ON WHERE THIS ONE LEAVES OFF SINCE THINGS HAVE BECOME SUCH A MESS IN THIS THREAD ... AND I'M SHOUTING TO MAKE SURE EVERYONE KNOW ABOUT THIS SO THEY CAN FIND AND PARTICIPATE IN THE NEW, CLEANER AND BETTER ORGANIZED THREAD.

Thanks and sorry for shouting, here's the new thread URL:

http://news.ycombinator.com/item?id=362810
Maybe Giz, but it eats up a lot of time though. I should just finish out my web apps. Already hearing some sites that need writers to establish themselves as the next VW.
Go the other way. Assume two strings of length k: string a composed of english words, and string b generated by reading /dev/random. String a will be more compressible, length(compress(a)) < k. You can now add MORE "information" to a and compress it until length(compress(a)) = k. Since you can't compress b any further, it already contains the maximum amount of information.

I put "information" in quotes above because the actual "information per bit" of English is pretty low, and that's where the ability to both compress it and comprehend it, not as single bits but as groups of bits, comes from. Still, compressed data is still comprehensible once you uncompress it, so it's a measure of the surface comprehensiblity. It's really a measure of information density. Compressed data has a high information to space ratio, whereas random, uncompressible data has low information to space ratio (or perhaps negative).

You can experiment with this yourself:

cat > /tmp/string.a
(paste in some English text cut and pasted from a web page)
dd if=/dev/urandom of=/tmp/string.b bs=1 count=`wc -c < /tmp/string.a`
bzip2 /tmp/string.*
ls -l /tmp/string.*

Experiment with the above for corpora of different lengths. You'll see that as there is more English text, which is information rich, it can compress better than shorter English text (as a compression ratio), and will compress better than random data (which we know contains little information) of the same length.

A string composed solely of 27 As would compress down to perhaps 2 bytes or less (not including the size of the decompressor). You are right: there is not much information in it. Less than 16 bits of information in 27 As.
Best days of your life lyrics by Pickler Kellie
Holding and semi-holding Vs. non-holding Companies | An Entrepreneur Daily Thoughts
The VC Model is Broken: Ideas, Anyone?
Plop was the name of another humor comic magazine along the lines of Mad and Cracked. It was on newsstands in the very early 80s.
Tony Hsieh: CEO of Zappos on Culture and Customer Service [video]
MySpace Finally Completes Mobile Trifecta with BlackBerry App
I might die.
So, its genetic programming.. right?
I'm looking for a definition of information that matches up with how we use the word in normal parlance. This definition has to strike some kind of medium between incompressible and completely compressible. I could make something up, but I was wondering what the official version is.
It's interesting to me that the spammers can recover their operations in merely a few days. Few organisations I can think of could move their entire infrastructure to another provide from a hard shutdown in that amount of time.

I also find it surprising that the obvious barrier exist to shutting these operations down. Since they seems to have a handful of these hosts shut down a year the impact on (illegal) spammers is minimal given their turn around time to redeploy on a new host. That is a sad failure in the legal system. I'd at least like to see the Western world have the legal infrastructure to push this stuff offshore to China, etc.
What jurisdiction will the auction be under - i.e. In most countries auctions are governed by various laws.

I worked at an online, realtime auction startup in the UK in '99 (they folded) - and they encountered a whole raft of legal challenges.

If so, this might influence the design. You might be in a legal predicament if (a) the system coredumps or (b) someone sues and you don't have a log of each bid or (c) both.

This might be a challenge for your in-memory approach. You might need to keep a redo log that can be used to replay/verify the auction after the fact.

EDIT: ps. The startup I was at used this approach. In memory, with redo logs. The client was a Java Applet - JavaScript wasn't an option in those days.

Is it real money changing hands? If so you need to be concerned about security. e.g. You might need SSL or another cryptography technique. HTTPS should be fine with the JavaScript approach, but it will be a big whallop on your server requirements... You might want to consider an edge device that has SSL acceleration.
Neat. Is there any cheap "commercial" ones? If I could delegate the "it-won't-kill-you" testing and pay double, I might go for that.

What I really want though, is that 100x100 tongue point video camera that Hawkins mentions in "On Intelligence".
Well you look at probable revenues, & guess.

Say facebook can bust up their % of the online display ad market. Now I believe this is a relatively arbitrary market. There are budgets. A certain amount is allocated to online ads. Their effectiveness will never be like Google's. In fact, Google has nothing to do with this. Search ads are different.

Google is like hotel booking sites or product comparison sites or Yellow Pages or anyone else who controls what item people find within a category. YP controls(ed) what businesses get found, hotel sites (and travel agents before them), what hotels get found & Google contols which sites get found.

Linkedin controls what business people get found, so it's different. Facebook controls which people get found.

There's no business model there. If you want to get a business model out of that you'd need there to be people willing to pay for friends. I doubt there is. Facebook just isn't part of this game. That's unfortunate because it is a more elegant & valuable game then the 'display advertising' but there you go.

So let's get to the business of guessing. What can happen if we think positive?

- The display advertising market can just grow buy itself. Budgets get bigger, their 1.1% is worth more. Likelihood? I'd say 60/40 for (just guessing). But how much bigger? I'd guess not much. Definitely not more then 200% 300%. There's also a chance it'll get worse. Especially this coming year.

- Someone else may figure out how to make these more effective. Google has a lot invested in generalist 'impressions.' So do a lot of other smart companies. They may figure out a way to make direct response display ads work. But they haven't yet, and that's telling. What are the chances? I guess I'd say under 50/50.

- Facebook will figure it out for themselves. They have the bulk to have a go. They might find a way to leverage data & connections between people & maybe smart advertisers might figure out cool things to do with that given some cool tools. I can think of scenarios: A 1% very interconnected segment of Texas knows all about some new energy drink. it seems to them that everyone knows about it. Everyone seems to be drinking it. But outside their circle, noone is.

- maybe. This last one is the wildcard. I don't think the chances of pulling it off are great. If they do there is potential upside. But how much? I dunno. I can't see them ever pulling in the $5b per year revenue. There is doubt as to whether they'll every hit $500m, but they probably can.

So what are they worth at $250m revenue what are they worth at $5b. The chance of maintaining at least $250m I say are at least 90% the chances of hitting 500m are 70%, 1b>> 50% 5b >> 10%. That's just guessing & I think I'm being generous. What are you willing to pay now for facebook with those numbers?
Hmm, or maybe not. http://science.howstuffworks.com/brainport2.htm That picture's not really sellin' it.
I'm not sure I follow. The definition of "information" I use everyday matches the "official" version -- I'm not sure how it could be different. What is "completely compressible"? Something can not be compressed beyond the shortest string that can represent it without losing information content.

http://en.wikipedia.org/wiki/Information#Measuring_informati...
Wait, meaning other sites are trying, or other sites are writing you asking for you on their staff?
Legal and financial issues are outside the topic of this thread, but I will say this:

If the auction proceeds to completion the RAM data is written to disk. This file becomes the 'auction log'. If the auction fails to complete because of technical issues we simply purge the data from RAM, re-schedule the auction and try again later.

There's no need to write data to disk during the auction itself because of this 'try again' policy if anything screws up. I also think this eliminates the need for a separate re-do log since each bid can be time-stamped and stored in the auction log that's written after the auction ends -- but this brings up a question:

Can timestamps be created to the nearest 1/100 of a second?
You also have to address these questions with people you've known for a long time. They're just easier to address with people you've known for a long time :-)
I don't think the startup party is over: I think the tourists have gone home. HN has taken off as an extremely useful community for serious entrepreneurs. And based on the vast majority of submissions and comments that I read, "most people here" want serious conversation.
Fair enough - just something to be careful of.

In the industry/jurisdiction I was working in, this was an issue - the view was that there was a liability if the system crashed. Perhaps in your case this doesn't apply or doesn't matter (if they are low cost items, you might just give them away for example).

Anyway. In our case a bid didn't apply until the redo log had been written. So it was pseudo-transactional in that sense.
Stop the OCLC powergrab
> Can timestamps be created to the nearest 1/100 of a second?

Sure - but you need to be careful of the hardware and operating system involved. Windows tends to give lower accuracy - but there are builds of Linux that are the same.
Totally agree–most people I've known for a long time would still make bad co-founders.

I guess I just feel like some things can't be known quickly, including possibly #6 on his list and other things he didn't list.
Thanks mleonhard, I've created a new thread to continue this discussion since this one has become way too disorganized, so please post future replies here:

http://news.ycombinator.com/item?id=362810
it's an auction based on a gaming concept, more details here:

http://news.ycombinator.com/item?id=362810
I second that, pcre.

It's a horrible horrible API but you can free-ride features like UTF-8 and Unicode support.
I've created a (hopefully) more organized thread with some details this one does not have. Please go to this thread to continue this discussion, thanks:

http://news.ycombinator.com/item?id=362810
> [TechCrunch] a supposed technology website [doing things] that make me want to cry
If we don't know whether the universe is continuous or discrete, how can we know how much digital information is in a raindrop? If the universe is continuous, the distance between any two particles can be measured to an infinitely more fine-grained level, yielding more information.
WAV is more appropriately the audio version of BMP: both try to store all the data they know about.

MP3 is the audio version of JPEG: both can be set to keep to a certain standard of quality, preserving what the algorithms detect are the most noticeable and important features of the media to a higher standard than less important features.

For GIF, I guess MIDI (musical notes) would be the audio version because GIF's generate very small files when there are few colors in an image, just like MIDI's are typically very small files, and can be used when the entire song is made up of only basic notes and instruments (no sound effects, post-processing, and you don't care if quite important notes sound different and generic, just like gif's can also make the colors different to fit the color space and lose detail.) Both try to describe their media about as succinctly as possible (i.e. smallest file size), and therefore both can be misused when the user expects to retain fine details of the original piece of work when converting to those formats.

New formats have been invented or appropriated since these came out, and I could be wrong about everything, but I think this is still useful info.
9.) What would each cofounder rather be doing, if they could do anything at all in the world?

This was an issue with my last startup. We started with 5 founders, but for 3 of them, the answer was "my day job", and we had to get rid of them at an early stage. My other cofounder was very upfront with me from the beginning - he was all in with this particular startup, unless he got into Stanford or Harvard Business School. Well, he got into Harvard, and that was the end of that partnership. For me - I probably would not have abandoned my partner, but once he'd already left, I folded the startup when I had the chance to work with a YC startup, even though I didn't end up taking the cofounder position with them.

10.) What're your expectations regarding the amount of pressure you'll be putting on yourselves?

This is more than just working hours - it's also how seriously you beat yourselves up when you miss a deadline or things don't go your way. It's really important to find someone who's compatible with you on this; a laid-back person and a neurotic one working together tends to lead to a lot of conflict.

Windows auto-update is kicking me off now, but I may think of more in the morning.
You already posted this and you are making comments from what looks like your irc logs or comments from the previous post.

http://news.ycombinator.com/item?id=361730
Open Source Apps in Your Brain
Amazon to Sell OLPC XO Laptops With the Get One, Give One Arrangement
Duplicate post.

http://news.ycombinator.com/item?id=360365

There seems to be a lot of duplicate posts lately.

Perhaps the 'submit' button should only appear starting on page 3! :-)
Shannon's information is a measure of the size of a set that an element is selected from, is that right? So a letter conveys log2(26) bits of information. That alone doesn't allow me to discriminate between the information content of a random string and an English sentence, since both supposedly contain the same amount of information by this bare metric.

If I look at the occurrence of subsets of the string, then that would be a better discriminator: the random string's subsets should follow a normal distribution while the English string's subsets will be highly skewed.

However, that doesn't work when I try to discriminate between an English string and a string generated by a simple algorithm, since the latter's subset distribution will also be highly skewed. What kind of metric discriminates the English sentence from either case?

Am I making sense here? I haven't had any formal training in information theory, and my brain is kind of fried right now.
IMHO you would be much better off just finding someone to deal with all this. A contractor or a partner.

[Edit: read someone experienced on this]
open funding prospects to popular vote. people will choose who they think are the most suitable to receive funding.

by making it competitive, and relevant to the consumers of the products, we inject realism to ventures. we will no longer be investing in projects that do not engage the user base.

think of it like taking all of the Y-combinator submissions and opening them up to rating by the readership of HN.

i feel the problem with VC is projects get funding that aren't in any way vetted by the consumers they intend to serve. and of course the whole "not having a monetization plan" thing...
Ah I see, it's because I linked directly to the pdf.
It's slightly irritating when a name is put beside a post, and I don't see any reason why this person is supposed to be famous. And the site is very focused on whoever that is.

If you write good content, let it stand for itself, and don't try to sell yourself together with your content. Sooner or later, people will turn against you.
I couldn't help but notice that almost your entire second to last paragraph sounds nearly identical to the current iPhones and iPod touches.

Anyway, the only reason why I keep coming back to the iPod rather than any other player in the market is because of its usability, not because of the hardware and generally not because of the design. It may not have all the bells and whistles, but it is drop dead simple to figure out how to use, partially since it doesn't have a dozen buttons just to control playback.

I feel that maybe the rest of the competitors (after all, there is also iriver, cowon etc. that are willing to spend the R&D money on new designs since they are in a market (south korea) where iPods and Apple do not dominate as much) just overthink things, and underestimate the now fairly significant tie-in most people encounter if they purchased significant amounts of DRM'd media from the iTunes store. Hopefully the latter will decrease as there's growing demand for unlocked music, but the former...I have a neighbor who insisted on buying a portable CD player, and you have no idea how complicated it can get. I about wanted to give up after staring at a recent Sony one that had, like I said, a dozen buttons just for playback. She eventually caved in and got an iPod nano, which she was able to figure out without reading any manual.
I need to understand the technical requirements first, but thanks for the suggestion.
Looking at the lognormal curves shown here http://en.wikipedia.org/wiki/Log-normal_distribution
I see tails. Don't you ? And this tail can be long.

The real question is whether it is possible to build a viable business by exploiting the tail. From what I understood this depends on two things: 1° on the benefit you can make by exploiting the tail, 2° on securing it against the businesses exploiting the belly of the curve.

What Will Page tells us is that the tail is a starving zone. Since he works for the belly business he simply tells us that the second rule applies. Whether the first rule apply depends on the costs and benefits which may vary greatly with the business model considered.

Once again the Register demonstrates a lack of common sense.
I seem to remember the CCC on Mark Watson's page says you have to download it from him.
This is a "follow-up thread" to the link you posted. I already explained this in the first post in this thread and also in the previous thread.

Look, there are lots of sub-topics to deal with in this project, and the previous thread was totally messed up and disorganized -- meaning I couldn't follow any of the sub-topics effectively. That's why I re-posted some of the comments in separate posts in this thread ... so I can continue to discuss them in a clearer and easier to follow manner.

Is there something wrong with doing this that I'm not aware of?
Charging Station
Charging Station
Charging Bull
Hi this post holds everything on charging. I will post every charging problem in this page.
$10k hacking contest announced
Stop Me If You've Seen This Word Before
Jealous? I'm interested in hearing what Gabor Cselle has to say. He worked at Xobni and seems intelligent.
I think you are saturating the channel. Why didn't you do one thing at a time, maybe one a day?

Think you are getting here responses that would cost at least a few thousand dollars with a decent contractor.

I've never seen posts in this form on HN, at least in the last few months. As a new user I try to understand the micro-culture and etiquette of the group before posting. Not just go and do things the way I think it should be.

Just a thought.
If I had a trick, I would have shared it...
I much prefer to use the term co-founder in indicating my role since it indicates that we're not a one man show.
Thanks jwilliams, I'll more than likely be using a high performance Linux system so hopefully it will give me the precision I need.