skysight-inc/HackerNews huggingface dataset:
Ask HN: Distributed File System
Hey peeps.

So I need some recommendations.

I've been building a distributed file system for work to store our hash tables with. These are 1 gig files (about 40TB worth of them) that are write once, read many.

It needs to duplicate the data across servers and make the file available via HTTP. Oh and it needs to scale quite well because as from next month we are potentially adding another TB per month.

So far I havent been able to find a DFS that does all the above so have been working on my own. But am nervous - the files are mission critical but I am not too worried about losing stuff per se (there are alternative backup solutions that make sure we have multiple static copies safe). Im more worried about not being able ot cope with the load. My current implementation is in Python and simply uses a central MYSQL server to track file locations.

So. Can anyone recommend a DFS option I have missed that fulfills my requirements. Or even better can anyone offer technical ideas to help with the development of our code.

:)
Yes. Eg. operations research can be quite rewarding from a math point of view. Esp. combinatorial optimization (as a subfield) has a lof of interesting algorithms and concepts.
Has Java ever been big on the Linux desktop?
Vuze is the probably only Java app that I've used for more then four minutes.
It's amazing how much memory Tomboy uses.
It's waay too much for a note-taking widget, that I barely use. It's crazy.

Someone should totally re-write it in C. Or Java ;-)
Ask HN: Never scroll up or down again
I want to make every screenfull of code I look at while programming discrete. I want to never scroll up or down again. To do this I make every screenfull of code its own file, and view a list of these files using emacs' list-buffers command.

What d'ya'll think? Think it'll work well, poorly, why or why not?

(Think fundamental design reasons, not language specific issues, like an excessive number of import statements in python caused by this style.)

Anybody here ever heard of somebody doing something similar?
Library of Free Data Models from DatabaseAnswers.org
Mono certainly aims for better integration into the Linux userspace, at the cost of portability.

The emphasis of Java on portability / platform agnosticity is one of it's strengths and one of its weaknesses. It's part of the platform's identity. Comparing the two platforms about linux desktop development is almost nonsensical, since for mono, linux desktop apps is the selling point (GTK# is designed as the main GUI framework for mono, and it gives a native feel on linux), whereas Java has a much broader audience at the moment.
It's being done : http://gitorious.org/gnote/mainline

Although i'd say this is going backwards. Improving Mono/TomBoy memory consumption feels more like the solution here, even if it's a bigger job
St. Pete Deep Sea Fishing
St. Pete Deep Sea Fishing - Bentcharters - Welcome to the Bent Charters photo album! Here you can view pictures from real charters that Lance has hosted.
21st Century Approaches to Discovering (Chemical) Data
Wedding Photos on Canvas
Our wedding photos are always special to us and we want to cherish them so that our entire family can get a glimpse of what it was like on our wedding day. We might occasionally get emotional about our wedding and have a look at the wedding album. This is such a waste. We should have our wedding photos on display for everyone to enjoy.
Anybizsoft PDF to Word 1.0 Beta Will Be Closed
AnyBizSoft PDF to Word 1.0 will leave beta on July 15th 2009. We will offer competitive discount for all the beta users when PDF to Word 2.0 is released.If you have not experienced the convenience and power of free Anybizsoft PDF to Word converter, please hurry up now !
With daring fireballs obvious and declared bias for Apple, its hard not to be a little cynical sometimes. Is it so ridiculous to talk about Chrome OS being a linux, or does it just make him feel better about Apple to downplay the relationship between BSD and OSX?
Python actually seems to be fairly popular in the desktop linux space. At least in the GNOME universe.
Unsecured Business Credit Lines | Start-Up loans
Getting a business line of credit is a great way for businesses to get the money that they need to run their business whenever they need business loans.Unsecured credit line financing is vital to the success of many businesses because it provides the temporary cash flow that keeps a business going. Start up loan financing can provide financial flexibility for entrepreneurs.Start up loan financing often includes an unsecured line of credit.
I in no way intend to start one of those conversations that ultimately end up in a flame war over which operating system is better. With that said, I don't think Microsoft has killed anything related to Linux. I'll elaborate by presenting my history with Linux.

2004 - Redhat; could not get it to work smoothly on my box; switched back to Windows

2006 - Redhat; got it to work on my box but I felt overwhelmed by the options available and the lack of a Windows feel; switched back to Windows

2007 - Mac; had a Windows feel while opening the door to the Unix world; started serious work on financial programming

2009 - Ubuntu; using it for everything at home (programming and entertainment), but will be buying a Windows notebook this fall for graduate school

Linux has come a long way since I started using it. It might be that the Mac was the key step in the progression from Windows to Linux. The Mac allowed me to develop a working knowledge of Unix features while maintaining the Windows point-and-click environment that I grew up with. I see all three OS's as having their own uses. I like Ubuntu because I can pick it apart; curiosity has forced me to do a full reinstall several times. I like Windows because I haven't found a replacement for some of the programs I like to use, specifically Excel. I don't see why all three OS's can't have a place in the marketplace.
That's what I thought when I read this article. I've been using linux for years and I only remember to have used on Java App: jabref.
Flickr started as a MMORPG! Let that be a lesson to all of us founders. Questions your ideas, because you're smart but not perfect.
Well that's not 100% true because the main interface for Firefox on Mac OS X looks more like Safari than it does Firefox on Linux.

I think they've actually done a really good job making it look like a native app instead of just a port from some other OS.
I'm not sure what you're expecting or what you're disappointed about. It's not like closed source commercial software have any better track records.
Chrome is faster than safari on Mac
http://www.manu-j.com/blog/safari-vs-firefox-vs-chrome-on-ma...

The point of google gears is that you can get the HTML5 features in IE and other older browsers. It also has some features like drag and drop which AFAIK is not present in HTML5
The problem with charging for news sites is less about how many of your regular readers are willing to pay for the content and is more about how people interact with your site. The majority of news articles on the web are accessed through linking from other sites (Google, blogs, etc). If you put up pay wall, you instantly lose most of that traffic.

WSJ works around this by giving access to people coming from outside links, but I'm not sure how well that model works:

http://machinist.salon.com/blog/2008/03/21/wsj/
Chrome OS is not for everyone and it will be successful in its target marget - Netbooks.

Netbooks are under powered devices. Even when you run a full OS on a netbook, it is not a pleasant experience to run resource heavy applications like Photoshop. No one buys a netbook to run those kind of apps anyway. People look at netbook as a device on which they can read stuff, do some emailing or some light editing of documents. These are functions which Chrome OS can fulfill pretty easily. So I do expect Chrome OS to be a success in the netbook market. Outside of netbooks, maybe on tablets. Unlike Techcrunch, I don’t subscribe to the idea that Chrome OS is a credible competitor to Windows. So Chrome might succeed in the netbook market but that won’t put much of a dent in Windows dominance.

My 2 cents: http://www.manu-j.com/blog/google-chrome-os-a-speculative-an...
What's the hint on level 30?
* Awesomebar. Some people hate it but I love it, and in Firefox 3.5 it's extremely easy to disable via the GUI.

* Bookmarks. I can easily bookmark any page by clicking on the star next to the address bar. Clicking it again allows me to edit it or remove it. I use this all the time.

* History with search.

* Spell check. I used to think it's useless but now I cannot live without it.
Ever thought about getting some super-duper SAN toys?
If the files are really that important shouldn't they have good permissions set? That is, if "Firefox shouldn't even have access to" them, shouldn't that be enforced by some OS level permissions?

And shouldn't any other programs one has around not be writing confidential information to temp files in the first place?

Seems like more of a dumb bug to me than a security problem.
No idea, it's one Firefox's killer features I miss in other browsers.

That said, Firefox 3.5 allows one to VERY easily disable the awesomebar: Preferences -> Privacy -> Location Bar.
Overactive Bladder Information|Overactive Bladder Tips
Now a Days Overacitve Bladder is the major problem in younger and old age people.Here you can get most of the information regarding the overactive bladder control,overactive bladder information,overactive bladder tips and treatments through mind over bladder training program!! Bladder control products and bladder fitness programs are also supposed to control the bladder.
Interactive Advertising Agency 
Ideaz Inc combines strategic advertising consulting with the acumen of a full service interactive advertising agency in India.
You don't read about it much because profit share is the devil! Communism! Unthinkable.

Profit share is reserved to the better people, investors, execs. You don't exist to share profit, you exist to create the damn profit.

Yes, I'm exaggerating a little. But in essence that mindset is pretty much the norm in "old economy".
Just tell me when it outpaces c++. How many mono apps are there? How many c++ apps? Mono has a really long way to go before it can seriously challenge C++/KDE. I hope not to be misunderstood, moving away from c++ is somewhat a good idea, but its much too early for statements like this, because how many java apps are there to be outpacet?.
If you found this interesting, read his book - Punished by Rewards. It's amazing. I read it at a young age (13 maybe?) and it totally changed my life.

Here's the real takeaway: in this life, almost everyone (at least in the US?) is conditioned, on every level possible, "do that, get this." It's not just about money, but approval, self-esteem, good grades, a place in society, a decent life, a good relationship, healthy children.

We learn this lesson at our parents' knees, it's confirmed in school, and cemented in employment/college education.

The only real reason to do something is because you love it enough to bring it into being. (This is not the same as "loving what you do," because the production of outcomes you love enough to create will inevitably contain work you aren't going to love in and of itself. Like, for example, exercise and practice -- not always loveable. The key here is that the love is aimed at the way you want to change the world, in whatever fashion.)

Anything else is some combination of carrot/stick -- they are flip sides of the same coin, and neither work well, in the long run, and both lead to burnout.

Even if you think you "have" to have pressure to produce, you don't. Even if you think you "have" to have a business goal to create, you don't. Even if you think you have to taunt yourself with images of what will happen if you don't do what you "ought to," you don't.

If you can step outside this reactionary type of living model, it's amazing what you can do.
Let's be realistic. The NYTimes will still get plenty of links. The first comment on all of them (heavily upvoted) will be "email: bugmenot@mailinator.com password: bugmenot". The second comment will be "Why do people keep posting links to the NYT? Paywalls sucksorz, yo."

And half of the people upvoting these two comments will think that their startup will succeed on ad revenue where the NYT failed.
Teensy bit expensive :)

We have 4000 clients atm but hope to increase that quickly (20,000 within a year). That's TB's of data a month. SAN was considered but it too expensive to scale :(
EDIT: well, not over the top expensive. But commodity servers with software is cheaper/better for us.

Also we at some point need a HTTP interface to the outside world.
I think sometimes a duplicate is valid, after all I read through HN 2-3 times a day and I still managed to miss the thread for this the first time around.
As usual, there's no conspiracy. They're working on a GStreamer backend for Firefox but it wasn't finished in time for 3.5. It will probably go in next time around; the current version is a functional stopgap until then.
Im a web developer (sorta).

IE8 is a perfectly good browser :) Ditch Opera PLEASE, they are a pain in the...

(im only kidding Opera is good too)
MogileFS is what you're looking for. Native http support with a MySQL tracker.
Not sure if something like Cassandra would help? http://en.wikipedia.org/wiki/Cassandra_%28database%29

I think there must be open source clones of the FS Google uses, but I don't know the names.
J2ME Mobile Development Series 5 - Basic Collision Detection
Sounds like a job for the Hadoop HDFS.
Writing non-trivial / multi-threaded desktop software with Python is a debugging nightmare.

I've been a volunteer Banshee dev (~20 KLOC) for about 4 years. With limited time to contribute I wouldn't be nearly as productive if I had used C or Python. Without Mono lowering the (time) barrier for entry I probably wouldn't be a long-term free software developer.

Personally, I'm sad to see JavaScript coming to the forefront with GNOME 3.0.
Timesheet
Replicon Timesheet Software offers easy to use web based time tracking software for tracking employee time, project time, as well as expenses, to streamline payroll and attendance as well as project tracking, project costing and client billing.
I'm from the UK and have traveled fairly widely - and I have to say that San Francisco has the highest number of homeless people I've seen anywhere in the developed world. Maybe it's the weather. Anyway, whatever's being done about it, it's either not working, or not enough.
"It's also worth noting that many of the high-profile Mono applications are written and maintained by Novell. That's a pretty classic platform strategy—try to get your platform broader distribution (in this case, integrated into the Gnome desktop) by creating compelling applications that require it."

The "Creation of Compelling Applications" isn't a strategy, it's the only reason that platforms exist.
OrnamentsAndMore.com
OrnamentsAndMore.com offers 2,500+ ornament choices with 6,000 ways to personalize your ornament for FREE. We offer free shipping and free gift bags on most merchandise. We are the leaders in outstanding quality, personal and friendly customer service and 24 hour (in-season) order processing. OrnamentsAndMore.com employs a staff of creative writers to hand personalize each ornament and carefully pack and ship your purchase promptly. Enjoy your shopping experience with OrnamentsAndMore.com
Your use-case doesn't sound like rocket surgery so I'd say you're on the right track. Write once, read many is trivial to scale. Just build it according to your needs and throw in memcached hosts as needed.

If your access pattern allows for good caching you could also skip all the hassle of maintaining physical spindles and just move your stuff to S3. Their storage fees are quite affordable, 100T would set your back around $15k a month.

The killer with S3 is in the traffic bill, though. So that's only an option when most of your requests can be served from cache.
Raja Raman Designs
S3 would fit our needs perfectly. But the boos vetoed it :( security issues (I know, I know but the data is fairly high value). Sucks.
Openfund - The Business Plan
a GFS clone was the angle I was looking at.

Cassandra looks pretty fun - are you suggesting that as the database right? Im thinking a quick python implementation for PUT (maybe DELETE) and meta operations, using cassandra as a backend and Lighttpd for the GET (high performance) might work..... cheers.
Hmm, sounds like tahoe may do what you want: http://allmydata.org/trac/tahoe
UNIX Compression Benchmark
It does have live updates which changes things contextually.
I can't find how to get to level 25... :(
Do you really need "complete access to your computer" to do 3D in Java? That's what the viewer applet is requesting according to the warning I get, and that sounds a bit over the top to me.
The big question about calorie restriction in humans is what effects it may have on mental functioning. I don't know about you, but I have trouble concentrating when I am hungry; and there is the further risk of poor mental development with calorie restriction in children.
Use encryption, perhaps?
Software Gold Rushes (Steve McConnell) [pdf]
Layar: The Countless Layers of Reality
HTML5 Drag'n'Drop: http://www.whatwg.org/specs/web-apps/current-work/#dnd

Gears as a Bleeding Edge HTML5 implementation:
http://almaer.com/blog/gears-as-a-bleeding-edge-html-5-imple...
Python vs. Perl vs. Java vs. C++ Runtimes
Funny, everybody on Linux seems to have exactly one Java app... mine's jBidwatcher.
Download Typebased Template for Blogger & Wordpress. | BLOGGER TIPS AND TRICKS
I always just "(Dis)allow pages using their own fonts" + a readable minimum font size. That makes browsing much more comfortable for me and doesn't break too many sites (only very rigid table layouts, most of the time crappy online shops).
I'm not sure I understand what the problem is. You say that you've got approximately 40000 files; a database which keeps track of where those files are stored is a trivially small problem.
Scalability Only One-Half the Reliability Equation
agree. You need to look into Hadoop HDFS. (a GFS clone...)
This just adds weight to what I've been saying all along: C# = The Java Killer. Also, there's more java on linux than you think. Hell, Android is a whole platform for java on Linux, which is the dumbest possible thing to do, which is why its failing miserably :-)
I'd never heard of MonoDevelop, and it seems to me this article was really scrounging to create a sense of its "primacy" that isn't real.

Neither sfbay-craigslist, Indeed, or SimplyHired come back with any job listings that mention 'MonoDevelop'. Meanwhile, even combining the three terms [linux eclipse java] gets multiple hits at Craigslist and over a thousand each at Indeed/SimplyHired.

There's a fair chance that Eclipse itself is the most popular Java desktop app on Linux. (Developers aren't blocked by outdated Debian packages nearly as much as the article implies, and the article's claims about what was missing in Eclipse 3.1 are false.)

Azureus and LimeWire have also been widely-used Java desktop apps, on Linux and elsewhere.
Top 5 Fastest Bikes in the World
If you want a GFS clone, you /really/ want HDFS from the Hadoop project - I'm reading the new O'Reilly book on Hadoop on the moment which is excellent.
@flamewhale fuck you retard... are you a 4chan member?

The irony is delicious!
The longest list of CSS frameworks you probably ever seen
Writing multi-threaded software in any language, that uses shared state and locking primitives is debugging nightmare. The list of those languages includes not only Python, but Java and C# too.
It needs to scale robustly to support at least a million files (with 3 duplications - so 4 million entries minimum) and accept really high load (min. 1000 requests simultaneously, better 5000 requests simultaneously). The main concern is MYSQL is the single point of failure.
Cool title, but completely misleading.

I thought this paper would be about writing software on top of an untrusted hardware architecture (an increasingly important national security concern, considering that so many chips are produced in foreign countries, and could have hidden flaws purposefully inserted).

Instead, the paper just provides a description of typical attacks against cryptography (man in the middle, side channel attacks). Maybe I missed something as I skimmed through this, but these subjects are much better covered in beginners textbooks.
This carries more weight than the article. In the same direction. ;)
Simplest HTML to PDF Converter
Free Shipping Will Bring a Great Deal of Savings
Most of us prefer to shop online because when we buy the things we need online we will be able to save a lot of money. However, if we don’t pay attention while shopping we will end up paying more than what we would have paid at our local retail store. When we buy things online, they need to be shipped to our door steps. This costs a lot of money depending on the shipping distance. Some online stores try to offset the cheap prices they offer for their products through high shipping prices.
C'mon, people. $5 is less than one visit to McDonalds.

And by subscribing with such little amount of money, you support the Internet environment, by increasing number of people paying for digital goods.
Twenty years ago, you'd get flamed to a crisp for that 'preprocessor' slip ...

[Amplification for those not watching back then: it was an important theological^Wtechnical point that 'cfront' (C++ -> C) was not just a simple macro expander like the C preprocessor.]
Yes, see that wikipedia article linked above. The problem has been solved.
My experience is limited to one application and I used my own hand-crafted system based on select for this rather than twisted (which I didn't grok until I'd written my own scheduling model). However, the experience was good (and it was a joy to debug - very straightforward) so I'll continue here.

If you're writing python apps you're better off using twisted than the threading system (which in my limited experience is indeed horrid).

Twisted is pitched as being a networking layer, but I find it better to think about it as a purpose built scheduler which happens to have poweful networking modules. To explain what I mean by scheduler: - you have a cycle of standard functions for the application which might handle input, look at the network for traffic, handle output. Sometimes in the course of execution more functions will be needed and you stick them into the cycle. So you have an extra item in the rotation that handles custom tasks. Once you've got a scheduler model like this you can tweak it to make your application more responsive if it needs it.

You can use pyinstaller to produce a packaged binary that can be easily transported between platforms without requiring users to install a stack of python libraries. If you use wxpython as your GUI the whole thing can be multiplatform.
So if you start a company during this economy-intervention you want it to be a prey company and not a predator company?
I wouldn’t be surprised if it’s only supported for use on new PCs that are specifically certified to work with it

Isn't it supposed to built on a Linux kernel? In that case, I think it will potentially run on a rather wide range of hardware.
I don't think so. IIRC, bugmenot does not allow paid subscription details to be posted to it..

But I still agree that NYTimes will still receive a fair bit of links but also believe it will decrease to some extent.
Why would he downplay that? And Gruber was never a blind Apple fanboy — when apple does BS, he calls it BS (just search for "shit sandwich" in the article and follow the link). I have a feeling that you completely missed the point there.
A bit of a misleading headline here. It's just a listing of five different YC-type programs. That counts as one alternative in my book.

This is like titling an article 5 Alternatives to the Traditional Mortgage and then listing five different but nearly identical ARMs from five different banks.