skysight-inc/HackerNews huggingface dataset:
Ask HN: Why do people use Puppeteer for webscraping instead of SaaS services?
Hi HN

As in the title, I am wondering what are the reasons anyone would use Puppeteer/Selenium/other-browser-emu for web scraping if there are already tens if not hundreds of SaaS services offering "scraping-as-a-service". Except for JS execution.

A handful of examples: Scrapehero, Webrobots, Apify, Scrapingbee, Scrapinghub, Promptcloud

Except for the ones that require setup fee, or have ridiculous pricing models. Why would anyone want to setup Puppeteer/Selenium/other scraping bots instead of using one of the "scraping-as-a-service" platforms?
I've lead two successful community "unbundling" efforts from reddit. Here are a few tips if you're thinking of doing the same:

(1) You HAVE to kill the original community. Otherwise momentum will keep your users using the old community. Our first launch saw a successful small website take off initially after it was launched only to have traffic slowly die down over the next several months. What finally got it off the ground and growing again was killing the original reddit community. There's no way around this.

(2) There WILL be resistance and/or a drop-off in users. The good news is now there are a lot more opportunities to grow your community: Advertising, cross-site collaborations, guest blog posts, etc.

(3) Do not strive for feature parity with reddit. Instead, strive to deliver MORE value. For instance changemyview could add categorization features. Maybe a search function that lets you search for posts that have been tagged as "view changed successfully!" or "view not changed".

(4) You have to have a business model. Selling advertising or taking donations are NOT business models. There was a time when it made sense but many online communities - reddit particularly - are averse to viewing ads. If you're not creating something, selling something, or taking a commission then you haven't got a business model. One model which can be successful for larger communities: Premium membership subscriptions. You just have to make sure the premium features provide sufficient value that people want to subscribe without making non-subscribers feel like you're just limiting features to be greedy.

(5) If you have multiple moderators - incorporate.
Those are all US companies. What does Europe have to do with them?

And like it's been noted many times before here, had Europe banned US companies like China had, maybe they would have a stronger software development culture, and they could control their own data instead of handing over everything to the US.
>Moderators have a lot of power to influence conversations on Reddit in non-obvious ways.

You had the above quote when I was commenting on your post, but may have edited it out. But I think this line right here is also why reddit is becoming unusable and going downhill fast.

Basically, moderators are given way too much power and users have no recourse against mod abuse really. Users can't vote a moderator out for example. Many times, mod abuse is also hidden, so you can't see a log of what they have been removing and hiding. Both of these things would show clear mod abuse in the open, but reddit seems to want to hide this.

Moderators should really only have the power to remove illegal stuff or have clear rules linked for each removal or ban.

Mods should not be able to turn on "filter" features that auto hide or auto delete posts, as they regularly just use it to target posters or topics they don't like, even if it is still on topic and popular in the subreddit.

Also, mods are more and more removing or locking posts mainly because they politically disagree with them. They use excuses or hide this corrupting behavior all the time. Many times a lock posts will be done with a claim "its too hard to keep moderating this post", when reality is they were just removing posts that didn't break rules and were posts the mod simply disagreed with.

Add all the above and more and combine it with clear corrupt interests, and you basically ruin what reddit once was. Which was a place to go to free discussion on many topics. It is not longer that. It is just a place where mods basically abuse their power on most subreddits and astroturfing is more and more the norm.

Its almost like people forgot their was a an upvote and downvote button. The mods really have no reason removing posts, outside of clear violating posts that may break laws, since the users can choose what they want to see with the upvote and downvote button.
Of course, the most populous states are the most in favor of NPVIC. I would expect that most of the states that are going to approve it have already done so; I expect that there will be very little further movement.
The Left-NIMBY Canon
Thanks for the recommendation! This looks powerful and useful. I will definitely check it out.
It's trained on fanart, mostly from pixiv via another site, which is tagged safe/sort-of-nsfw/nsfw but many things are tagged safer than they actually are.
"Technically, they are judges at court and should rule according to the law not their political leanings."

If that were true you wouldn't have liberals appointing liberal judges and conservatives appointing conservatives, and fighting tooth and nail over it.

Also, the Supreme Court effectively makes the law, not just follows it, and that's true to some extent of lower courts as well, as most cases never make it to the SC.

It's not a coincidence that now that we have a conservative Supreme Court it's widely expected that Roe v Wade is going to get overturned soon. The law didn't change, but who's on the SC did.

"They should do their job irregardless of who is in power."

They have a lot leeway as to how to do their job, and many of them will use their power to pursue vendettas against people they don't like, as we saw with Abu Ghraib, Gitmo, Bagram, black sites around the world, and all the torture and murder that the police engage in.

Prison guards and employees of ICE and Homeland Security are also probably mostly Republicans, and it's hard to argue that their politics and world view doesn't affect the way they do their job or why they are in those jobs in the first place.
Well, to be fair, MS Word maintains a full history of every edit that was made.

And, find-and-replace for specific formatting is supported out-of-the-box.

Finally, cmd-i / cmd-u / cmd-b are pretty easy default bindings, and setting up shortcuts for more intricate, specific styles is straightforward in Word.

I share your preference for using an efficient tool—I'm a die-hard md+git practitioner. But the "most people I know" argument is not a good basis for rejecting real, rich features in software that works well for other people.

I am also curious: why do you care about semantic correctness? It's OK if that's just your preference, but it's not something that moves the needle for me....I'm not sure why I should care!

(edit to add: my comment sounds glib and a little sharper that I intended. That wasn't my intent!)
I've been critical of Brave in the past, but I'm glad to see IPFS gaining more traction. IPFS getting browser based support is what's needed to connect in traditional consumers and let it take off.
Awesome, thanks for the info! Do you have more info on what you mean by "We do this by connecting directly to CloudFormation"?
Your risk is being left behind. It's unfortunate but true.
Sorry, I cannot follow.

To be clear, I'm no aws advocate.
I dont disagree with the theory of a lockdown when necessary, but now that I look back at our year in the US...it now seems so obvious that this was never going to work. This place is simply too big with too many selfish idiots that want to do their own thing for how virulent COVID is. Leadership is a huge failure that didn't help at all, but I honestly wouldn't be surprised if we'd be in the same exact position of an uncontrolled spread even if we had a different sitting president with different views. 330 million people spread out across 50 states, across various local governments, some of who might not care as much is just too many people to get on the same page regardless of the message from the top.

Australia is 7% of the population of the US. Barely a comparison
But isn't that exactly what's happening here?
Missing any reference to the one and only editor, every commentator was actually having in mind: Emacs.
Eltanin OS
According to this site what really doesn't exist is an anime with a male character in it.
How is this different from other media? It's already common practice with magazines that contain product reviews. There's nothing stopping companies from doing this with bulletin boards and blogs.
yeah agreed, i don't think they are connecting the dots. they are missing large sections of knowledge that would protect them and the lives of those around them. but for some odd reason they have mental blocks when trying to help them out flawed framework
I think we can look to mobile to see how feasible this might be: consistently over the past decade, iPhones have matched or exceeded Android performance with noticeably smaller capacity batteries. A-series chips and Qualcomm chips are both ARM. Apple's tight integration comes with a cost when it comes to flexibility, and, you can argue, developer experience, but it's clearly not just the silicon itself that leads to the performance we're seeing in the M1 Macs.
- It could happen to you!
- It already has?
- That just proves my point!

Ignoring other people's experiences when issuing grim warnings is not a great style of argument, because it treats all hypothetical situations as equivalent and interchangeable.
Haha whoops... I will add a correction for this.
I'm not certain, but doesn't a 'read only' reddit page still get changed when a user who was in it deletes their account and posts? Often when a search engine sends me to reddit, the comments that probably contained the relevant information are long gone.
For years there were daily posts on /r/movies that got to the top like "Remember how good <80s movie> was? Watch it now on Netflix!"

It wasn't even thinly veiled, but of course you got downvoted to oblivion if you even mentioned dropping a named platform from the title.
It's not like Japan does a lot of incest in real life, they just seem to think romances are boring without violating taboos. Japanese people are so against incest they won't have sex after marriage 'cause you're related now.
Working code is a side effect of being a Software Engineer.
Vuejs Camera Component
> Introducing taxes on externalities is a form of central planning.

It isn't considered that by economists. It's a means of internalizing the externalities, not deciding what, when, how and by whom to provide goods and services.

> many people will argue that if a market is subject to such taxes, then it is no longer a free market.

They're wrong. Externalities are pushing costs onto other people without their consent, which is not free market.
"Malbolge" does at least sound sinister, so if you're going for an evil hacker vibe it's perfect.
Reddit’s former CEO is now in the forest planting business
Thanks! What's the best way to reach you? Much appreciated.
I noticed this recently when they redesigned the comments view to make the user's profile picture a prominent part of the design. Except after the last few weeks of scrolling through comments, I have come across less than 10 users sitewide who have actually set a picture. It's now just a sea of default icons and a ton of wasted space.

I can't imagine people use the chat function either. It seems like they are prioritizing features for the sake of ad revenue or product team egos over making much needed infra updates. General latency and error rates while browsing reddit.com are almost unbearable now. And search always was and still is a mess. And don't get me started on the mobile apps..
You're right, it was more than dozens. I think my mental model is something like:

- Dozens of severely bad actors - many of which actually did intend abduction or murder. These people are terrifying.

- Hundreds of bad actors. People attempting to subvert democracy which, despite sounding alarmist or cheesy without proper context (You need to see the videos to understand, I think) is also very scary. These people will hurt you for their political cause.

- Thousands of people convinced that their political system, their democracy, isn't serving them. This is alright with me - I think they were convinced by the bad/severely bad actors, and I believe this happens on all political spectrums. I know many "left leaning" folks with arguably insane ideas about politics and society in general, so this isn't a republican disorder. I don't resent them or even myself for coming to unreasonable conclusions and I believe they're actually trying to improve their situation in the way they know how. Being wrong in and of itself isn't a crime, I suppose. Protesting on bad information is unfortunate, but not criminal.

I have sympathy for all of these people. Things seem pretty awful for them right now, whether it should or not. It all seems very self-inflicted. Regardless, it seems a lot of manipulation has been involved. It's very cult-like. Power hungry folks have recruited disenfranchised individuals to serve their agendas. It's really sad.

The rest of the republican voters will be a mix of these groups, but I suspect they've come to terms with the election despite disappointment. They don't want to zip tie anyone in congress or illegally overturn the election.

The bad actors really should experience consequences. They are a rot that contaminates the people around them. Whether they intend that or not, have good intentions or not, they have become so unreasonable and dangerous that they need to be contained at the least. I believe that includes Donald Trump most of all.
> The problem is that this organic startup is now a enterprise behemoth that can be trusted as much as CCP: Microsoft.

Remind me again when MS killed millions of people and committed genocide? I mean, Windows is horrible, but this is a bit much.

I apologize for the snark (I really don't think it's a good way to communicate) but this is a shockingly bad opinion. Comparing a bad company to a totalitarian regime that routinely abuses human rights cheapens real suffering and numbs people to real tragedy.
Optional Value and Life
New book by thedeepdish.org. Looks at how option value is critical in all domains of life.
Yes, but I've been in it since 2011. Your humble brag doesn't apply to me.

My point being that this doesn't warrant the attention its getting when you see the bigger picture of outright fraud, and agencies should look into bank fraud and money laundering and stock manipulation of things like large pharmaceutical corps that have made billions from a litany of scams along the years, Moderna administration in CA already had to be halted due to patient complications. How many Corps have made out like bandits from bailouts and crushing the competition with perpetual lock downs and doing shady trade deals with politicians?

I'm so over this, that the paper wealth pales in comparison to the asset inflation we're all seeing with no real benefit other than more billionaire wealth accumulation.

PS: I don't trade, I'm accumulating in order to exit this insane system.
Yes, I would definitely trust the CCP to host my source code /s
When were you in NYC? During the first 3-5 months of COVID, NYC was a complete ghost town with nothing open aside from essential stores. Once we got our numbers down to some of the lowest in the country (compared to other cities with similar populations), everything then became lax again, and now it's sort of this half-assed situation as I described above. Cases out of control, but you're still allowed to do various things that seemingly havent been very well thought out
The elephant in the room is voter fraud. If we switch to raw vote count then it takes just one loosely regulated precinct to flood the election with fraudulent votes to win the election, and anyone can do this. Sure, this time around one group may win the fraud vote, but next time it will be another group, and the the best fraudsters will conspire to run the country and we will cease to be a democracy and become a kleptocracy. At least with the electoral college the fraud has to be much better organized and spread more across the country.
Suspicion and tech glitches blight India’s vaccination campaign
Ask HN: How to improve communication and articulation skills?
Well, there was that whole Civil War thing. We are less divided than that, at a minimum.
> I'd actually be interested what turns out if an a.i. actually be trained from a completely random selection

Something I noticed is that AIs appear to have trouble with differentiating perspective vs art style. What I mean by that often characters are in 3/4 view, which naturally means one eye will be smaller than the other due to perspective. But some art styles have bigger eyes than others. What I see sometimes is that the AI then produces an image where each eye comes from a different art style.

It doesn't appear to account for the concept of a three-dimensionally spherical head and how perspective works with respect with such shapes.

It most certainly would not produce good results unless you specify that stuff like shirokuma cafe[0] characters are not human.

[0] https://en.wikipedia.org/wiki/Shirokuma_Cafe
> With "wrong opinion" do you mean inciting hatred and violence, and spreading misinformation?

Hey, how about you show me where Parler incited violence or where Trump did that. I don't care about buzzwords like misinformation or hatred - Twitter cares neither - but I do care about inciting violence.

In exchange I show you tweets from Khamanei - ayatollah of Iran.

> Because by that logic, we should also allow ISIS, Al-Qaeda et al. to do their thing.

ISIS/Al-Qaeda/Al-Nusra/Taliban whatever are terrorist groups; they don't only incite violence. Also putting misinformation and hatred in the same category as actual terror groups is really not wise.
I mean just look at geographic subreddits like r/canada and r/vancouver they all have people, moderators with extreme alt-right views often banning people for sharing opposing views.

Astroturfing is common as aged reddit accounts are easily obtainable and Reddit algorithm isn't sophisticated enough to detect this.

/u/maxwellhill is also the biggest mystery of all. Even pinging the username results in a ban. This single account has been responsible for almost 70% of what you read on r/worldnews and its not far fetched to suggest that a small group of people actively dictate the world landscape.

Majority of people on reddit do not read beyond the headlines much like other social networks. People simply do not care to objectively ask for truth and are punished for doing so on Reddit.

Viral content is recycled over and over for hoarding karma points. Some users were caught fabricating heart wrenching stories for virtual internet points. I believe that these are farmers, creating thousands of accounts to be sold to people with commercial interest.

Very different than what HN does and it is the right way to maintain meaningful discussions. Too much of Reddit is just trolls and maladjusted individuals creating their own pseudo-realities like r/aznidentity or other Red-Pill subreddits.
I respect Japaneese people, their culture and history, but their love for sexualized young looking girls makes me uneasy.
I am not saying anyone should of shouldn't be able to sell information. I am just saying it affected the business.

Oh btw, the next time you read a non-fiction book with a thesis that grabs your attention, but that could probably fit into a long essay.. there is probably a book editor that said it had to be X number of pages because there isn't a business in selling pamphlets.
Why was AMREN banned from youtube then?
Ahh - touche - one time out of 6 - not looking like good odds for Republicans to be in favor of going the popular vote route.
> If most people live in the cities, I don't see an argument for letting rural areas have more than their proportional share of representation.

It does act to prevent the rural areas from becoming totally neglected hinterlands, help them implement polices that may be necessary because of the different conditions of non-urban areas.

That said, I do think the current bargain tips the balance too much, so I'm now in favor of abolishing the Electoral College, and giving the House an advise-and-consent role (in addition to that of the Senate), at least for judges. These smaller states can keep their extra representation in the Senate.
Oh boy! It sounds like you are in a challenging situation. I was more interested in technical challenges of health-related apps but I wish you all the best with this app!
The funny thing about using Craigslist as the example for unbundling is that Craigslist, to this day, works better than anything that's attempted to replace or unbundle it. Craigslist remains a stable, profitable company that provides an enormous amount of value.
Me as well for the same reasons. I get curious about what's happening there sometimes, and at this point it's just a wasteland.

It's difficult to find a thread that isn't politicized in some way, or does not have at least some toxic elements to it. I've been told sometimes that it's 'just the mainstream communities', however the most toxic encounters I have had were always in the more niche threads. It devolves into downright vitriolic flame wars if you disagree with someone, and generally "agree to disagree" just inspires more wrath.
Probably because people who are doing web scraping aren't professional scrapers, they're just programmers who need some data quickly. And since they're already familiar with Selenium, they think that's the state of the art. I've never seen an ad for a scraping service, so I also didn't know that they existed.
Are any of the FBI staffers of modern day also ones that were present 50 years ago? Is it possible the people that compose the FBI currently have different morals and viewpoints from the ones from 50 years ago? The organization known as the FBI is just a collection of people, and the members that make it up can change over time, it's not an everlasting entity itself. Thus it seems unfair to say/imply the FBI is being hypocritical in their message.
Inlining pattern matches allows you to eliminate the value being matched on, preventing an allocation. It transforms an allocation into a function call.
Just the face of media disinformation being more apparent for industry practitioners.

Society as a whole has a major dearth of healthy skepticism - perhaps because a consumerist economy doesn't benefit from skeptics, and more from true believers.
This is really interesting to hear about. Were you the creator of both communities that you unbundled from Reddit? Or did you bring the moderators on board with your business plan? Also how did the users react when you killed the community?
You might be right, in which case Google should offer a link to the corresponding Internet Archive wayback page (where the deleted content should still exist).

Failing that, there are browser extensions for both Chrome and Firefox that enable this functionality.
Pwning an online retailer via public .git directory
I have already repeatedly stated that they were within their legal rights, including in one of my replies to you. I assumed this was a given. When I said they "can't", I did not mean "can't (legally do this)", but "can't (do this in good concience)."

Do you have anything else to add?