skysight-inc/HackerNews huggingface dataset:
Sure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset.

Intent:
Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv.

input:
$1) A text file containing keywords of interest that need to be searched in the files separated by `|`.

$2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so.

# Note: Awk string commands have indexes that start with 1 (not 0).
# The first Awk command just removes the newlines in file $2 and replaces them
# with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'.
# --re-interval allows use of, well, regex intervals.
awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
{
# both variables are initialized with full-content of text file
# with the part after '==== Refs' removed.
cut_content = substr($0, 1, match($0, "==== Refs"))
orig_content = substr($0, 1, match($0, "==== Refs"))

# index (could be better named as offset, but life is too short to refactor).
idx = 0

# match the keywords, even if it appears between non-alphabet chars.
# e.g -keyword1. is a match.
regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"

# loop until matches are found.
while (match(cut_content, regex)) {

# RLENGTH is the length of the matched keyword.
if (RLENGTH > 0) {

# set the matched keyword to the variable `prot`.
# RSTART is the character position of the matched keyword in the file.
# This is the first occurrence of it in the file when read from the start.
prot = substr(cut_content, RSTART-1, RLENGTH+1)

# clean-up punctuation and/or space from the beginning and end of `prot`.
gsub(/[:punct: ]$/, "", prot)
gsub(/^[:punct: ]/, "", prot)

# print the `prot` and 400 chars before and after it using idx
# as an offset. Separate the results by tab char.
print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)

# update offset to character position right after the current matched keyword
idx += (RSTART + RLENGTH -1)

# cut-out all the text up until the position of the
# last matched keyword (including it) and on the remaining text
# repeat the above steps in the loop. This is so that the search
# continues throughout the file and doesn't keep finding the same
# keyword over and over again. See `while` condition.
cut_content = substr(cut_content, RSTART+RLENGTH)
}
}
}'

Edit: Formatting. HN doesn't wrap text in code-block.
Well... not really. In the sense that shareholders can sue a corporation they own if the management is not acting in their best interest while society cannot, a corporations main purpose is to provide profit to shareholders
It's absolutely bizarre how almost no one seems to realize this.
Thanks, it only shows "on Wed Dec 31 1969" for me
You are making my argument for me. Thank you.
Notwithstanding that explanation of a double post you quoted, I didn’t intend to highjack, my comment is about the ability for manufacturers, such as OP I replied to, being able to avoid paying tariffs on component parts which OP’s top post is specificallu about.

It also seemed relevant to mention the original comment being downvoted (without any replies) in an entire thread where no one else mentions the ability for manufacturers to avoid these types of tariffs.
> If you can only afford 10 people, you can usually also get by with 9.

With that kind of logic why should the company even bother hiring a 10th person?
I am bilingual and spend a lot of time interacting with people who are not.

‘Buscar el auto’ Means, literally ‘To look for the car’. The question ‘Puedo buscar el auto’ means ‘can I look for the car.’

As an English speaker you might see that it’s possible to understand this as ‘search the car’. Wrong. In Spanish, if you ask this question… … (and this is key) … … the car cannot be present.

If you are to ‘buscar el auto’, it means

a) There is a car that is familiar to you
b) It is somewhere, but you don’t know where it is.

The article’s use of ‘find’ was close but it missed the mark. ‘Look for’ is the most accurate translation.

A Spanish speaker at this point would have NO idea what the policeman is trying to say. (‘What is this car you’re talking about? Why are you asking me if you can look for the car?')

Moreso, a Spanish speaker with no understanding of how English works would simply not make the link from ‘buscar’ to ‘registrar’ (which is the word the policeman SHOULD have used) even despite the policeman being present and car searches being a normal thing.

The two words are just too far off.

The reason he's saying 'Si' is because this is what one does in Spanish to jog along a conversation where one

a) Is not following
b) Doesn't think there is any hope of asking for clarification without making the conversation even more difficult.
Some societies do.

I'm writing from Canada. Mother and/or father can share up to 35 weeks of paid leave under the national insurance program. This maxes out at, I think, approx $500 per week. It's not a lot. But it helps.

Regarding societal values, this the same program that would normally pay you temporarily if you were laid off (not by choice). Normally you wouldn't be eligible to collect benefits if you willingly left a job. Pregnancy is generally considered to be a choice, so I think making these benefits available to parents shows society does value the choice to parent and will subsidize it.

Parents who adopt a child receive significant additional benefits on top of the standard allowance.

Recently there have been changes to the program to improve benefits for self-employed. As we on HN know ... sometimes a startup founder isn't "employed" or receiving salary in the usual sense and historically it was difficult for such professionals to receive employment insurance benefits but it's changing.

Anyways, progress? :-)
Does the company have a paid leave policy? Does said policy restrict when this paid leave can be taken in respect to hiring date? If people follow company policy, then why are you complaining again?
Python 3 is a different language :)
Biology doesn’t have much to do with it. Being a parent of a newborn is incredibly difficult, no matter if you’re the one bleeding continuously for eight weeks or not.

Women “need” to take more than zero days off. That’s true, at least in our modern conception of child rearing. Both parents should take at least two months off, especially in the current system where people don’t have much community or family support.
They could address it going forward by mandating a specific kernel version which will then receive long term support. Say something like 6 years. And then periodically update the required kernel version to another release which will receive the same 6 years of support.

In addition they could decouple the rest of the Android stack so it can upgraded and run on either of the currently supported kernel versions.

Oh wait, already done.
I thought it was spelled Faux News.

My definition of the media is pretty expansive. If Trump is in so much trouble, I would think the Washington Post or New York Times would lead with it. Maybe they buried the trouble in the Style section.
Ideally, find a contracting role / freelance locally, but work remotely. Doing it "locally" will help build up your network and reputation and lead to more opportunities down the road. Maybe meet 1-2 times every 1/2 weeks and work remotely. More frequent communication should be limited to agile-like questions that come up and should be able to completed over email if you pick your clients well.
"The system" is logical and ethical, in your opinion? I'm trying to call into question here if "a crime" on paper is a crime worth enforcing.

When the black man in New Orleans is thrown in jail for smoking weed, but the black/white/asian male in Iowa goes unprosecuted, how much of a crime is the crime really?

You either need 100% prosecution or 0%. Murder is murder, fraud is fraud, isn't it?
Yam, cultivate and cook
The upcoming movie starring Jennifer Lawrence as Holmes should be a blockbuster.
> Your score dictates what major you’ll study for the next 3-4 years.

Do you have more info available on that? That sounds really bizarre to me.
"...Cohen's messages were stored on iCloud..."

???

Your comment had me struggling to determine how one could store WhatsApp messages on iCloud. (From a Blackberry no less.)

But then I followed your link. I don't think that article was about Cohen. You may want to change your post.

On an equally important note, why is it that people out there assume that ANY form of electronic communication is impervious from government surveillance? The reality is that if a three letter agency is after you, it's probably unwise to be using WhatsApp and Signal in an incriminating fashion. (Or anything else for that matter.)

It's like locking your physical spaces. Yeah, you should go ahead and lock your house or office and turn on the alarm system while you're gone...

but you should also go ahead and assume that those three letter agencies planted surveillance devices in that house or office even in the face of your security measures.
Stalin's propaganda was even effective in the U.S. He really deserves to be at the top of the list when evil leaders are discussed. Yes, Hitler was awful, but Stalin was far worse. But saying so often draws only criticism. That's how good his propaganda was.
I wonder if Thai prison is more hospitable than US. Both sound gnarly compared to Norwegian.
Would love to know more if you have a link about this group. For some reason I'm fascinated by the space gun concept. Even if it were only used to launch raw materials into space cheaply that could still be a huge boost to space exploration.
That's incredible... Is this real?
If we can make provisions for permanently disabled people to have jobs, then we can make provisions for temporarily disabled people to also have jobs.

The article you are commenting on largely deals with large employers (Merck, Walmart), so your complaint about small companies is best directed elsewhere. Exceptions for small companies can be codified into law - GDPR does this, for instance, by letting small companies simply update their terms of service and avoid having to do some of the more onerous revisions to data collection.

And there is a simple formula to leveling the playing field for those people who do want to work, regardless of their bodies (if you are female and would like to have children naturally, you have no choice in the matter - your body will need some accommodations).

1. Increased parental leave (not just maternity), which enables gay parents to have equal rights, and which allows working moms and working dads to work out between themselves how they'll adjust to the baby, according to each of their preferences. It also accommodates the cases of single dads, single moms, etc.

2. Federally funded and regulated daycare. We can look to countries that largely required female manpower to get back to speed after WWII (specifically, Germany and Eastern block countries with high male casualty rates). They had no choice but to figure out how to get the highest percentage of women back to work AND producing children or risk economy collapse and a drastic decline in their population further than already suffered. The solution was efficient, state-wide, state-regulated daycare.

Actually, as an aside - such a daycare system used to exist in the United States, as well, but was revoked. Some speculate an aversion to anything vaguely resembling what the "communists" were doing as why.
[1] https://www.bbc.com/news/magazine-31051633
When you start looking at the code of the tools, and how older systems were design, it's really is a testament to good engineering practices because it all scales so well. Small, simple, "one task well" utilities that process data in streams means the megabyte files of the 80's and 90's are today's terabytes. Sure they require some work to get things scripted correctly but it really is pretty amazing what coreutils and a shell script can do.
This is nice for iterating, but for an actual deployment it's not so nice for three reasons:

(1) Prevents outside navigation into named anchors.
(2) Makes the content slow to load.
(3) Suffers from a flash of unstyled content.

You can use chrome to bake out the HTML with a one-liner:

chrome mydoc_md.html --headless --dump-dom --virtual-time-budget=1000 > mydoc.html

The resulting HTML is fast to load and just as beautiful.
How can you say this with a straight face?! It's a fact that (some) women need to stay off work (bed-ridden before birth, in hospital after birth). It's also a fact that men don't need to.

Obviously, men don't need to work and women don't need to take care of the kids (above and beyond what's biologically required of them), but (1) on average, outcomes will be influenced by underlying biological reality, and (2) it's a fantasy thinking that people can take time off work - e.g. a poor family with a newborn, you really think that any parent will take off more time than they absolutely need to, biologically?

Finally (3) yeah there are possible social levers we can use to counter this biological reality and enforce equality of outcomes, but ultimately, we need to decide what we want - why is women working more better than women spending more time with the children? Personally, I look up to Netherlands, one of the happiest countries on Earth, where many women work just part-time.
So what happens with the tariff when it is collected? Does it go to subsidize the industries hit adversely by this?
medman77, who hasn't been downvoted.

We don't know that. You can only downvote comments for a day(?).. but you can upvote forever. Considering it's been 5 years, and people keep linking to his comment... it's hard to say if he gained those points that day or years later after theranos was found to be a fraud.
1 year vacation? ROFL. In America 3 months of paid maternity leave is a good dead. Also, in America both parties can end a job at will at any time. Also also, referring to parental leave as "vacation" is pretty weird.
Sometimes it's not you who has the bad soft skills. The last company I was with versus the company I'm with now are like night and day in terms of how we communicate. I work remotely and my boss actually reads what I write, comments on it, and is able to tell me when I need to do something without belittling me about not having it done already.
"Oh, my insignificant family life is ruined, but at least I still have options when it comes to a fulfilling and meaningful career!"

"Oh, I have cancer. But at least I'll be saving money on haircuts soon!"

... not quite good enough.

"Going on an air trip with your family of four? Well, if you divert your attention to handling four parachutes, you're more likely to make a mistake, so you'd better just prepare one parachute just your yourself. That way, when the plane's in its terminal dive, you can abandon your family to their deaths and make it out alive."

Something more like that, although that's absurd.

Your view of the situation is inverted and immoral. A relationship is what might actually make life worth living for you. Your skills are how you get a job, and "ten years of home-making" just means that your lonely, pathetic existence, post-relationship, has a few less luxuries in it.
Being anti discrimination started with not discriminating people for things they can't control. Race, gender, etc.

Now we're moving into territory of not discriminating against optional things people choose (having children).

This is a very significant change, IMO. I wonder how far it can be taken.

I think there's a fair argument that since pregnancy only affects women then it counts as gender discrimination. However, if that line of thinking was applied everywhere then nearly everything would be sexist because few things affect men and women exactly 50/50. So in and of itself, that reason cannot fully explain it.
Check out stackblitz
It may be referring to the fact that WhatsApp has a cloud backup feature built in to the app that keeps a copy of your messages in a cloud storage provider. The Android app at least explicitly warns you that this database is not end-to-end encrypted and that doing this backup defeats the point of the encryption.
Fathers face pressure to provide for their pregnant wife. I too would take five years off to look after babies. However, without a husband of my own, I cannot. The fact is that between my wife and I it makes more sense to invest in my career. We both graduated at the same time with the same degree. We both worked and earned the same salary. Then she wanted a baby. For the past two and a half years we’ve been dealing with very complicated pregnancies and my wife wants more babies. Okay, I love babies, but the fact is that while she could have worked had her pregnancies been easy, she cannot work now. Thus, I work because I am forced to provide for our family, not because I don’t want to spend time with babies. It’s just biological reality that no matter what we do there will be women like my wife who have complicated pregnancies and good men and husbands who will make the sacrifice of not spending time with children to ensure their children’s future. It’s not just about money and food. It’s about securing a place in society, which is not something any government welfare program can provide. Making business connections, colleague relationships, etc now will have a lasting impact on my family’s social status and it simply makes more sense for men to do that. Believe me, my wife and I would love for things to be different, but it really comes down to biology.

When I express my sadness that I have to work and can’t be with children all day long (same complaint my father and grandfather had), my wife often jokes that I should find a husband. It’s funny, but it’s ultimately true. Men are simply better able to do the sorts of things I take on in my family and women at the things my wife does
Yep. It's second in the article :D
I mostly think of awk as a query language for text that's in record format. It doesn't seem to handle unformatted text very well and I tend to use sed or some other kind of tool to clean it up before feeding it to awk.

I like the baked in logic for dealing with record/field formatted data. One thing that seems to hang people up is the lack of a concatenation operator. You just put things next to each other. It looks a little weird. Having associative arrays is a nice plus-up from shell programming.
Really? Seems like they are a theocracy that bullies their neighbors.
There's something that has always nagged at me in these discussions, and I've always been somewhat afraid to bring it up lest I misconstrue, but I want to try and figure this out.

What's a state of affairs we can reach such that people don't become preemptively discriminatory for lack of being able to examine factors that would impact workplace performance? We agree, societally, that we shouldn't discriminate against parents. However, we don't live in a world where a business is spared the cost incurred by that. That's just not the pragmatic reality of employment in the US at this time, and my fear is that if unable to ask "do you have a good track record or gameplan for keeping up with the unique obligations of childbearing" we'll err on the side of paranoia in hiring, and the discrimination will occur in seeking to AVOID potential conflict or discrimination.

A govt stipend was one way I could see this, in that the business doesn't have to bear the cost and can offload work. I'd still be concerned that a business might be afraid about context switching/onboarding, but it's a step forward. Unfortunately, I don't even see that becoming a reality in a country where paternity leave is still the exception and not the rule.

There almost seems to be a "psychological game theory" question lodged in there, apologies if I was rambly in stating it.
Not useless, just not as good as Jarek liked to claim. It's a neat technique but hardly a breakthrough. In the end, for AV1, Daala_EC was better.

As for 'needed a defensive patent', need is the wrong word. In the current patent game, you pile up defensive patents to be used in aggregate as part of a MAD-style game. It sounds insane and it is-- but it's also currently the least insane option.

In any case, Google isn't technically in the wrong here. They're not patenting jarek's work, they're patenting improvements they made to it that were necessary to use it in AV1, which they dedicated engineers to do because Jarek declined to do it himself. Only after Google put in that work (and implemented multiple suggested improvements on top of it to try to bring its performance up) could we even measure the results.

But just because Google isn't wrong, doesn't mean they weren't assholes. Having negotiated with them for things in the past, they will promise the moon and the sun and the sky, and commit only to the parts that get them what they want. I can fully believe they verbally promised Jarek far more than he got.
You may be trying to make a point, but you're not succeeding. A sarcastic guess: robbing banks makes money, and having a job makes money so having a job is logically equivalent to robbing banks and therefore being employed is bad.
Can confirm. It has happened to deaf people at times.
But if you were transporting quite a stash of meth and cocaine I'm sure you'd want to find a lawyer who could convincingly argue confusion.
Funny, the PhD taught me to keep what I say fairly limited. On the other hand, I find generalists will say all kinds of crap that "sounds true" but they have at best a weak theory and a couple of anecdotes.
This is awesome, I had no idea you could use Chrome to run a webpage with JavaScript and dump its contents. Thanks for sharing!
I'm pretty sure anything public on Google Drive is...
Thought experiment: a scrappy startup is struggling to stay profitable, they literally (and not figuratively) can't afford their developers leaving for extended amounts of time and they are very upfront about this in interviews. Are they legally allowed to discriminate against candidates that are pregnant or expressed a desire to have children in the near future since it has the potential to kill their business if they hire them?
My point is that if I know a bit about a subject, I don't fall into the trap of believing that I can point to "an obvious mistake" which was signed off by multiple professional people working in that area, especially in an area like civil engineering where you have criminal liability for "an obvious mistake".

Sure, I would never cross that bridge if I saw those fake stayed cables damaged after a storm, because again, I am in no position to judge the design of the bridge, but it is obviously damaged and I must assume that it is unsafe. Maybe a structural engineer would immediately spot that those stayed cables are fake and wouldn't worry about cross it. But if the bridge looks intact, I would just assume that the engineers knew what they were doing.
You are, obviously, completely correct on the point that you raise. However, I am sold on those sort of laws on the basis that employers are discriminating on irrelevant dimensions, like race and gender.

An employee expecting, or indeed planning, to take a X months leave Y months after joining, with X > Y, isn't an irrelevant dimension. Especially if parental leave is paid, but I don't know if that is how the US handles leave.

An employee literally cannot be a high quality, productive worker if they are not working. Employers should not be forced to ignore relevant factors when hiring. That isn't fair on them.
I don't doubt that that's true, but it's also educated, intelligent people who are more likely to have the kind of money that makes them targets for con men.