It is the unrestricted digital lending that crosses the line IMO, yes. (And my opinion is very much not firm and I'm open to being convinced in either direction.)

First, authors get revenue from books in libraries. When you read "Pat the Bunny," the library had to actually own a copy - and in particular your local library had to own a copy, meaning thousands and thousands of copies got sold to libraries. For each physical book, only one 4-year-old at a time could read it, so for popular books (e.g., when you turned 14 and had the same summer reading list as everyone in your school district) each library tends to buy many copies.

So, yes, libraries don't implement a strict "each person who wants to read causes the author to get paid" property, but they do provide a meaningful balance between getting paid for writing a popular book and also making books available in a cost-effective way to large numbers of people. Authors get quite a bit more than $0, even if they don't get money from each reader. And I think that's a good place to make the argument that these aren't lost sales - libraries make books available to people who often otherwise wouldn't pay for them and wouldn't read them.

Second, authors know how libraries work. They've been around forever. When you decide "I'm gonna have a career based on writing," you know approximately how the market works. You know that if you write a book that's about as good as this other book, you'll get this much in revenue from purchases by readers, that much from purchases by libraries, etc., and you expect that to be reasonably stable. The Archive is changing the economics suddenly, and at a particularly bad time to have your career's economics change. I know for my part that I'd be upset if some random non-profit said "During this time of crisis, we've found a possibly-legal way to allow companies to not spend money on SREs and we're gonna do it," and I don't think, just because I don't know how that could possibly work, that my livelihood is inherently more worthy of stability than the livelihoods of authors. (And I'd still be upset with the non-profit even if I didn't think it was realistic that my company would go for that option.)

As a consequence, my feelings are a lot different here with regards to works published by someone who makes a living doing something else like being a professor, and happens to write books, than works published by someone who relies on copyright for their living. I'm also okay with, say, Sci-Hub, because while it threatens Elsevier's business model, it doesn't (as far as I know) threaten the livelihood of researchers themselves.

The website for Controlled Digital Lending justifies the practice in two ways (https://controlleddigitallending.org/): one, it preserves the market properties that libraries have previously relied on, and two, it specifically solves the problem of books unavailable in e-book form. I'm fine with both of those, particularly because they both address the question of whether the author/rightsholder needs to consent to this use. (Because it preserves the market properties the impact on an author isn't different, and in the case of abandoned works, the author or published might no longer be around anyway, and unofficial e-books certainly aren't messing with an existing market.) I am also fine with uncontrolled digital lending with the consent of the author - if the arguments you're making about it not affecting the market hold up (and I think they do, to be clear), why not simply invite authors to join on an opt-in basis?

(I suspect I'd even be okay with it if they said, hey authors, you don't have the rights to make e-books of your work, but that's okay, we have this legal workaround we're willing to try. If you want people reading e-books but your publisher isn't cooperating, we'll fight the legal fight alongside you.)