I'm not sure the OPs approach is reasonable, and I would decline any candidate who tried to put constraints like this on the process. In particular I don't think it's reasonable to insist on being interviewed by the direct manager and I don't think it's reasonable to refuse to engage with algorithm and other whiteboard questions. As a meta-point, a company should try to standardise their interview process as much as possible in order to provide each candidate with an equal chance to succeed. This is very important in attempting to address inherent biases etc.

Apart from filtering out unsuitable candidates, one of the purposes of the interview process is to determine where exactly a good candidate might best fit. Although you get a good guess at this from their CV, oftentimes I find myself suggesting a good candidate be routed to a particular team based on strengths and weaknesses they have demonstrated in the interview process that were not apparent before. There is an information asymmetry at the heart of this that is hard to solve - I know a lot about my company, its teams and their needs, and the candidate (hopefully) knows a lot about themselves, what they are actually good at and want to do. What this means is that the direct manager is not necessarily knowable ahead of the interview (and subsequent review).

Secondly, just because someone is a team lead/manager does not necessarily presuppose they are a particularly good interviewer. Interviewing is a skill that not everyone has and there are good reasons to keep the interview pool to a set of people who see a wide swathe of candidates and are able to calibrate their interview based on experiences of lots of candidates. This means the hiring manager (even if you know who she is) may not be on the slate for interviewing.

Finally the "I'm not prepared to answer questions like x". Although this is becoming a trendy thing to say on HN (most typically about algorithm/whiteboard type questions), refusing to engage constructively with a question is a huge red flag for me, and likely to indicate a personality type that I don't want to hire. I would readily admit that the data shows that algo questions often don't have as much signal as people used to think. However, structured work exercises have been shown time and time again to be a very good indicator of subsequent performance. I do need to know that you can actually program, and often offer a choice of an algo question or a structured code review (where the candidate and I talk through some code on their github or that they can provide me for the review).

There are good and bad ways to ask algo questions as there are with many questions. It's pretty pointless to ask an experienced programmer to write the algo for heapify; they either know it or they don't. However, if you're a recent compSci graduate and you don't know these things I'll suspect that you didn't really pay attention in class and mostly coasted.

One of the hardest and very best interviews I have ever had consisted of someone deliberately re-asking one of the algo questions I had performed poorly on at the beginning of the interview process (after I had a couple of weeks to think about it). This proceeded into a very very tough interview that pushed me beyond the bounds of my compsci knowledge to how I think about solving hard problems. It wasn't about what "book knowledge" I could recall but rather about how good I was at applying my knowledge in a tough situation.