You're absolutely right, Michael. I have some people who have reviewed many of my book manuscripts. I keep going back to them because they're so valuable. They always have the right to say no when I ask, though.
It's fairly easy to find people who say, "Sure, I can review your manuscript." It's far harder to find people who give you lots of helpful feedback, on time, that really improve the work.
On each book, I invite some new people who I haven't worked with before to participate. A few turn out to be gems, others provide no feedback at all, and everything in between.