> So all neglect laws should be removed?

That was the position of Murray Rothbard [1], who was one of the major intellectual founders of modern libertarianism, particularly anarcho-capitalism. He was a founder of the Ludwig von Mises Institute. Most internet libertarians seem to be followers of his school of libertarianism (although they often do not know it...).

His book "The Ethics of Liberty" covers children in chapter 14. You can find the PDF online. There's a big chunk of chapter 14 here [2]. Here's the most relevant part:

Even from birth, the parental ownership is not
absolute but of a "trustee" or guardianship kind. In
short, every baby as soon as it is born and is
therefore no longer contained within his mother's
body possesses the right of self-ownership by virtue
of being a separate entity and a potential adult. It
must therefore be illegal and a violation of the
child's rights for a parent to aggress against his
person by mutilating, torturing, murdering him, etc.
On the other hand, the very concept of "rights" is a
"negative" one, demarcating the areas of a person's
action that no man may properly interfere with. No
man can therefore have a "right" to compel someone
to do a positive act, for in that case the
compulsion violates the right of person or property
of the individual being coerced. Thus, we may say
that a man has a right to his property (i.e., a
right not to have his property invaded), but we
cannot say that anyone has a "right" to a "living
wage," for that would mean that someone would be
coerced into providing him with such a wage, and
that would violate the property rights of the people
being coerced. As a corollary this means that, in
the free society, no man may be saddled with the
legal obligation to do anything for another, since
that would invade the former's rights; the only
legal obligation one man has to another is to
respect the other man's rights.

Applying our theory to parents and children, this
means that a parent does not have the right to
aggress against his children, but also that the
parent should not have a legal obligation to feed,
clothe, or educate his children, since such
obligations would entail positive acts coerced upon
the parent and depriving the parent of his rights.
The parent therefore may not murder or mutilate his
child, and the law properly outlaws a parent from
doing so. But the parent should have the legal right
not to feed the child, i.e., to allow it to die. The
law, therefore, may not properly compel the parent
to feed a child or to keep it alive. (Again, whether
or not a parent has a moral rather than a legally
enforceable obligation to keep his child alive is a
completely separate question.) This rule allows us
to solve such vexing questions as: should a parent
have the right to allow a deformed baby to die
(e.g., by not feeding it)? The answer is of course
yes, following a fortiori from the larger right to
allow any baby, whether deformed or not, to die.
(Though, as we shall see below, in a libertarian
society the existence of a free baby market will
bring such "neglect" down to a minimum.)

[1] http://en.wikipedia.org/wiki/Murray_Rothbard

[2] http://mises.org/library/children-and-rights