The other way I have heard the same idea, but phrased with slightly different motivation:

Your children are important part of your life, and you are important part of their life (especially in the early years). This has also an implication: Who actually is this person they get to know?

I don't have kids, but coincidentally discussed these kind of childhood memories with my spouse quite recently after someone made similar complaint. I probably hsve no idea of time commitments required of very small children. However, about at the age we were old enough not to be disruptive (if supplied with reading material when necessary), both our parents took us to their adult things: bird watcher society meetings, social events, art museums, bookshops, and seminars. It is probable our parents did their peronal own things less than they would have if we had not existed, but we agreed we did not suffer from it, and learned quite a deal.

There is anyway the peculiar something about way kids' experience thing and what of them remain when we grow anyway. Undoubtledly I have learned things much useful I don't remember much about, like how to walk. For me, the images I can call up are the odd, random, small ones. To a child many things are novel in this world, and one knows there is lots of things in adults' world to uncover. Yet because of that lack of perspective, various things may be treated as the normal. As an added complication, one's perspective on childhood memories changes and gains different color when one grows up.

Once, while visiting relatives on a road-trip, tthere was half a day long detour to see an exhibition of non-figurative sculptures whose details I don't recall. I do remember the calm, cool, air-conditioned atmosphere, the white bright walls, the smell of the museum, and the impression of uncanny puzzlement and imcomprehension left by the lone melancholy video installation in a small, separate darkened room in one of the exhibition halls. I was about 12 years old and quite bored. I surmise I was brought along mostly because father wanted to go, maybe because he wanted to give me educational experience about modern art, but most certainly there was any no other option suitable for preteen's inclinations given the circumstances of road trip, and as I said, he wanted to go. Nevertheless, despite the boredom, today the memory certainly is far more interesting than what I recall of the most of the kid-oriented activities I liked then better and also took lot of my parents' time.

It is difficult for me to judge the value of that museum experience -- is it because of the exhibition was remarkable or does it feel remarkable because I remember it? It might even be a partially constructed memory from several similar museum trips. And after all, the thing I remember especially well, much better and concretely than anything else, one of my earliest childhood memories (that I have photographied evidence of), is something utterly mundane, namely the large, black, late 80s mat we had on the living room. In retrospect, it was horrendously ugly on its own right as soon as the TRON aesthetic went out of style, but the reason I remember it was that it had this repeating, neon-colored rectangle pattern, rows and columns, one symmetrically after other. To my eternal dislike, the pattern repeated regularly until it did not: the final row of rectangles at one end were shorter than they should have been given the rest of the pattern. No matter how I counted or measured or tried to mentally rotate the damned thing in my head, there was no solution, no escape, the thing was always asymmetrical. (I amuse myself thinking this may have something to do with me deciding to study math and CS in university later.) Thus one of possibly life-defining experiences was the rug my parents bought years before I was born, and presumably thrown away while I was in primary school (it disappeared and I never inquired about it).

Of childrens' fiction authors, I think ones such as Roald Dahl, Dickens, Lemony Snicket, maybe Rowling (the very first chapters of Philosopher's Stone) capture something of this aspect to watching the world unfold.