Right now as a pulse of light hits your retina (or a radiotelescope nearby) the source of that pulse must be some event at a finite time in the past. That is the Hubble time, which is non-identical to the age of the universe because the Hubble time captures the history of the expansion of space, which is not linear. That nonlinearity makes the present Hubble time greater than the present age of the universe. In general, however, the Hubble time grows in the future and shrinks in the past. One can also think of the Hubble time at any moment as the amount of time it would take the universe to expand to its size at that moment, if the expansion were linear; this is typically how it's taught [*].

If we multiply the Hubble time by c, we have the Hubble length, which is what is being talked about with expressions like "the size of an apple". That is, if the universe was "apple-sized" the Hubble length was on the order of an attoparsec rather than the ~4.4 gigaparsecs today. Extending the Hubble length in all directions gives us a spatial surface at the Hubble time, which means that on (or inside) that surface is everything that generates something which can in principle be detected here-and-now. That's the "size" which is being talked about. Because the Hubble time is shorter in the past, so too is the "size" of the Hubble volume in the past.

It is somewhat easier to think about your question in respect to the universe a bit later, like during Big Bang nucleosynthesis (BBNS) when the Hubble length was on the order of 100 parsecs and adiabatic expansion had cooled the contents of the Hubble volume (Hubble length in every direction) enough that hydrogen, helium, and lithium nuclei could form. Going back to the Hubble time we turn ~100 parsecs into ~300 light years, divde that by c and have a then-Hubble-time of 10 billionths of a second. So something ~speed-of-light (actual light, gravitational radiation, neutrinos, ultrarelativistic charged particles) emitted billionths of a second earlier and at maximum distance (~100 parsecs) would be reaching our galaxy now, billions of years later.

Anything emitted then towards us but from further than that ~100 parsec boundary would not reach us at all. Anything emitted from then but travelling on longer paths (gravitational lensing, interaction with gas or dust, or simply having intrinsic mass like a neutrino) will arrive later. And of course anything emitted then might not be aimed at us, so might never get here at all. So, and I'll touch on this again below, what happens to a BBNS proton that formed about 100 parsecs from protons that ended up in the Milky Way, if it happens to get a kick in a direction away from those Milky-Way-BBNS protons? It becomes unobservable, leaving the observable universe. But what do we call wherever it ends up? And in that region, whatever we all it (the unobservable universe?), are physics the same as in the parts of the observable universe we've studied?

Back to your apple-sized observable universe. We can continue a regression before BBNS, which takes us through epochs where electrons and photons and weak force counterparts are so hot and compressed they become something else (the fields of the electroweak unification), and so on. As we do so the Hubble time is shortened, and so too is the Hubble length. Eventually the Hubble time is so short that gravitational time dilation (from differing concentrations of energy and momentum) becomes a significant fraction. Thus it, and c*Hubble_time -> Hubble_length is subject to fluctuations of significant and growing fractions of the the average (mean or mode). [Ultimately this is where there is an early-universe conflict in the predictions of General Relativity (which is fine with this; we just stop splitting spacetime into spaces indexed by an averaged Hubble time {technically, the derivatives of the Robertson-Walker scale factor \a becomes un-useful}, and index on some other notion of time (or treat the whole region of spacetime as a block) and quantum field theory (which needs a clearer and quite linear notion of time to plug into its equations).]

Regressing further we will calculate a lower-bound Hubble volume which is for all practical purposes infinitesimal, and thus "a single point" is a reasonable description for the observable universe at about that early age. Anything outside the then-infinitesimal Hubble volume will never reach us. Anything inside could in principle, but again lots of that wasn't aimed in our direction and/or ended up becoming "frozen" into non-massless matter (some of which will eventually radiate, like galaxies full of stars, with some of that radiation aimed at us).

> didn't start from a single point; it has always been infinite

We don't know what if anything is outside the Hubble volume at any time, including in the most distant past. It's strange to think that there is nothing at all one parsec beyond the present Hubble length of ~ 4.4 billion parsecs. Where does the light and cosmic rays from distant galaxies radiated radially away from us go? Does each distant galaxy have its own Hubble volume[*], just like each of us locally have? We can only decide to adopt the Copernican principle and say that some distant barred-spiral galaxy has the same view as the Milky Way does -- galaxies scattered similarly in every direction. Adopting that view motivated Guth, one of the originators of studies of cosmic inflation, to calculate that there may be something like 10^23 Hubble volumes "outside" our own.

These would be 10^23 100-parsec volumes adjacent to each other all arranged around our 100-parsec BBNS volume. Each such volume may have the same early expansion history as ours, so (regressing) this would be a lot of apple-sized Hubble volumes, and ultimately a lot of effectively point-sized Hubble volumes. 10^23 is finite, of course, but I don't think we can preclude a much much larger finite number, and we can go so far as to ask questions about a literal infinite number. There will be some lower bound of Hubble volumes adjacent to, outside of, and causally disconnected from our own; that lowest number is much greater than one, however, or cosmic inflation fails to solve certain geometry problems. Additionally, zero ("our Hubble volume is it") raises the hard question of what happens even just a metre beyond the Hubble length. Is it new physics? Would there be some evidence close to but inside the Hubble length? Maybe even in galaxy clusters only tens of millions of parsecs away their radiation or morphology may show proof that they're close to an observer-independent sharp Hubble edge? JWST is already generating relevant data.

Finally, from a comment of yours downthread,

> An apple is only a certain number of atoms, far fewer than the 10^80 that we have today

When the Hubble length was about half the length of an apple, there weren't any atoms at all -- it was too hot and dense for any of the particles you'd recognize in the Standard Model. That's why I chose the BBNS epoch (~100 parsec) instead of the deep elecroweak or GUT epoch at "apple-size" (~ 1 attoparsec). (And additionally atoms can be wholly ionized at much lower temperatures, so I write here about nuclei instead).

At BBNS only hydrogen and some helium and trace lithium atomic nuclei were formed. All the carbon and so forth in an apple today was formed through stellar nucleosynthesis.

There are fewer atomic nuclei today than at the end of BBNS because stellar processes fuse multiple hydrogens into carbon, oxygen and other "non-metal" (in the astronomy sense) nuclei. Also, stellar black holes have taken lots of atomic nuclei out of the picture. And also some nuclei will have taken trajectories away from us and are now further than the present day Hubble length.

In cosmology we talk about the average density of baryons, compared to the averaged density of other things (among them radiation (mostly photons), relativistic and cold neutrinos, dark matter, dark energy). That is we are interested in the fraction of energy at a point that is baryonic (atomic nuclei, and more particularly protons, being the bulk of that). At earlier times, when the Hubble length is shorter, the (averaged, expected, typical) contribution of baryons at a (typical, average) point is higher. Baryonic matter has diluted with the expansion of the universe.

Your 10^80 figure will come from taking this pointwise baryon density and applying it to all the points in space at the present Hubble time, cross-checking it with estimates of the masses of typical galaxies and an estimate of the galaxy count. The baryon density today can also be arrived at by comparing the baryon density at the time the cosmic microwave background radiation formed (that density left imprints in it) and the expansion history since then (which we get by various means).

- --

[*] Unlike the Hubble time as the reciprocal of the Hubble constant, it is useful to think of the Hubble time at a point (in spacetime) as capturing the expansion history at that point. Since gravitation retards the expansion, every point (in spacetime) has its own Hubble time. Importantly, the Hubble time of a hydrogen nucleus peacefully floating in deep inter-galactic-cluster space will be longer than that of a hydrogen nucleus within our sun, even if they were both formed primordially. The gravitation of the sun and the Milky way and other nearby masses means the solar hydrogen has experienced less expansion. This takes us a tiny step closer to understanding the expanding universe in systems of coordinates other than the standard comoving coordinates, which is more relativistic in spirit. It reminds us that in comoving coordinates we are working with uniform densities of matter thanks to averaging, and are explicitly ignoring significant overdensities of matter (which slow down the ticking of clocks nearby).