In the comics, Swee'Pea is a baby found on Popeye's doorstep (actually delivered to him in a box) in a July 24, 1933 strip. Popeye adopts and raises him as his son, or, as he puts it "boy-kid". Initially, Swee'Pea's speech consisted entirely of the sound "glop". As the years went on, Swee'Pea apparently aged enough to speak normally, and could throw punches if necessary; however, his appearance remained that of a crawling baby. In the strip for August 17, 1933, Popeye christens Swee'Pea as' Scooner Seawell Georgia Washenting Christiffer Columbia Daniel Boom'. Although Swee'Pea remains his most common sobriquet, he is occasionally referred to as Scooner by Popeye and others in later strips.

Based on this passage, who did sweet pea belong to on popeye? Let's think fast. Stream of consciousness:
To answer the question, consider the following: Popeye adopts and raises him as his son, or, as he puts it "boy-kid". So the final answer is Popeye.