A former Goldman Sachs Group Inc. programmer was convicted in a split verdict of stealing the bank’s “secret sauce” high-frequency trading code, ending a tumultuous trial but leaving open the chance the yearslong saga could continue.

A New York state court jury in Manhattan on Friday convicted Sergey Aleynikov on one count and acquitted him on another. The jury was unable to reach a decision on a third count. Justice Daniel Conviser dismissed the jury Friday afternoon. Mr. Aleynikov, 45 years old, faces the possibility of up to four years in prison.

The conviction was the second time in five years a jury concluded that Mr. Aleynikov stole portions of Goldman’s computer code. He was convicted in federal court in 2010, but acquitted by an appellate court in 2012. Manhattan District Attorney Cyrus R. Vance Jr. charged him under state laws in 2012.

The latest trial was marked by the arcane and the bizarre. Jurors appeared to struggle with how to interpret and apply laws designed well before the computer age, asking several times for charges to be read and for definitions of various words. Midway through deliberations one juror accused another of poisoning her lunch. Both were dismissed by the judge.

Even the trial arguments were unusual. Both sides agreed Mr. Aleynikov stole the data. But the defense argued he didn’t break the law, just Goldman’s confidentiality policy. Mr. Aleynikov’s lawyer, Kevin Marino, repeatedly accused prosecutors of acting in bad faith, prompting outrage from the district attorney’s office.

Mr. Marino said the split verdict was proof the jury didn’t understand the charges, and that Justice Conviser shouldn’t have allowed the jury to deliberate over a case whose core debate is a legal matter, not a factual one.

“You’ve had a jury struggling with this because they’re performing a judicial function they’re ill-equipped to perform,” Mr. Marino said.

Mr. Vance praised the verdict and said his office would continue to pursue similar cases to “ensure the integrity and fairness of our marketplaces.”

“By allowing individuals, corporations, and government entities to safeguard valuable information, we are encouraging and supporting the research, innovation, and entrepreneurship that is so important to our city in an increasingly competitive digital and global market,” Mr. Vance said in a statement.

Mr. Aleynikov emerged as a hero in some circles, in part because of his prominent feature in Michael Lewis’s best-selling book “Flash Boys,” about high-speed trading. Some in legal circles felt his second prosecution was overly aggressive, making Friday’s verdict a victory for Mr. Vance, if an incomplete one. But, as with the first trial, the verdict may not be the end for Mr. Aleynikov.

Justice Conviser said Friday that he would rule within six weeks on a defense motion to dismiss the case on grounds of legal insufficiency. The judge has previously appeared to express doubt about whether the decades-old state law under which Mr. Aleynikov was charged applies to his conduct. The defense also could appeal if the judge rules against the motion.

The appeals court had acquitted Mr. Aleynikov because it said federal law didn’t apply to the case. Some legal analysts took that ruling as an invitation for the Manhattan district attorney’s office to charge him under state laws, a move that didn’t trigger double jeopardy immunity.

Mr. Aleynikov served a year in prison after his first conviction. He went through a divorce and lost his career as the case has unfolded. He walked around the courtroom anxiously after the verdict was read Friday and the jury had left. On Friday, the judge released him from bail conditions and said he wasn’t a flight risk.

The trial was unusual in that both sides almost entirely agreed on the facts, but disputed whether New York state law applies. Two of the charges accuse Mr. Aleynikov of unlawfully using secret scientific material to appropriate the “major portion” of its economic value, and a third charged him with duplicating it for his own use.

The jury acquitted Mr. Aleynikov on the duplication charge, convicted him on one secret scientific-material count and couldn’t reach a verdict on the other. He was charged twice on that front because prosecutors alleged he stole bits of the code on two separate days during his last week at Goldman Sachs.

Throughout the deliberations the jury appeared to wrestle with the law. The jury sent repeated notes to Justice Conviser, requesting clarifications on legal terms and asking to have his instructions on the law read back to them.

The jury was further distracted when a quarrel between two jurors erupted this past week when a female juror accused a male juror of poisoning her lunch. She told the judge she became suspicious when an avocado went missing from her lunch, and that her concerns were amplified when the male juror used a pharmaceutical company as an example during deliberations. Justice Conviser said he had concluded the accusations had “absolutely no basis in fact” and dismissed the two jurors before allowing the remaining 10 to continue deliberations.

Mr. Marino, Mr. Aleynikov’s attorney, has alternated between antagonizing the judge and praising him, and Friday was no different. After the jury had been dismissed, Mr. Marino urged the judge to give his ruling on dismissing the case Friday, sparking a rebuke from Justice Conviser.

“I’d be embarrassed by this verdict if I were you,” Mr. Marino said.

“Well, I’m not,” Justice Conviser replied.

Later, Mr. Marino said he had the utmost respect for the judge and appreciated his handling of the trial.

Mr. Marino has argued that because Mr. Aleynikov didn’t appropriate the code to Goldman’s detriment, and because he didn’t intend to profit from it, he didn’t break the law. Goldman continued to use the code and earned profits from its high-frequency trading system after Mr. Aleynikov transferred some of the code, he said.

The judge appeared sympathetic to another argument that Mr. Aleynikov didn’t illegally duplicate the code because he didn’t make a “tangible” copy of the code, as binary computer bytes don’t have a physical form.

Mr. Marino has long alleged Goldman flexed its political muscles to persuade prosecutors to target Mr. Aleynikov. The bank’s shadow loomed throughout the proceedings, and Mr. Marino frequently referred to the allegation, even pointing out that Goldman lawyers were sitting in the courtroom.

Goldman has previously declined to comment on these allegations.

Write to Christopher M. Matthews at christopher.matthews@wsj.com

Corrections & Amplifications

The jury was unable to reach a decision on the third count. An earlier version of this story misstated that the jury acquitted Mr. Aleynikov on that count.