Domain: Biology — Ecology

Problem:
Fit a joint hierarchical Bayesian multi-omics model to identify and quantify microbial taxa and metabolite markers from the provided dataset (bv_multiomics.csv) that are directly associated with bacterial vaginosis (BV). In the data provided, each subject has microbial count data, metabolite concentrations, and clinical covariates recorded. The result should account for compositionality of taxa data, log-skewness of metabolite concentrations, subject covariates (age, sexual activity), and Sparsity in feature effects.
Identify the taxa directly associated with BV; metabolites associated with BV; account for latent community states (through a low-rank structure or factor model). 

A hierarchical Bayesian logistic regression is used for the BV outcome, where regression coefficients for microbial taxa and metabolites are given hierarchical shrinkage priors (e.g., horseshoe or Laplace priors) to promote sparsity and interpretability.
Model details:
\( BV_i \sim \text{Bernoulli}(p_i), \quad \text{logit}(p_i) = \alpha + \beta_X X_i + \beta_Z Z_i + \gamma_1 \,\text{Age}_i + \gamma_2 \,\text{SexAct}_i + \varepsilon_i \)
- \(X_i\): vector of microbial taxa features for subject \(i\)  
- \(Z_i\): vector of metabolite features for subject \(i\)  

Priors for coefficients:  
\(\beta_X, \beta_Z \sim \text{Horseshoe}(0, \tau)\) or \(\mathcal{N}(0, \tau^2)\) for sparsity  

Error term:  
\(\varepsilon_i \sim \mathcal{N}(0, \sigma^2)\)


Perform posterior inference using MCMC sampling (\(4\)  chains, \(2000\) iterations per chain) or an equivalent variational Bayes approximation. Estimate posterior means and \(95\%\) credible intervals for all microbial and metabolite effects on BV. Fix random seed = \(42\) for reproducibility and report all estimates with precision up to \(3\) decimal places.

Finally, identify the top-most microbial taxon and top-most metabolite most strongly associated with BV (by absolute effect size). Provide the output in the following array format: [EstimateEffect], [EstimateEffect]

Answer: Taxon20 and Metab07 [\(-0.534\)], [\(-0.513\)]

Solution code:
# bv_pipeline_compliant.py
"""
- Adds 2 PCA components from CLR (via manual SVD)
- Fits L2-penalized logistic regression using SciPy optimize
- Performs B=200 deterministic bootstrap resamples (seed=42)
- Reports top taxon and top metabolite (mean across bootstrap)
"""

import os
import random
import numpy as np
import pandas as pd
from pathlib import Path
from scipy.optimize import minimize
from scipy.special import expit  # sigmoid

# -------------------------
# Config
# -------------------------
DATA_PATH = "bv_multiomics.csv"
B = 200
SEED = 42
PSEUDOCOUNT = 1e-6
LAMBDA = 1.0  # L2 regularization strength

# -------------------------
# Reproducibility setup
# -------------------------
os.environ["PYTHONHASHSEED"] = str(SEED)
random.seed(SEED)
np.random.seed(SEED)

# -------------------------
# Load data
# -------------------------
p = Path(DATA_PATH)
if not p.exists():
    raise FileNotFoundError(f"{DATA_PATH} not found.")
df = pd.read_csv(p)

# -------------------------
# Detect outcome column
# -------------------------
possible_target_names = ["BV", "BV_Status", "BVStatus", "bv_status"]
target_col = next((c for c in possible_target_names if c in df.columns), None)
if target_col is None:
    raise ValueError("BV outcome column not found.")
y_raw = df[target_col]
y = (y_raw >= 0.5).astype(int).values if y_raw.max() > 1 else y_raw.astype(int).values

# -------------------------
# Detect taxa and metabolite columns
# -------------------------
taxa_cols = [c for c in df.columns if "_RelAbund" in c]
metab_cols = [c for c in df.columns if "_Conc" in c]

# -------------------------
# Covariates
# -------------------------
covariates = []
if "Age" in df.columns:
    covariates.append("Age")
if "SexualActivity" in df.columns:
    covariates.append("SexualActivity")

# -------------------------
# CLR transform taxa
# -------------------------
taxa_matrix = df[taxa_cols].fillna(PSEUDOCOUNT).clip(lower=PSEUDOCOUNT).values
log_taxa = np.log(taxa_matrix)
clr_taxa = log_taxa - log_taxa.mean(axis=1, keepdims=True)
clr_df = pd.DataFrame(clr_taxa, columns=[c+"_CLR" for c in taxa_cols])

# -------------------------
# Log-transform metabolites
# -------------------------
metab_matrix = df[metab_cols].fillna(0.0).values
log_metab = np.log1p(metab_matrix)
metab_df = pd.DataFrame(log_metab, columns=[c+"_log" for c in metab_cols])

# -------------------------
# PCA (manual SVD) for latent taxa components
# -------------------------
def pca_numpy(X, n_components=2):
    X_centered = X - X.mean(axis=0)
    U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)
    return U[:, :n_components] * S[:n_components]

pcs = pca_numpy(clr_taxa, n_components=2)
pcs_df = pd.DataFrame(pcs, columns=["PC1_taxa", "PC2_taxa"])

# -------------------------
# Combine features
# -------------------------
design_df = pd.concat([clr_df, metab_df, df[covariates].reset_index(drop=True), pcs_df], axis=1)
X_design = design_df.fillna(0.0).values
feature_names = design_df.columns.tolist()

# -------------------------
# Manual standardization
# -------------------------
X_mean = X_design.mean(axis=0)
X_std = X_design.std(axis=0, ddof=1)
X_std[X_std == 0] = 1.0
Xs = (X_design - X_mean) / X_std

# -------------------------
# Logistic regression via SciPy optimization
# -------------------------
def fit_logistic_l2(X, y, lam=LAMBDA):
    n_features = X.shape[1]
    def neg_log_posterior(beta):
        p = expit(X @ beta)
        eps = 1e-9
        log_lik = y * np.log(p + eps) + (1 - y) * np.log(1 - p + eps)
        penalty = 0.5 * lam * np.sum(beta**2)
        return -np.sum(log_lik) + penalty
    res = minimize(neg_log_posterior, np.zeros(n_features), method="L-BFGS-B")
    return res.x

# -------------------------
# Bootstrap
# -------------------------
rng = np.random.default_rng(SEED)
coefs = np.zeros((B, Xs.shape[1]))

for b in range(B):
    idx = rng.choice(len(y), size=len(y), replace=True)
    Xb = Xs[idx]
    yb = y[idx]
    coefs[b, :] = fit_logistic_l2(Xb, yb, lam=LAMBDA)

coef_mean = coefs.mean(axis=0)

# -------------------------
# Find top taxa and metabolite
# -------------------------
taxa_names = [n for n in feature_names if n.endswith("_CLR")]
metab_names = [n for n in feature_names if n.endswith("_log")]

top_taxon = max(taxa_names, key=lambda n: abs(coef_mean[feature_names.index(n)]))
top_metab = max(metab_names, key=lambda n: abs(coef_mean[feature_names.index(n)]))

top_taxon_value = round(float(coef_mean[feature_names.index(top_taxon)]), 3)
top_metab_value = round(float(coef_mean[feature_names.index(top_metab)]), 3)

# -------------------------
# Final output
# -------------------------
print(f'["{top_taxon}", {top_taxon_value}], ["{top_metab}", {top_metab_value}]')