github.com/DataBiosphere/terra-scientific-pipelines-service/SubsetVcfByBedFile
Authors: Terra Scientific Services
Type: Software
Published: 2026-08-03

<h1>Terra Scientific Pipelines Service</h1>
<p><a href="https://sonarcloud.io/summary/new_code?id=DataBiosphere_terra-scientific-pipelines-service"></a></p>
<h2>Overview</h2>
<p>Terra Scientific Pipelines Service, or teaspoons, facilitates running a number of defined scientific pipelines
on behalf of users that users can't run themselves in Terra. The most common reason for this is that the pipeline
accesses proprietary data that users are not allowed to access directly, but that may be used as e.g. a reference panel
for imputation.</p>
<h2>Supported pipelines</h2>
<p>Current supported pipelines are:</p>
<ul>
<li>[in development] Imputation (TODO add link/info)</li>
</ul>
<h2>Architecture</h2>
<p><a href="https://docs.google.com/document/d/1dAPwOG2z1h0B5CszeQ0DfyToniNV_3y1OBV7x7L8ofI/edit?usp=sharing">WIP architecture doc</a>
<a href="https://lucid.app/lucidchart/2f067b5e-2d40-41b4-a5f3-a9dc72d83820/edit?viewport_loc=-72%2C25%2C1933%2C1133%2C0_0&invitationId=inv_97522cca-1b6d-44fe-9552-8f959d410dd7">Linked LucidChart</a></p>
<h2>Development</h2>
<p>This codebase is in initial development.</p>
<h3>Requirements</h3>
<p>This service is written in Java 17, and uses Postgres 13.</p>
<p>To run locally, you'll also need:</p>
<ul>
<li>jq - install with <code>brew install jq</code></li>
<li>vault - see DSP's setup instructions <a href="https://docs.google.com/document/d/11pZE-GqeZFeSOG0UpGg_xyTDQpgBRfr0MLxpxvvQgEw/edit#heading=h.1k9ij99wmle2">here</a>
<ul>
<li>Note that for Step 7, "Create a GitHub Personal Access Token", you'll want to choose
the "Tokens (classic)" option, not the fine-grained access token option.</li>
</ul>
</li>
<li>Java 17 - can be installed manually or through IntelliJ which will do it for you when importing the project</li>
<li>Postgres 13 - multiple solutions here as long as you have a postgres instance running on localhost:5432 the local app will connect appropriately
<ul>
<li>Postgres.app https://postgresapp.com/</li>
<li>Brew https://formulae.brew.sh/formula/postgresql@13</li>
</ul>
</li>
</ul>
<h3>Tech stack</h3>
<ul>
<li>Java 17 temurin</li>
<li>Postgres 13.1</li>
<li>Gradle - build automation tool</li>
<li>SonarQube - static code security and coverage</li>
<li>Trivy - security scanner for docker images</li>
<li>Jib - docker image builder for Java</li>
</ul>
<h3>Local development</h3>
<p>To run locally:</p>
<ol>
<li>Make sure you have the requirements installed from above. We recommend IntelliJ as an IDE.</li>
<li>Clone the repo (if you see broken inputs build the project to get the generated sources)</li>
<li>Run the commands in <code>scripts/postgres-init.sql</code> in your local postgres instance. You will need to be authenticated to access Vault.</li>
<li>Run <code>scripts/write-config.sh</code></li>
<li>Run <code>./gradlew bootRun</code> to spin up the server.</li>
<li>Navigate to <a href="http://localhost:8080/#">http://localhost:8080/#</a></li>
<li>If this is your first time deploying to any environment, be sure to use the admin endpoint <code>/api/admin/v1/updatePipelineWorkspaceId/{pipelineName}/{workspaceId}</code> to set your pipeline's workspace id. Workspace id can be found through the terra ui workspace dashboard or through the Rawls GET workspace endpoint.</li>
</ol>
<h4>Local development with debugging</h4>
<p>If using Intellij (only IDE we use on the team), you can run the server with a debugger. Follow
the steps above but instead of running <code>./gradlew bootRun</code> to spin up the server, you can run
(debug) the App.java class through intellij and set breakpoints in the code.  Be sure to set the
GOOGLE_APPLICATION_CREDENTIALS=config/tsps-sa.json in the Run/Debug configuration Environment Variables.</p>
<h3>Running Tests/Linter Locally</h3>
<ul>
<li>Testing
<ul>
<li>Run <code>./gradlew service:test</code> to run tests</li>
</ul>
</li>
<li>Linting
<ul>
<li>Run <code>./gradlew spotlessCheck</code> to run linter checks</li>
<li>Run <code>./gradlew :service:spotlessApply</code> to apply fix any issues the linter finds</li>
</ul>
</li>
</ul>
<h2>(Optional) Install pre-commit hooks</h2>
<ol>
<li>[scripts/git-hooks/pre-commit] has been provided to help ensure all submitted changes are formatted correctly.  To install all hooks in [scripts/git-hooks], run:</li>
</ol>
<pre><code class="language-bash">git config core.hooksPath scripts/git-hooks
</code></pre>
<h3>Running SonarQube locally</h3>
<p><a href="https://www.sonarqube.org">SonarQube</a> is a static analysis code that scans code for a wide
range of issues, including maintainability and possible bugs. Get more information from
<a href="https://dsp-security.broadinstitute.org/appsec-team-internal/appsec-team-internal/security-activities/sast-1#">DSP SonarQube Docs</a></p>
<p>If you get a build failure due to
SonarQube and want to debug the problem locally, you need to get the sonar token from vault
before running the gradle task.</p>
<pre><code class="language-shell">export SONAR_TOKEN=$(vault read -field=sonar_token secret/secops/ci/sonarcloud/tsps)
./gradlew sonarqube
</code></pre>
<p>Running this task produces no output unless your project has errors. To
generate a report, run using <code>--info</code>:</p>
<pre><code class="language-shell">./gradlew sonarqube --info
</code></pre>
<h3>Connecting to the database</h3>
<p>To connect to the Teaspoons database, we have a script in <a href="https://github.com/broadinstitute/dsp-scripts">dsp-scripts</a> that
does all the setup for you. Clone that repo and make sure you're either on Broad Internal wifi or connected
to the VPN. Then run the following command:</p>
<pre><code class="language-shell">./firecloud/psql-connect.sh dev tsps
</code></pre>
<h3>Deploying to dev</h3>
<p>Upon merging to main, the dev environment will be automatically deployed via the GitHub Action <a href="https://github.com/DataBiosphere/terra-scientific-pipelines-service/actions/workflows/tag-publish.yml">Bump, Tag, Publish, and Deploy</a>
(that workflow is defined <a href="https://github.com/DataBiosphere/terra-scientific-pipelines-service/blob/main/.github/workflows/tag-publish.yml">here</a>).</p>
<p>The two tasks <code>report-to-sherlock</code> and <code>set-version-in-dev</code> will prompt Sherlock to deploy the new version to dev.
You can check the status of the deployment in <a href="https://beehive.dsp-devops.broadinstitute.org/apps/tsps">Beehive</a> and in
<a href="https://ap-argocd.dsp-devops.broadinstitute.org/applications/ap-argocd/tsps-dev">ArgoCD</a>.</p>
<p>For more information about deployment to dev, check out DevOps' <a href="https://docs.google.com/document/d/1lkUkN2KOpHKWufaqw_RIE7EN3vN4G2xMnYBU83gi8VA/">excellent documentation</a>.</p>
<h3>Tracing</h3>
<p>We use <a href="https://opentelemetry.io/">OpenTelemetry</a> for tracing, so that every request has a tracing span that can
be viewed in <a href="https://cloud.google.com/trace">Google Cloud Trace</a>. (This is not yet fully set up here - to be done in TSPS-107).
See <a href="https://broadworkbench.atlassian.net/wiki/x/AoGlrg">this DSP blog post</a> for more info.</p>
<h3>Running the end-to-end tests</h3>
<p>The end-to-end test is specified in <code>.github/workflows/run-e2e-tests.yaml</code>. It calls <a href="https://github.com/broadinstitute/dsp-reusable-workflows/blob/main/e2e-test/tsps_e2e_test.py">the test script defined
in the dsp-reusable-workflows repo</a>.</p>
<p>The end-to-end test is automatically run nightly on the dev environment.</p>
<p>To run the test against a specific feature branch:</p>
<ol>
<li>Grab the image tag for your feature branch.</li>
</ol>
<blockquote>
<p>If you've opened a PR, you can find the image tag as follows:</p>
<ul>
<li>go to the Bump, Tag, Publish, and Deploy workflow that's triggered each time you push to your branch</li>
<li>From there, go to the tag-publish-docker-deploy task</li>
<li>Expand the "Construct docker image name and tag" step</li>
<li>The first line should contain the image tag, something like "0.0.81-6761487".</li>
</ul>
</blockquote>
<ol>
<li>Navigate to the <a href="https://github.com/DataBiosphere/terra-scientific-pipelines-service/actions/workflows/run-e2e-tests.yaml">e2e-test GHA workflow</a></li>
<li>Click on the "Run workflow" button and select your branch from the dropdown</li>
</ol>
<ul>
<li>Enter the image tag from step 1 in the "Custom image tag" field</li>
<li>If you've updated the end-to-end test in the dsp-resuable-workflows repo, enter either a commit hash or your git
branch name. If you don't need to change the test, leave the default as main.</li>
</ul>
<ol>
<li>Click the green "Run workflow" button.</li>
</ol>