---
license: cc-by-nc-nd-4.0
library_name: transformers
extra_gated_prompt: >-
  This model and associated code are released under the CC-BY-NC-ND 4.0 license
  and may only be used for non-commercial, academic research purposes with
  proper attribution. Any commercial use, sale, or other monetization of the
  FEMI Model and its derivatives, which include models trained on outputs from
  the FEMI Model or datasets created from the FEMI Model, is prohibited and
  requires prior approval.


  Please note that the primary email used to sign up for your Hugging Face
  account must match your institutional email to receive approval. By
  downloading the FEMI Model, you attest that all information (affiliation,
  research use) is correct and up-to-date. Downloading the FEMI Model requires
  prior registration on Hugging Face and agreeing to the terms of use. By
  downloading the FEMI Model, you agree not to distribute, publish, or reproduce
  a copy of the FEMI Model. If another user within your organization wishes to
  use the FEMI Model, they must register as an individual user and agree to
  comply with the terms of use. If you are a commercial entity, please contact
  the corresponding author.


  Further, by downloading the FEMI Model, you agree you will only use the FEMI
  Model for academic research purposes and will not use, or allow others to use,
  the FEMI Model to:


  - Diagnose, cure, mitigate, treat, or prevent disease or any other conditions,
  including for Investigational Use Only (“IUO”), Research Use Only (“RUO”),
  commercial, clinical, or other similar use, and including as a substitute for
  professional medical advice, a healthcare opinion, a diagnosis, treatment, or
  the clinical judgment of a healthcare professional, as no license or right is
  granted for any such purposes.


  - Re-identify the deidentified data used to develop the FEMI Model.


  - Violate the law or others’ rights, including to:
    - Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content;
    - Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals;
    - Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services;
    - Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices;
    - Collect, process, disclose, generate, or infer the identity of individuals or the health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws;
    - Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the FEMI Model or any related materials;
    - Create, generate, or facilitate the creation of malicious code, malware, computer viruses, or do anything else that could disable, overburden, interfere with, or impair the proper working, integrity, operation, or appearance of a website or computer system.

  - Engage in, promote, incite, facilitate, or assist in the planning or
  development of activities that present a risk of death or bodily harm to
  individuals, including the use of the FEMI Model as a medical device, clinical
  support, diagnostic tool, or other technology intended to be used in the
  diagnosis, cure, mitigation, treatment, or prevention of disease or other
  conditions, including for Investigational Use Only (“IUO”), Research Use Only
  (“RUO”), commercial, clinical, or similar use.


  - Intentionally deceive or mislead others, including representing that the use
  of the FEMI Model or its outputs is human-generated.


  Further, you agree that you will appropriately disclose to end users any known
  dangers of your AI system.
extra_gated_fields:
  First Name: text
  Last Name: text
  Affiliation: text
  Model Access Purpose: text
  Job title:
    type: select
    options:
    - Student
    - Research Graduate
    - AI researcher
    - AI developer/engineer
    - Clinician
    - Other
tags:
- healthcare
- ivf
---

# FEMI: A Foundation Model for Embryology

FEMI is a foundation model for embryology, trained on time-lapse images of embryos. It leverages the Vision Transformer (ViT) architecture with Masked Autoencoder (MAE) pretraining to learn rich representations from embryonic images. The model is available in both PyTorch and TensorFlow implementations and can be fine-tuned for various downstream tasks in in vitro fertilization (IVF) research.

## Model Description

- **Architecture**: Vision Transformer (ViT) with Masked Autoencoder (MAE) pretraining.
- **Input Modalities**: Time-lapse images or videos of embryos.
- **Frameworks**: Available in both PyTorch and TensorFlow.
- **Pretraining Objective**: Self-supervised learning using MAE, where patches of the input images are masked and the model learns to reconstruct them.

## How to Use

You can use the FEMI model with the Hugging Face Transformers library.

### Example

```python
from transformers import ViTMAEForPreTraining, AutoImageProcessor
from PIL import Image
import requests
from huggingface_hub import login

login()

# Load the model
model = ViTMAEForPreTraining.from_pretrained("surajraj99/FEMI")

# Load the image processor
processor = AutoImageProcessor.from_pretrained("surajraj99/FEMI")

# Load an example image
url = "http://example.com/path_to_embryo_image.png"
image = Image.open(requests.get(url, stream=True).raw)

# Preprocess the image
inputs = processor(images=image, return_tensors="pt")

# Forward pass
outputs = model(**inputs)
loss = outputs.loss
reconstructed_pixel_values = outputs.logits
```

## Intended Uses & Limitations

### Intended Uses

FEMI is intended for research and development in embryology and IVF. Potential use cases include:

- **Classification Tasks**: Predicting embryo viability, classifying developmental stages, etc.
- **Regression Tasks**: Estimating developmental progress, predicting time to specific milestones.
- **Feature Extraction**: Using embeddings for downstream analysis or as input to other models such as image segmentation, witnessing, etc.

### Limitations

- **Data Requirements**: Effective fine-tuning requires a sufficient amount of labeled data.
- **Domain Specificity**: The model is specialized for embryology; performance may degrade on unrelated tasks.
- **Ethical Considerations**: Ensure compliance with ethical guidelines when using the model, especially in clinical settings.

## Training Data

The FEMI model was pretrained on a large dataset of time-lapse images of embryos collected during IVF procedures. The data encompasses various developmental stages, capturing the dynamic processes of embryo development.

## Training Procedure

- **Pretraining**: The model was pretrained using a Masked Autoencoder approach, masking random patches of images and training the model to reconstruct them.
- **Optimizer**: Adam optimizer with a learning rate schedule.
- **Epochs**: The pretraining was conducted over multiple epochs to ensure convergence.
- **Framework**: Both PyTorch and TensorFlow implementations are available.


## Terms of Use

This model and associated code are released under the **CC-BY-NC-ND 4.0** license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of the **FEMI Model** and its derivatives, which include models trained on outputs from the FEMI Model or datasets created from the FEMI Model, is prohibited and requires prior approval. Please note that the primary email used to sign up for your Hugging Face account must match your institutional email to receive approval. By downloading the FEMI Model, you attest that all information (affiliation, research use) is correct and up-to-date. Downloading the FEMI Model requires prior registration on Hugging Face and agreeing to the terms of use. By downloading the FEMI Model, you agree not to distribute, publish, or reproduce a copy of the FEMI Model. If another user within your organization wishes to use the FEMI Model, they must register as an individual user and agree to comply with the terms of use. If you are a commercial entity, please contact the corresponding author.

Further, by downloading the FEMI Model, you agree you will only use the FEMI Model for academic research purposes and will not use, or allow others to use, the FEMI Model to:

1. **Diagnose, cure, mitigate, treat, or prevent disease or any other conditions**, including for Investigational Use Only (“IUO”), Research Use Only (“RUO”), commercial, clinical, or other similar use, and including as a substitute for professional medical advice, a healthcare opinion, a diagnosis, treatment, or the clinical judgment of a healthcare professional, as no license or right is granted for any such purposes.

2. **Re-identify the deidentified data used to develop the FEMI Model**.

3. **Violate the law or others’ rights**, including to:

    **a.** Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content;

    **b.** Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals;

    **c.** Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services;

    **d.** Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices;

    **e.** Collect, process, disclose, generate, or infer the identity of individuals or the health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws;

    **f.** Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the FEMI Model or any related materials;

    **g.** Create, generate, or facilitate the creation of malicious code, malware, computer viruses, or do anything else that could disable, overburden, interfere with, or impair the proper working, integrity, operation, or appearance of a website or computer system.

4. **Engage in, promote, incite, facilitate, or assist in the planning or development of activities that present a risk of death or bodily harm to individuals**, including the use of the FEMI Model as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions, including for Investigational Use Only (“IUO”), Research Use Only (“RUO”), commercial, clinical, or similar use.

5. **Intentionally deceive or mislead others**, including representing that the use of the FEMI Model or its outputs is human-generated.

Further, you agree that you will appropriately disclose to end users any known dangers of your AI system.


## Citation

If you use FEMI in your research, please cite:

TBA

## Contact

For specific questions or inquiries, please contact:

- **Suraj Rajendran**: [sur4002@med.cornell.edu](mailto:sur4002@med.cornell.edu)
- **Iman Hajirasouliha**: [imh2003@med.cornell.edu](mailto:imh2003@med.cornell.edu)