File size: 4,048 Bytes
e1b2e7e e2b9e50 6e8e5ab |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
---
license: other
base_model:
- deepseek-ai/Janus-1.3B
pipeline_tag: any-to-any
library_name: transformers.js
tags:
- text-to-image
- image-to-text
- image-text-to-text
---
https://huggingface.co/deepseek-ai/Janus-1.3B with ONNX weights to be compatible with Transformers.js.
## Usage (Transformers.js)
If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library from [NPM](https://www.npmjs.com/package/@huggingface/transformers) using:
```bash
npm i @huggingface/transformers
```
**Example:** Image+text to text
```js
import { AutoProcessor, MultiModalityCausalLM } from "@huggingface/transformers";
// Load processor and model
const model_id = "onnx-community/Janus-1.3B-ONNX";
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await MultiModalityCausalLM.from_pretrained(model_id);
// Prepare inputs
const conversation = [
{
role: "User",
content: "<image_placeholder>\nConvert the formula into latex code.",
images: ["https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/quadratic_formula.png"],
},
];
const inputs = await processor(conversation);
// Generate response
const outputs = await model.generate({
...inputs,
max_new_tokens: 150,
do_sample: false,
});
// Decode output
const new_tokens = outputs.slice(null, [inputs.input_ids.dims.at(-1), null]);
const decoded = processor.batch_decode(new_tokens, { skip_special_tokens: true });
console.log(decoded[0]);
```
Sample output:
````
Sure, here is the LaTeX code for the given formula:
```
x = \frac{-b \pm \sqrt{b^2 - 4a c}}{2a}
```
This code represents the mathematical expression for the variable \( x \).
````
**Example:** Text to image
```js
import { AutoProcessor, MultiModalityCausalLM } from "@huggingface/transformers";
// Load processor and model
const model_id = "onnx-community/Janus-1.3B-ONNX";
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await MultiModalityCausalLM.from_pretrained(model_id);
// Prepare inputs
const conversation = [
{
role: "User",
content: "A cute and adorable baby fox with big brown eyes, autumn leaves in the background enchanting,immortal,fluffy, shiny mane,Petals,fairyism,unreal engine 5 and Octane Render,highly detailed, photorealistic, cinematic, natural colors.",
},
];
const inputs = await processor(conversation, { chat_template: "text_to_image" });
// Generate response
const num_image_tokens = processor.num_image_tokens;
const outputs = await model.generate_images({
...inputs,
min_new_tokens: num_image_tokens,
max_new_tokens: num_image_tokens,
do_sample: true,
});
// Save the generated image
await outputs[0].save("test.png");
```
Sample outputs:
|  |  |  |  |
|---|---|---|---|
|  |  |  |  |
Want to play around with the model? Check out the [online WebGPU demo](https://huggingface.co/spaces/webml-community/Janus-1.3B-WebGPU).
<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/61b253b7ac5ecaae3d1efe0c/KftT9sqguYRC4O7K0vwAP.mp4"></video>
|