Sylvain Filoni

fffiloni

AI & ML interests

ML for Animation

Blog posts

Organizations

fffiloni's activity

replied to their post 16 days ago
view reply

Nice ๐Ÿ˜Š Have you tried other models yet ? My favorite is Mustango so far, really good with NL prompts

posted an update 19 days ago
view post
Post
I'm happy to announce that โœจ Image to Music v2 โœจ is ready for you to try and i hope you'll like it too ! ๐Ÿ˜Œ

This new version has been crafted with transparency in mind,
so you can understand the process of translating an image to a musical equivalent.

How does it works under the hood ? ๐Ÿค”

First, we get a very literal caption from microsoft/kosmos-2-patch14-224; this caption is then given to a LLM Agent (currently HuggingFaceH4/zephyr-7b-beta )which task is to translate the image caption to a musical and inspirational prompt for the next step.

Once we got a nice musical text from the LLM, we can send it to the text-to-music model of your choice:
MAGNet, MusicGen, AudioLDM-2, Riffusion or Mustango

Instead of the previous version of Image to Music which used Mubert API, and could output curious and obscure combinations, we only provide open sourced models available on the hub, called via the gradio API.

Also i guess the music result should be more accurate to the atmosphere of the image input, thanks to the LLM Agent step.

Pro tip, you can adjust the inspirational prompt to match your expectations, according to the chosen model and specific behavior of each one ๐Ÿ‘Œ

Try it, explore different models and tell me which one is your favorite ๐Ÿค—
โ€”โ€บ fffiloni/image-to-music-v2
ยท
posted an update 21 days ago
view post
Post
InstantID-2V is out ! โœจ

It's like InstantID, but you get a video instead. Nothing crazy here, it's simply a shortcut between two demos.

Let's see how it does work with gradio API:

1. We call InstantX/InstantID with a conditional pose from cinematic camera shot (example provided in the demo)
2. Then we send the previous generated image to ali-vilab/i2vgen-xl

Et voilร  ๐Ÿค— Try it : fffiloni/InstantID-2V

โ€”
Note that generation can be quite long, so take the opportunity to brew you some coffee ๐Ÿ˜Œ
If you want to skip the queue, you can of course reproduce this pipeline manually
  • 1 reply
ยท
replied to their post 22 days ago
view reply

There is no DM here โ˜บ๏ธ, everything is open and transparent, you can expose your idea, iโ€™ll make my best to light the way ๐Ÿค—

posted an update 22 days ago
view post
Post
Quick build of the day: LCM Supa Fast Image Variation
โ€”
We take the opportunity to combine moondream1 vision and LCM SDXL fast abilities to generate a variation from the subject of the image input.
All that thanks to gradio APIs ๐Ÿค—

Try the space: https://huggingface.co/spaces/fffiloni/lcm-img-variations
ยท
replied to abidlabs's post 30 days ago
replied to their post about 1 month ago
posted an update about 1 month ago
view post
Post
Just published a quick community blog post mainly aimed at Art and Design students, but which is also an attempt to nudge AI researchers who would like to better consider benefits from collaboration with designers and artists ๐Ÿ˜‰
Feel free to share your thoughts !

"Breaking Barriers: The Critical Role of Art and Design in Advancing AI Capabilities" ๐Ÿ“„ https://huggingface.co/blog/fffiloni/the-critical-role-of-art-and-design-in-advancing-a

โ€”
This short publication follows the results of two AI Workshops that took place at ร‰cole des Arts Dรฉcoratifs - Paris, lead by Etienne Mineur, Vadim Bernard, Martin de Bie, Antoine Pintout & Sylvain Filoni.
ยท
posted an update about 2 months ago
view post
Post
I just published a Gradio demo for AliBaba's DreamTalk ๐Ÿค—

Try it now: fffiloni/dreamtalk
Paper: 2312.09767
โ€”
DreamTalk is a diffusion-based audio-driven expressive talking head generation framework that can produce high-quality talking head videos across diverse speaking styles. DreamTalk exhibits robust performance with a diverse array of inputs, including songs, speech in multiple languages, noisy audio, and out-of-domain portraits.
posted an update about 2 months ago
view post
Post
just setting up my new hf social posts account feature ๐Ÿค—
  • 1 reply
ยท