Translating videos

[ghost]

Did you know, that AI can translate videos into another language?

I just came across heygen.com. I've tried it with a short clip from the Czech movie Kruh. I chose something difficult on purpose and didn't have high expectations.

The result is really very impressive. Even the lip movements were changed.

But look (listen) for yourself: [Code]

[E9A7F3bD]

Guess it combines some model flow and gets a available result.

STT -> Translation -> TTS -> Lip Sync (Vid Gen)

[E9A7F3bD]

But we desperately need to see some open source solutions, and I don't think most people could afford the Pricing & Plans sufficient to translate movies.

Actually, I don't need that much. I'm satisfied with a good enough STT & Translation model.

Whisper is great, but still underperforms in some movies.

I still haven't found a good enough model for full-language offline translation.

[ghost]

Yeah, the pricing is the problem. In the free plan they give you only 3 videos/month up to 3 min. duration.
I imagined how much a second audio track could help to create subtitles with Whisper.

[E9A7F3bD]

In the free plan they give you only 3 videos/month up to 3 min. duration.

Even in "$89/mo/seat" there is still only "up to 30-mins".
Although you can always do the slicing and combo work.

In addition, its positioning is not aimed at content like movies that include multiplayer images and overlapping dialogues.

[E9A7F3bD]

I actually avoid seeing redundant "graphic" editing, especially in movie works, it ruins people's original micro-expressions.

I don't think that would look better any ways, that won't exceed real-person expression at least for the visible next few years.

But for commercials or introductory videos, maybe it's worth it.

[ghost]

I was only talking about translating videos. I always prefer movies in their original language. But this can really help a lot for creating subtitles. Especially for movies in "exotic" languages.

[E9A7F3bD]

I was only talking about translating videos.

Well, I thought the context here was always the movies.

I always prefer movies in their original language

Yes, so do me.

But this can really help a lot for creating subtitles.

How? Do you mean STT only?

[ghost]

How? Do you mean STT only?

I mean when you're making subtitles for a movie in a language that Whipser can't handle well and it gives you nonsense, you could listen to e.g. English audio (the translated video). This would help a lot.

[E9A7F3bD]

I mean when you're making subtitles for a movie in a language that Whipser can't handle well and it gives you nonsense, you could listen to e.g. English audio (the translated video). This would help a lot.

OK, I understand now. But I don't think it will do better than those pure STT models or online services.

And it may have information distortion caused by multiple conversions and processing.

But I can see the scenarios you mentioned, such as you just want to manually process the content for a few minutes and be covered by the free quota.