Artificial Intelligence upscaling of old scenes

[monkeymagic27]

Machine Learning has advanced to the point that it's possible to convincingly upscale old footage, and stills. It takes a lot of processing, trial and error, and some manual tweaking to get the best results, but I really think that the older standard definition scenes here deserve to be upscaled, re-released and seen in the best possible quality. I did a test, upscaling the Rosanna & Chloe B scene, and also tried to improve the white balance slightly in the process. The results aren't perfect, but I think it looks a lot better than the current version.

Thoughts? Worth doing?

[happytappy1]

Yes,,,worth it without a doubt.I'd love to see the older shoots in an enhanced quality.It's such a pity that the quality is rather "fuzzy.....or cloudy".I can see a marked improvement in these pics and would wellcome any improvement!!,,,Thanks for proving it can be done!!

[georgexxx20]

I'm no expert on this sort of thing, but the quality of the old videos certainly need improving. As for the old stills I find the quality very good, if not better then the present ones.

[Jacksonpe49]

I agree about the stills, at least as far as the composition and content are concerned. Although there were a few years when we would get stills sets consisting of 250 images of which 150 were more or less identical shots of a pink hole and nothing else. They were pretty bad, and happily the obsession with gynaecological instruction has passed. Technically there may be some differences between the old stills and newer ones, but I can’t see any.

The content of the old videos was far better, but the quality was nowhere near today’s standard. If AI can remedy that, I’m all for it.

What does that come down to?

Don’t bother about the stills. They’re fine. But if you can improve the old videos, go for it.

[monkeymagic27]

Very true that the older stills are still decent quality. It can be nice to have XL images an an option though.
Older scenes like this also have really intrusive massive logos, not sure if Abby Winters still has the original footage without overlayed logos? Because AI can only do so much. I'd absolutely love to upscale the Gabrielle & Jamie scene, but the version available here is possibly just too heavily compressed, plus interlaced.

[morfeuth]

i would love to see the orchard girls upscaled..

[Frans]

Best source for higher quality vids of older shoots is probably the DVD's.

[monkeymagic27]

I'd gladly provide upscaled scenes to Abby Winters, if they were interested.
Frans, I'm not even really sure where to buy old DVDs?

[Frans]

I'd gladly provide upscaled scenes to Abby Winters, if they were interested.
Frans, I'm not even really sure where to buy old DVDs?

https://dvd.abbywinters.com/dvdline/...ardcore/page/3 still exists, links to a dvd retailer. Not sure if they'll stil have older dvd's of course but if you want a specific one it's worth looking.

Orchard girls was not a hardcore girl-girl and thus never released on dvd, as far as I know. All the flowers and colors in it would make it look very nice in high-def I'll agree.

[monkeymagic27]

I tried upscaling the Orchard Girls scene. The compression on these older scenes is so severe, which really limits the quality unfortunately. But upscaling to 1280x720 looks pretty decent. Original footage on left, upscaled on right:

[happytappy1]

Wow,,,these improvements are awesome if I may say.The back catalogue holds soo many gems that would be worth having if the quality were better!!Thanks for the effort in demonstrating the results if tried!

[garionhall]

I really want to explore this, but I am busy with another urgent project for the next six weeks. Once that's resolved, this is close to the top of the list.

I have also found our stash of old MiniDV taps from 2000 to 2008 or so (thousands of them!), along with a MiniDV "deck". Not sure if the tapes will still work, but it's worth a try... Now I just need a computer with a firewire input! I recall that we output each scene to MiniDV tape when we were encoding, so we don't need to edit, just need to capture and encode. But it's HUGE job.

Frans, DVD's are only Standard Definition, but they might be better quality. I have a copy of pretty much every DVD we made, so we can do to them if necessary (will be a little better quality than what's on the site for the pre-2010 IM and GG shoots).

[kayla_j]

I really want to explore this, but I am busy with another urgent project for the next six weeks. Once that's resolved, this is close to the top of the list.

I have also found our stash of old MiniDV taps from 2000 to 2008 or so (thousands of them!), along with a MiniDV "deck". Not sure if the tapes will still work, but it's worth a try... Now I just need a computer with a firewire input! I recall that we output each scene to MiniDV tape when we were encoding, so we don't need to edit, just need to capture and encode. But it's HUGE job.

Frans, DVD's are only Standard Definition, but they might be better quality. I have a copy of pretty much every DVD we made, so we can do to them if necessary (will be a little better quality than what's on the site for the pre-2010 IM and GG shoots).

Funny, I found a firewire card at a local tech shop yesterday! I had no reason to purchase it, but it seemed like something that could be really valuable to someone's future project...

Maybe I should go back and get it for you! Lol -- I should be able to get it to Australia within the next 6 weeks.

KJ

[garionhall]

thanks... but we have them here too - and electricity as well! There are a few problems;

Getting the card
As you say, they are around, plenty of used ones around, on ebay for example. But in my experience, there's 50% chance a 15-year old $10 card works.

And, need to have a computer that has ISA / EISA bus slots to put the card into (I joke... but only a little).

Getting drivers for the Firewire card
This page seems authorititive and helpful,https://www.studio1productions.com/A...Firewire-1.htm...

Getting software to import MiniDV footage
WinDV is often recommended, but it was last updated in 2003... Movie Maker 10 by Microsoft also seems to be an option, as does VirtualDub. Or maybe ScenalyzerLive (last update 2006, but often recommended for this). Or maybe HDVSplit (2006).

Getting the right codecs
These days, software handles codec gracefully, but in the 2000's this was frontier territory (rememebr TWAIN drivers?). Canopus DV Codec was the go-to then... latest version was 1999, not sure about the chance of that working in 2020!

Deinterlacing
Argh, never thought I'd have to deal with this shit again.

Encoding
On the cloud, thank goodness. But might need to get a Snowball to get the data up!

AI improvements
Then we can get to the AI improvements possible.

---

It's not impossible, but these are the sort of things that take days to get working right in my experience. This thread from 2018 seems typical.

Below, some of the tapes. So many tapes.

[monkeymagic27]

That's so fantastic to hear Garion! Amazing.
Good luck with the DV tapes! It's such a pain that old devices and formats lose support and stop working.
Really exciting. I think re-releasing remastered old videos would be a great option in this socially distanced era.

[kayla_j]

thanks... but we have them here too - and electricity as well! There are a few problems;

Getting the card
As you say, they are around, plenty of used ones around, on ebay for example. But in my experience, there's 50% chance a 15-year old $10 card works.

And, need to have a computer that has ISA / EISA bus slots to put the card into (I joke... but only a little).

Getting drivers for the Firewire card
This page seems authorititive and helpful,https://www.studio1productions.com/A...Firewire-1.htm...

Getting software to import MiniDV footage
WinDV is often recommended, but it was last updated in 2003... Movie Maker 10 by Microsoft also seems to be an option, as does VirtualDub. Or maybe ScenalyzerLive (last update 2006, but often recommended for this). Or maybe HDVSplit (2006).

Getting the right codecs
These days, software handles codec gracefully, but in the 2000's this was frontier territory (rememebr TWAIN drivers?). Canopus DV Codec was the go-to then... latest version was 1999, not sure about the chance of that working in 2020!

Deinterlacing
Argh, never thought I'd have to deal with this shit again.

Encoding
On the cloud, thank goodness. But might need to get a Snowball to get the data up!

AI improvements
Then we can get to the AI improvements possible.

---

It's not impossible, but these are the sort of things that take days to get working right in my experience. This thread from 2018 seems typical.

Below, some of the tapes. So many tapes.

Firewire cards AND electricity? I appear to have "misunderestimated" your country.

Also, the complexity of the project.

Also... The tapes. I underestimated the tapes! (so many!)

[kayla_j]

thanks... but we have them here too - and electricity as well! There are a few problems;

Getting the card
As you say, they are around, plenty of used ones around, on ebay for example. But in my experience, there's 50% chance a 15-year old $10 card works.

And, need to have a computer that has ISA / EISA bus slots to put the card into (I joke... but only a little).

Getting drivers for the Firewire card
This page seems authorititive and helpful,https://www.studio1productions.com/A...Firewire-1.htm...

Getting software to import MiniDV footage
WinDV is often recommended, but it was last updated in 2003... Movie Maker 10 by Microsoft also seems to be an option, as does VirtualDub. Or maybe ScenalyzerLive (last update 2006, but often recommended for this). Or maybe HDVSplit (2006).

Getting the right codecs
These days, software handles codec gracefully, but in the 2000's this was frontier territory (rememebr TWAIN drivers?). Canopus DV Codec was the go-to then... latest version was 1999, not sure about the chance of that working in 2020!

Deinterlacing
Argh, never thought I'd have to deal with this shit again.

Encoding
On the cloud, thank goodness. But might need to get a Snowball to get the data up!

AI improvements
Then we can get to the AI improvements possible.

---

It's not impossible, but these are the sort of things that take days to get working right in my experience. This thread from 2018 seems typical.

Below, some of the tapes. So many tapes.

"That is some nerdyawesome shit. I would do that job. I would do that forever." -Jake, 2020

[YtB3QQ6dm2M8vF-H]

Is it possible to upscale audio with AI as well? I tried upscaling a few of the older videos and had passable results, but the sound was still highly compressed.

[YtB3QQ6dm2M8vF-H]

I ran a couple of tests using Esrgan, Vid2x and Video Enhance AI and I think by far VEAI has the best results. Was kinda funny to upscale a 480p 4:3 ratio video from 2006 to 4k! I'm a total tech muggle though so I'm sure my results could be better. What settings did you use monkeymagic? I extracted the video frames to TIFF and then recombined them in ffmpeg. I wouldn't even know where to begin with color grading. Only took forever to do! I'm running an i7 10700k with a 2080 ti so I can imagine doing this at AW would take ages. Maybe release a couple per month of the more popular shoots? I'm going to try Chloe B and Angie's epic shoot next. I seem to remember that there was a lot of crazy motion in the 2nd vid so it will be interesting to see how upscaling handles that.

[YtB3QQ6dm2M8vF-H]

I think I might be having a bit too much fun messing around with this lol. I used gigapixel to blow up a 1472x981 picture of Katherine F from back in the day to 4x scale and the results are just incredible. 200kb to 26.2mb! Then I blew up a XL image from 26.2mb to 260mb for fun. Still its pretty weird to think that you are just seeing a computer's interpretation of what the image would look like if it was natively shot at a higher resolution. AI is wild stuff.

I definitely think video benefits more but some of the older pictures before the XL era really shine. Kristy is an incredible photographer and her work definitely deserves this technological treatment!

[YtB3QQ6dm2M8vF-H]

After a gazillion tests, I think the technology is promising but needs a lot more work. It generally looks good on a HDTV since you have some viewing distance, but you really notice all the flaws when looking at it on a monitor. I've tried almost every permutation of techniques that people have recommended but you still see artifacts and oversmoothing of bodies, particularly in high motion situations. Sometimes it begins to look very uncanny valley... like CGI almost. It does a lot of weird things with hair, and when a large amount of models are sporting pubic hair, it can look kinda odd. Mouths and faces can get distorted sometimes, which is kinda nightmare fuelish. Maybe its the settings I've used, but I can't find anything that works in all instances. You have to adjust per scene for best results.

[monkeymagic27]

YtB3QQ6dm2M8vF-H, I think Topaz Video Enhance is the way to go, just for the user friendly experience. It's amazing at removing blocky compression artifacts, but there's only so much it can do with footage that's too low quality. There's some promising research papers happening all the time though, so hopefully tools will improve further. I got the best results by doing a 50/50 blend between HQ and HQ-CG modes. That meant processing twice, so a LOT of processing time, and a lot of disk space filled up, but it averages out any weird artifacts from the two different modes.

[garionhall]

Here's my notes on MiniDv capture. Not good.



Using Sony HVR-M15P MiniDV deck.
Using Garion's old work PC (HP Z400) (Win10)
Connected by 4-pin to 6pin Firewire cable, to built-in firewire port on mobo


Using HDVSplit
---------------------
Software to capture (https://www.videohelp.com/software/HDVSplit)


M15P deck needs powering off and on again to make HDVSplit recognise the input. I can get HDVSplit to control the deck fine. I cannot get a preview of the video to show up (many people seem to have this problem).


I click record, it seems to work - counters tick over, tape is transporting, "Capturing has started" message.


I click "Stop" to stop recording, "end of capturing" is reported, deck stops playing back.


But, file is 0 bytes. The Internet says I need to run the app as Admin. I do that, it seems to make no difference - still 0-byte files.


Using WinDV
-------------------
WinDV shows a "Can't find DV output pin" error. Most of the internet has this problem, no one has a solution. https://forums.afterdawn.com/threads...corder.690160/


STOIK Capturer
---------------------
Error on launching, "Ready2go Video couldn't use the video source, because it's in use by another program."


Persisted after rebooting.


Windows Movie Maker
-------------------------------
Can only capture from a DV camera, not a deck.

So I think it's a no go. :/

[garionhall]

Regarding Audio, here's what our lead boffin had to say:

Well, audio becomes a bit of its own thing (full disclaimer, audio is not my area of expertise so everything I'm about to stay is to the best of my current understanding).

When trying to upscale old masters, none of the algorithms that we are discussing will do much or nothing for the audio (they're focussed on the image). So if we do want to increase the quality of the audio we would have to process it separately.


I think the main challenge would be trying to standardize the process since different clips will have different challenges or issues to fix, so in the end, some time of diagnosing and adjusting per clip will be needed.


For example, almost all of the old clips (especially if we are using transcodes as a source) will suffer from audio compression. For that, we would need to use some sort of audio expander. However, that's where I imagine most similitudes end. From there on some clips might suffer from reverb, while others from noise, others from distortion... Usually fixing these issues becomes a balancing act between all those restorative filters, since overdoing one, might break the next, etc. So I think this needs to be tackled per clip. In the end, it'll probably be the same 4-5 same filters being used, but with slightly different settings and combinations. And of course, results will vary highly depending on the source.


On a positive note, post-processing audio is relatively computationally cheap and can be done easily within Audition, ProTools or within any other audio editing platform (even NLE's have very decent audio options these days if you know what you are doing).

[garionhall]

And here's the lead boffin's feedback after doing some tests:

Topaz Gigapixel:

• Overall the best result I believe. While not completely perfect seems to be able to deartifact and scale while attempting to preserve detail. There are still some artifacts visible and feels like in high frequency areas it might go a bit too far and oversharpen but I feel like that is to be expected.
• As you know this software is sold per license (200USD) and can't be used in distributed rendering. It relies heavily on NVIDIA graphics cards. I have been rendering using CPU with extremely low speeds, so use of NVIDIA cards becomes mandatory. This month a new generation of cards has been released (the 3000 series). Doing a bit of research I've seen that a 2080ti card renders at 0.15 sec/frame for a similar scenario as ours. A new 3070 should match that performance with half the price with the 3080 and 3090 rendering substantially faster (at least on paper, most of these cards have yet to reach the market so performance is theoretical but for what we know so far very promising). I was considering upgrading my workstation in the near future, if we decide to go this way this is something to keep in mind since AMD cards won't cut it for this (I'm not sure what hardware [video editor] is running)
• Feels like the best option if we plan to optimize for quality. (a good option if we also are considering in the future to upscale some HD media to 4k).
• I can see two ways of running this:
  • Either way keep it in house with a workstation that we use to render this (whether that's mine, or someone else's in the AW team) and we process a limited number of videos a month (the exact number would depend on render times and the downtime available for that machine, but given the above numbers something between a handful and a couple of dozen seems reasonable with the appropriate hardware).
  • We rent an industrial GPU server to process videos extensively during a short period of time (obviously more expensive). This option would need more research to determine which configuration would be optimal vs cost.

PIXOP: (https://www.pixop.com/)

• Thought I'd try another AI upscaling algorithm. Pixop seemed like the obvious choice. Quality wise I thought it did not that great of a job when dealing with the compression artifacts. Tending to over sharpen the blockiness of the compression at times.
• The positive is that it is cloud-based and they have enterprise plans so seems like scaling shouldn't be a problem.
• Feels like a middle option. Further testing would be needed to see if we can increase the quality of the upscaling (I don't expect it to match Topaz's but maybe it can be optimized a bit).

MPEG Streamclip / FFMPEG;

• Open-source video encoding frameworks. A little bit of control when upscaling but nothing revolutionary (basically we can adjust the type of filtering to regain a little bit of sharpness vs a plain upscaling but nothing fancy like the AI algorithms mentioned above).
• Software is obviously free. We can run it in as many computers as we have access to. CPU based.

AE (After Effects upscaling plugins):

• Similar to the FFMPEG/MPEG Streamclip solution (maybe slightly better quality).
• Main benefit is that we own already the infrastructure needed (basically any machine capable of running AE can do this, it'll depend on the CPU clockspeed how fast or slow the render times are).
• Our only cost would be energy consumption while rendering.

Amazon Elastic Transcoder:

• Haven't done any test with this one, but having gone through the documentation I have no reason to believe that is doing anything rather than a simple interpolation when scaling up, so I expect barely minimum to almost no increase in quality. Should be a simple transcode,
• The only upside obviously is that is incredibly easy to upscale and integrates easily with our existing S3 infrastructure (probably the cheapest as well, rates starting at $0.0075/minute)

[garionhall]

I hasten to add, there's no question the output is a significant improvement from what we have. But we need a way to do this at scale to make it worthwhile, I think?

[Frans]

I hasten to add, there's no question the output is a significant improvement from what we have. But we need a way to do this at scale to make it worthwhile, I think?

Depends.. do you want to upscale ALL videos, or selected ones? How far back would it be useful to go anyway, as below a certain source bitrate/quality I'd imagine upscaling like this simply won't work. You can't upscale vids from 2001 mpeg1 to mp4/HD 4k.. well you *could* but not have them watchable

It's a fun tech challenge, but I do wonder how to ever it get it done cost-effective...

[monkeymagic27]

A few thoughts on Garion's MiniDV capture woes (this may all be obvious stuff you're already aware of, sorry).
Legacy firewire drivers will probably be necessary in windows 10, see this link: https://www.studio1productions.com/A...Firewire-1.htm
HDVSplit requires ffdshow to be manually added in a specific way, which could be easily missed. The procedure is described in the readme file.
Windows legacy support is so bad, Linux and dvgrab may actually be an easier process to get working?
An alternate option would be capturing the video out signal with an Elgato, or similar device. Though, the video out signal on that particular deck is probably limited to standard definition.
What a pain, best of luck! Really, it's just wonderful to see that Garion is looking into this stuff.

[IlikeTowatch]

This is a huge plus. I love the classics. Plus I just bought a laptop with 4K resolution.

[garionhall]

monkeymagic27 sorry, I missed your post!

A few thoughts on Garion's MiniDV capture woes (this may all be obvious stuff you're already aware of, sorry).
Legacy firewire drivers will probably be necessary in windows 10, see this link: https://www.studio1productions.com/A...Firewire-1.htm
HDVSplit requires ffdshow to be manually added in a specific way, which could be easily missed. The procedure is described in the readme file.

I don't think I am patient enough to delve back into this... and from other comments online, once one issue is found, there are always more.

Windows legacy support is so bad, Linux and dvgrab may actually be an easier process to get working?

I am not a Linux expert, so this is a no go. :/

An alternate option would be capturing the video out signal with an Elgato, or similar device. Though, the video out signal on that particular deck is probably limited to standard definition.

We only ever shot SD on MiniDV, so no worries there.

I had mentally blocked out the idea of capturing via analog - seems so regressive to go digital --> analog --> digital - but it if works we'll be in a better spot than we are now, for sure!

The Australian suppliers are out of stock of the Elgato device, but I found one for one-twentieth of the price of the Elgato on Ebay (here), so I bought that. I will give it a go as soon as it arrives, and report back (assuming the problems I am having are not with the MiniDV deck itself!).