Tuesday, February 7, 2012

The nexus between upscaling and decompression

Now that everybody has a Full HD display, you’ve probably formed your own opinions on the quality of SDTV content.  In the early days, it was a bit of novelty to have your own broadcast-monitor experience, like being able to judge whether a news studio had upgraded to a digital tape format yet.

But SDTV will be with us for a long time;  quite apart from the diversion that YouTube, cellphones and the scene have taken us on — something of a long-run historical blip — it’s inevitable that the standard “standard” still has another 20 years left in it.  A large body of work is still being created in this format, so it’s not about to lose its replay value the way black & white did in the ’80s.

So we’ll be dealing with non-square pixels for some time to come.  But I digress.

More than you might realise, our experience (and opinion) of SDTV is influenced hugely by the scaling process.  The quality of the upscaling has often left a lot to be desired, even by broadcasters:  All but one of the five national networks in Australia used only field-based rendering until 2007 … this meant that each field (1080i) or frame (720p) of HDTV broadcast had only the detail of 288 lines, or “288p”, half the SDTV potential 576 lines in a static image.

The tell-tale signs of field-based scaling
For several years, it was more likely that a $500 decoder box with HD outputs could do a better job than the networks.  And by 2005, the last generation of 1376x768 “non-full” HDTVs could finally scale 480i or 576i properly, deriving the full detail from a static SD image.  Full HD panels soon became a baseline feature, and the momentum was such that in 2006, it became de rigeur for HDTV displays to take a 1080i signal and derive a real 1080p picture with a static image.  When you think about it, this was quite an achievement, with 3 gigabits per second being processed by the internal chipsets.

24 bit colour × 1920 × 1080 × 60fps == 2,985,984,000 bps
double again for "100Hz" / "120Hz" interpolation

But by 2010, after a lot of network upgrades, the situation finally turned.  The networks were finally creating a nice image on their HD channels from their studio-quality SD sources (as the layman had always presumed);  now the only thing unnecessarily limiting quality on all those Full HD displays was the SD channels — or, to be accurate, the way that SD was handled.

A few things have helped mislead viewers into believing that everything on the HD channels was “in HD”, or HD-native, because the end-product of the SD channels was very far from SDTV best practice.  One thing was “small” screen sizes — anything under 60 inches makes it hard for an untrained eye to tell the providence of an HD end-product.  Another thing not helping matters was that the SD channels were more aggressively starved of bitrate;  yes that’s a factor, but a more important one has been the state of the art of consumer-side scaling.

So, what's “best practice” with SDTV, even if you can find channels with DVD-like bitrates?

True, field-based processing has long been history, but when you’re done deinterlacing at very high quality and are faced with a 720x480p60 or 720x576p50 sequence that requires display on a panel that has anything but those pixel dimensions, there are many ways to skin a cat.  As people soon found, an “upscaling DVD player” with HDMI output often did a much better job at exactly the same thing that the TV itself was supposed to do.  The SDTV channels as viewed on TiVo, Sky+HD, IQ2 looked better than the TV’s own tuner, or indeed other decoder boxes.

Both are examples of 576i upscaled to 1080i.  Both are deriving frames via 576p.
On the left, TiVo.  On the right, ‘another’ chipset, not utilising Faroudja technology :-)
The example on the left is definitely what you want to live with day-to-day, unless you're lucky enough to live in Japan.  You will have a very pleasant viewing experience on your Full HD panel if you apply this to studio-grade SD, and to SD sources derived from HD material — this is getting very common as cameras get upgraded more quickly than delivery mechanisms.

But we’re still left with a real-life problem:  What about the macroblocks?  In broadcast material, you’re often lucky to get as high as 720x480 or 720x576;  in streaming technologies, the problem is even more common.  You’re most likely to see this unavoidable drop in resolution on broadcast during a “quiet” scene if all hell is breaking loose on the other channels of the same transponder, and the broadcaster has an aggressive statmux’ing regime that rips the megabits out of the channel you’re watching:  The i-frames will become very visible and blocky, at the start of every MPEG GOP as the video starts “pulsating” with every group, generally once per second.

Here is one example of just such an occurrence, with the anamorphic 720x576 image @ 100% on the left.  On the right, I've converted the image to 90x72, and blown it up to view @ 800%.

Spot the difference  :-(
So, yes, for one frame we are watching the equivalent of a 72x72 postage-stamp sized video.
Hello, 1991 called — they want their QuickTime v1.0 video back.

Of course, with twenty years’ strides in technology, we should be able to do a better job than this.
The decoder should be signalling an alert to that fancy 1080i scaler connected to the HDMI port.
Or, at least, if we’re not going to interfere with the 1920x1080 frame buffer, let’s at least do what we can with the 720x576 frame buffer — presumably we still have to manufacture not-very-integrated chipsets that have to process video sequentially in intermediate frame buffers like this?

Left:  Bilinear upscaling.  Right:  Bicubic upscaling.
The challenge here — the only challenge — is to identify how low to go before applying a sane upscale, not some ridiculous nearest-neighbour copyblt lazy engineering.  Of course, this information is already available during the MPEG decoding process, and, of course, it’s only really needed during the lightweight MPEG-2 and SDTV decoding that’s long been conquered — but shall be with us for decades yet.  (In heavier scenarios, with MPEG-4 AVC deblocking and/or HDTV frames, this technique isn’t necessary).

Left:  Different frame, far less severe bitrate starvation.  Right:  Yet another frame, with ‘normal’ bitrate.
(note the quality recovery comes largely as a result of p-frames coping a lot better under harsh conditions)
Compare it to some more “normal” frames, above, to see how successful it is to integrate scaling and decompression stages.

By the way, the video sample here has been through a fairly typical process for older American TV shows in PAL/DVB countries:

film 24fps  480i 23.976pSF  576i 25pSF  540x576  padded to 720x576 for 16:9

And following on that, you can get all the 1080i scaling mentioned further above.

Sunday, January 29, 2012

Squeezing the most out of H.264 QuickTime


Presumably you already know about keeping the native resolution, aspect ratio,
frame rate, interlacing, etc., taking into account the final delivery format.

At this point it's worth repeating that Vimeo allows "Plus" users to download
each other's raw upload file ("raw" as in what got uploaded to the site). This can
be any .MOV or .MP4 you wish, getting around the 30fps limits of "online video".

How to get the most quality out of your QuickTime encoding:
   
  • Most importantly, install an x264 implementation encoder.
    It still decodes with Apple / VLC / MPlayer / etc.

    http://www.macupdate.com/app/mac/20273/x264-quicktime-codec

    This is Henry Mason's x264 implementation.  After extensive trial and error,
    I've found it more effective than the 'lavc' implementation, with far fewer buttons.

    Here's what I mean:
The 'lavc' x264 options panel.  Thankfully, we can avoid this.


  • Advanced Settings:

    This is simply a question of DB or not DB.  Deblocking is the pre-filtering
    craze that makes people with a semi-trained eye believe they have conquered
    the MPEG quality issue.  "No More Blocky Pixelization!"

    Unfortunately, it's also a killer of HD.  With time, you'll come to appreciate
    that deblocking kills the "oversampled" pixels which make the best HD film
    transfers, or classy 4K to HD content.  Indeed it can take away the full
    potential of SDTV (the best SD is always produced in HD).

    If you come across a high bitrate MPEG-2 source, you'll notice the lovely detail
    in the noise which really isn't noise at all.

    If your video feature contains grains, rocks, forests, or textures of any kind,
    then turn off DB.
      This is the key to ensuring that the feel of the video does not
    change … it is the only setting that has a real impact on changing the nature of
    the picture.  This sets it apart from the amateurish look of archive-unfriendly
    codecs like Cinepak, DiVX and Xvid, historically abused by "the scene".
    x264 Advanced Settings:  No DB
    Turn on CABAC (the real grunt behind H.264) and everything else, except B-frame
    Pyramids:  All the others put a burden on the
     encoder (you) and not the decoder;
    the random forward-lookup nature of the B-frame pyramids 
    means you would be
    placing an undue burden on the viewer, whose Pentium III 
    clunky laptop probably
    can't cope with the decoding.


    If your video feature contains mostly glossy surfaces
    blue sky, cloud or water,
    then use DB.  Set it to maximum.
     These are examples of where DCT technologies
    come up short;  the MPEG family will try to be "faithful" to the noise but in our minds
    we want to see the smoothness of these surfaces.
    x264 Advanced Settings:  DB Max
      
  • Aiming for a bitrate?  Use 2-pass and turn off the frame resampling.
    Of course, if you're aiming for a bitrate, you'd use "Restrict to" and enter kbps!

    There's a big fat bug:  If you don't select "Current", the 2-pass encoding won't hit
    your target;  instead it will just give you a "medium" quality encoding, regardless of
    the target bit rate, spatial quality percentage and temporal quality percentage.
     
  • Generally, though, you're not aiming for a bitrate.  You would be:

    1.  Uploading a short YouTube feature that doesn't go near the 20GB "pro" limit
    2.  Uploading a very short Vimeo feature that doesn't go near their 5GB limit
    3.  Uploading a low-resolution or sub-15-min YouTube clip;  you're going to be
         nowhere near the 2GB "amateur" limit, just wanting a quicker upload than
         you'd get if you uploaded the original source — without losing too much quality

    In this case you should set the Data Rate to Automatic, and select one of the following
    for Spatial Quality (temporal quality is ignored):
     
    • 50% Medium for "looks good to most people"
      (equivalent to bit-starved broadcast quality)
    • Halfway between 50% Medium and 75% High "looks good to pros"
      (equivalent to a typical DVD or BluRay)
    • 75% High only if you really need to maintain existing artefacts
      (e.g. for another generation of editing)
    • Halfway between 25% Low and 50% Medium for just scraping by
      (e.g. if you had a really bad source anyway,
      and just need some further bitrate reduction)


       

Friday, December 30, 2011

World First YouTube 60fps footage © 2011

(or)

YouTube Goes Nuts with 2160i
(cross converted from 1080p60)

For now, it's just 15 seconds:



How to view 60fps:
  1. Download the 4K version
  2. Play in VLC using "Linear" deinterlacing filter.
  3. Set size to "half" or play full-screen.
VLC being VLC, you'll have to muck around with getting playback to kickoff again once you've forced deinterlacing to activate.


In the next few weeks I'll be uploading the rest of the footage.

This will push the new upload limit of 20GB.

It will create a 4K download of 40GB though …
YouTube's own encodings are typically twice the file size and half the quality.

Monday, December 26, 2011

Undoing the Fisheye lens

Needed to shoot with maximum field of view?
But now have black vignetting to dispose of?

You already know you can crop it, but that's the whole point of why you did this Google search;
you were looking for a way to avoid losing that field of view.

There are plenty of ways to do this wrongly:
  • Turn on the Fisheye filter and bring it to a little bit negative.
    (It does very little doesn't really bring in the edges anyway).
  • Turn on the Fisheye filter and bring it to a massive negative.  Max radius.
    (You get point-pinching — very noticeable in motion when items cross it, especially a light pole).
  • Turn on lots of little Fisheye filters and line them up in a row.
    Equal spacing along the horizontal axis.  Set to very little, e.g. "-0.5" each.
No.  Stop it.  Stop!

Besides being very tedious, that last one has a silent killer:  You lose resolution.
After approximately 2 or 3 filters end-to-end, Fisheye sucks the life out of your HD.

But other than that, the effect is starting to achieve something remarkable:
The edges are coming in as if the anti-Fisheye effect were set very high;
it is the same as setting to the sum of the filter values.  Just without the pinching!

But unless you want to upload your video as 480p (and waste a lot of setup time),
you really need a one-step filter that doesn't blur your picture.

(Plus you probably have a gut-instinct that concatenating filters was wrong).

Do this instead:
  • "Bulge" filter set to maximum in reverse.  Export to Final Cut Effect.
Note on the left pane:  First choose "supertab" Inspector, before looking for the Filters tab. 


The key to this is to shell out $50 for Motion.
And to have lots of patience with its spinning cursor.  Unbelievable!

Inspector —> Properties —> Scale.  Try 179% for vertical, and something less for horizontal
(undershoot if you need to rotate later).


Once you "save" the project, it automagically appears in your FCPX filters list.
(I call him, "AntiBulge"!)

You can change and re-save, and the changes take effect only after deleting and re-applying in FCPX.

Now you can see the result, which is phenomenal:
Before
After … add x-axis scaling to suit taste

You can see it really pulled those corner details out of thin air … in fact, it's going at it so hard on the bottom left that you can even make out the (blue) colour bias that occurs due to chroma misregistration at very high resolutions on small-to-medium lenses.

I've left some of the vignetting in to allow for rotation correction (& cropping) but you can see how much you'd save compared to the original — line up with your eye, the left edge necessary to crop on the "After" image, and see how far you've really come in on the original?  Hardly anything at all.


If you turn off the scale & crop on the Motion side of things completely, you'll see everything gently pushed through the centre with a telltale "bubblegum" squeeze.  This is the natural counterpart to the fisheye lens.  If you play any motion video through that effect, including tilting and panning, you will see that this is the true counterpart;  not the Fisheye filter set to negative.

Sunday, December 11, 2011

A Pixel Is a Point Sample

A Pixel Is Not a Little Square (and other memos):

http://alvyray.com/Memos/MemosCG.htm#PixelIsNotSquare
Click on 'download' to get the PDF.

Precis:  It is a point sample.  Important if you want to upscale or downscale (I'm looking at you, Sky News Multiview).

Monday, October 3, 2011

Single-camera slowmo cutaways in FCPX

So you want to replace a bit of hasty panning with slow-motion of the cutaway angle?
Keeping the audio track intact and using a bit of classy motion flow, eh?

But you've got FCPX ... so instead you're getting ...

  1. Stabilisation refusing to turn on — you were wanting to use that, right?
  2. Audio timeline collapsing when you don't want it

The trick is to do everything in this order:

Cut Primary  Stabilise  Delete Audio  Fill Audio  Trim Cutaway

1.  Go blade cutting at the maximum extent of your primary angle.
     (worry about trimming the cutaway footage later)

2.  Turn on Stabilisation for cutaway angle.
     FCPX won't allow dominant motion analysis after step 3:
     If you delete your audio track first, a bug prevents DMA.

3.  Separate, then split off audio track of cutaway angle.

4.  Delete cutaway audio;  then extend (de-trim) previous audio track.
     Extend it to the start of the next audio track.
     This provides a reliable anchor in the new FCPX paradigm.

5.  Temporarily disable Stabilisation if required.

6.  Now apply re-speed and trim your cutaway angle.  Rinse & repeat.
     Using opt-[ and opt-] is generally the most fun way to go about trimming.

     The de-trimmed audio track will keep a hold of your endpoint.

7.  Turn on Motion Flow and reactivate Stabilisation as necessary.


Interestingly ... you may find that Motion Flow does a better job at Stabilising
all by itself.  After all, a wobbly camera is less wobbly at 25% rate.  I've never
seen a handheld video at 240fps that looks like it needs a tripod or steadicam.

H.264 workflow: Flawless 1080i to 1080p with JES

OK, so you've got some 1080i60 footage in H.264.

Freaking out because:
  1. Final Cut is showing just black
  2. QT Player X is showing just black
  3. QT Player 7 re-export is showing jitter: 1-in-60 fields out of place in the timeline
You're starting to regret not shooting 1280x720p60, because it's such a pain in the arts.

Never fear — just skip over the running waters with these 4 stones:

1.  Convert from ersatz 1080p60 to an "embedded 60i" 30fps movie:
     — export with QTP7 not QTPX
     — enforce 29.97fps
     — use H.264* with high bitrate / low crunch
     This gets a better result than the QuickTime 7.5.5 hack to allow 1080i compatibility
     (a crude 100% threading in-playback filter, changing dominance EVERY frame).

     * you could use intra (pixlet/AIC/prores) but it's a once-use-only intermediate.
     Plus, bypass another colourspace transform (funnily enough, with Apple's codec).

     QuickTime Player 7.5.5+ tells you H.264 1080i60 is 1080p60, by faking it.

2.  Check your "30p" movie has true interlacing, NOT field doubling:
     — open file in QuickTime Player and view in Original Size (Cmd-1)
     — skim forward to a frame with no global motion, and some horizontal edges
     — look for any staircasing, effectively 540p not 1080p

     If you find problems, repeat step 1 but remove a frame from the source.
     Delete one or three or five frames ("60p" ersatz) from the original & save a copy.


3.  Convert to 1920x1080p60 using JES Deinterlacer:
     — input = NOT progressive, top field first*
     — enforce 59.94fps
     — enforce 1080 lines, 1920 width in 'custom' under Standards Conversion
     — slide open that Inspector, and select NTSC and 'Video' range
     — video output 'Direct' to Apple Intermediate Codec
* you may have to switch TFF to BFF if you've been trimming

I've seen all sorts of problems with colour shifting, if you select "HD" instead of "NTSC";  brilliant green dresses become dull or turquoise, and skin tones lose their pink blushing highlights.

Living in a PAL world I still see the occasional problem with NTSC colourspaces looking dull on TV, but things have improved a lot since the big shift of 2009.  The only persistent offenders now are ESPN 1&2 HD and SD, which continues to insist on heavier shadows and a brown/purple bias on PAL displays and HDTVs in PAL land.

4.  Check your original / target with an overlay test:
     — open both files on the same frame in QuickTime Player
     — slide windows exactly over the same space
     — use Cmd-` (command-backtick) to repeatedly switch

Bright areas unequal are a likely sign that gamma values are wrong.
Dark areas unequal are a likely sign that NTSC/HD colourspace wrongly selected.

In a previous project I found I had to use 'Use Separate Gamma' with values of 2.11 and/or 2.12.  If brighter areas are mismatching, try this.  (I think I may have been using JES for step 1 as well as step 3).