In last month's HDV Camcorder Showdown, Part One, we reviewed the features and usability of the Canon XL H1, JVC GY-HD100, and Sony HDR-FX1/HVR-Z1U; this month our focus is on quality. In addition to testing JVC's 30P HDV footage against the 1080i produced by the Canon and Sony, I also took a preliminary look the latter two cameras' faux-progressive modes, Canon's 24F and Sony's Cineframe.
In the abstract, comparing the quality of 1080i, with 2,070,600 total displayed pixels, against 720p, with 1,310,720 total pixels, is a little like comparing a 7-megapixel still-image camera against a 4-megapixel camera; obviously, the first should reproduce more detail when shooting the same image.
In addition, at least for the moment, neither HDV mode is a viable delivery format for most event videographers. Though I frequently shoot in HDV these days, at least with one camera, I've never delivered video to a client in any form other than DVD. In this regard, the actual quality of the HDV file itself is irrelevant if it doesn't translate to better DVD quality.
For these reasons, whenever I analyzed the resolution and detail of the video produced by the cameras, I first compared video at its original resolution, and then encoded the files to DVD-compatible MPEG-2. Under the assumption that most folks shooting in 720p would output in progressive mode, I output video from the JVC to a progressive video file at 720x480.
Thinking that most folks shooting in interlaced modes would deliver in the same, I output the Canon and Sony video to interlaced MPEG-2 at the same resolution. I captured all video and produced all MPEG files and screen shots in Adobe Premiere Pro 2.0 and prepared all screen shots in Ulead PhotoImpact.
How We Tested
For indoor testing, I used new widescreen test patterns and images for our test from DSC Labs (www.dsclabs.com). The larger chart is the ChromaDuMonde, which includes color vectors, grayscale, and skin-tone patches as well as horizontal and vertical trumpets to assess detail. I also used several test patterns from the CamBook 4, which now comes with me on all shoots, primarily for the white balance but also for the BackFocus test pattern.
In previous, primarily SD tests, I simply shot an older 4:3 pattern while locked down on a tripod and analyzed the results. However, with the higher resolution in HDV camcorders, misalignments between the test charts and CCD pixels can produce suboptimal results. To avoid this, I slowly moved the cameras up and down and side by side while shooting, and then captured the highest-quality frames in both the horizontal and vertical resolutions for comparisons. Credit DV magazine's Adam Wilt for originating this technique.
I lit all indoor shots with the 1,000 W Lowe's lamps featured in last month's article, diffusing the light with Roscoe diffusion fabrics. I shot all indoor shots at a shutter speed of 60 with gain disabled. I controlled brightness with the aperture controls, setting the zebra pattern indicators to 95% and increasing the aperture until most of the zebra pattern disappeared. As a check, I enabled the automatic exposure on all cameras and made sure that the aperture setting I manually produced was within one stop of the setting I chose.
For the record, I shot the charts from about 7' away; nothing scientific, just the most comfortable distance for my cramped offices. I attempted to frame the images identically, and make sure the tripod was perfectly level, but minor differences did creep in.
I shot two synthetic resolution tests, the ChromaDuMonde and the MultiBurst Test Pattern from the CamBook. As perhaps you can see in Figure 1, the resolution produced by the Canon was spectacular, with noticeable line separation in the horizontal and vertical axis at both extremes (near the 800 lines per inch marker).
Figure 1. ChromaDuMonde at the original frame resolution and format. Canon's quality is spectacular.
With 1280 pixels of resolution to play with horizontally, the JVC matched this performance at that axis, but lost steam vertically at around 600 lines per inch. The Z1U/FX1 optics performed well vertically, almost matching the Canon, but horizontally it trailed the JVC, even at the higher resolution.
After rendering all files to MPEG-2, the picture changed considerably (Figure 2). First, JVC took the lead, with a noticeable advantage over the two 1080i camcorders. In addition, though the images were close, Sony appeared a touch sharper than Canon.
Figure 2. Compressed ChromaDuMonde (720x480 resolution). JVC takes the lead, and Canon a back seat to Sony.
The same blueprint unfolded with the MultiBurst test pattern (Figure 3). The uncompressed frames of all three camcorders showed similar horizontal resolution, somewhere north of 500, while the Canon unit seemed slightly superior in vertical resolution, showing slightly more detail at 600 lines per inch than either other camcorder.
Figure 3. CamBook MultiBurst test pattern, original frames. Canon has the lead.
After compression, the JVC again seemed just a touch more crisp in both axes, though the Canon looks slightly clearer than the Sony (Figure 4). Tough to know what conclusions to draw from these results, but let's hold off a bit, as evidence from other tests weighs in.
Figure 4. CamBook MultiBurst test pattern, compressed. Again, the JVC frame looks sharper with slightly more resolution.
As mentioned in the features and usability story, I found the JVC difficult to optimize in the field for both color and focus. The previous images illustrate that focus wasn't an issue in our tests. Similarly, in the lab, with more than adequate lighting and without the pressure of a real-time shoot, the JVC produced outstanding color quality.
This is shown in Figure 5, the CamBelles chart from DSC Labs. Here, the JVC produced excellent colors, especially the red ball and flowery batching suit on the left, though Canon's sky is more accurate. The Sony image looks a touch muted, with slightly less contrast in the face on the right. Still, without side-by-side comparisons, all of these images would easily be considered very high quality.
Figure 5. The CamBelles, all original frames, manually sized to match in PhotoImpact.
The greenscreen test measures both image quality and fidelity, since irregular colors in the background will prevent a clean overlay. To complicate this test, I wore a green shirt despite shooting against a green background, which is definitely not recommended.
As you can see in Figure 6, all three camcorders produced a clean overlay, with relatively smooth edges. This is quite a trick in Premiere Pro, which relies on After Effects for most high-quality color keying. Here, the XL H1 reproduced the shirt color most accurately with the best overall exposure, a nice middle ground between the slight overexposure on the JVC and again muted colors of the Sony.
Figure 6. That's one tired-looking cowpoke. Greenscreen shots, post-SD compression.
Note that these screens are from the video files compressed to SD MPEG-2, like the resolution tests shown in Figures 2 and 4. In real-world shots, the JVC's advantage in detail doesn't really show through. Overall, in terms of shooting for chromakey, all three cameras, plus their HD formats, clearly make the grade.
Meet Mr. Lance
I've always used a shot of Lance Armstrong under my editing station for low-light testing. This year's images are a bit brighter because my desk is up on cinder blocks so I can edit standing up, one of the ergonomic changes my neurosurgeon recommended. The higher the desk, the more light sneaks under the table, the easier the shot. Long story short, you can't compare these images to those from previous tests.
All cameras produced very good results in manual mode, where I adjusted gain up to 12 dB and aperture to increase brightness but kept the shutter speed to a reasonable 60. Color-wise, Sony proved the most accurate, with the JVC noticeably off color in the maille jaune, but as we'll see in a moment, this was probably user error. None of the frames in Figure 7 show much chroma noise, if any, which is remarkable. All three videos exhibited slight noise during real-time playback, perhaps slightly worse in the Sony, but nothing that would be noticeable without side-by-side comparisons.
Figure 7. Low-light Lance. All cameras produce good results, though the colors are a bit off in the JVC.
Figure 8 shows the same image shot in automatic mode. The JVC camcorder did a better job with white balance in auto mode than I did with manual, because the yellow jersey has a more accurate color, albeit a bit faded. I was surprised at the relatively dark Canon image, since it's easier to brighten an underexposed image than darken an overexposed image; at least Canon erred on the right side. The Sony automatic image was virtually identical to the manual image, highlighting the camera's ease of use in automatic modes.
Figure 8. Low-light Lance in automatic settings.
Outdoor Tests--Normal Mode
How can you buy a dog and deduct the costs? If you're a videographer, you can buy a solid-black German Shepherd and tell the IRS you needed a way to test for dynamic range in outdoor shots. Well, that's my story and I'm sticking to it.
For the record, I shot these outdoor shots in fully automatic mode--trying to meet the attention-span window of two little girls and a dog while fine-tuning three cameras was more than I wanted to handle. The Sony and Canon camcorders suggested ND filter settings, which I followed, while using the 1/4 ND setting for the JVC.
In my view, the Canon and Sony images are very close. The Sony image is a touch darker, with richer colors, but shows less detail in the dog (named Jane, by the way). That's the dynamic range test I paid $500 for.
Had I dialed the Canon's ND filter up a notch, color may have improved without too much loss of detail in Jane's fur, but we'll never know. Overall, both cameras performed well in the sun in these and subsequent tests in progressive mode. In contrast, while the JVC shows plenty of dynamic range in Jane's fur, the entire frame shows a slight color cast, with muted sweaters and facial tones that are subpar compared to either other camera (Figure 9).
Figure 9. Outdoor shots, colorful with a significant dynamic range.
All Progressive, All the Time
Both the Sony and the Canon have interlaced CCDs which are technically incapable of capturing "true progressive" frames for transferring video to film or for achieving the "filmic" look in video. However, both companies have introduced techniques that allow their cameras to produce faux-progressive frames. Sony's CineFrame mode, which has been in the market longer than Canon's Frame mode, has been widely criticized for both loss of resolution and jerky motion during playback.
Some of the critiques have been highly technical and convincing, while some have been knee-jerk, advocates of JVC, 720/24p, or just "true progressive" snobs who obviously aren't basing their results on side-by-side comparisons. Even if they were, oftentimes bad results are produced by inadequate editing and rendering software, not poor video. Since I had all three cameras in-house, I decided to run some quick tests.
Those claiming loss of resolution in the faux-progressive modes contend that since the CCD is interlaced, the camera must deinterlace the video to produce a progressive frame, throwing out every other frame and losing detail in the process. To test this, we shot the CamBook rez chart in CineFrame and 24F mode and compared the results to the frames shot in normal HDV1080i. The results were quite interesting.
Specifically, there was a noticeable loss of resolution in the Sony camcorder, as shown in Figure 10, all in the vertical resolution, where you would expect to see it if produced by deinterlacing. On the right, in the 60i image, you can see detail at the 600 lines per inch on the right, with very good separation at both 400 and 600. On the left, in the 24p image, there is noticeable loss of detail at 400 lines, with 500 looking similar to 60i, but no separation at all at 600 lines.
Figure 10. The FX1 in faux-24p and 60i, both at full HDV resolution.
However, the Canon produced no noticeable degradation in the same tests; in fact, the frame is slightly clearer in 24p mode (Figure 11). Clearly not all all faux-progressive schemes are created equal.
Figure 11. Canon XL H1 in 24P and 60i, both at full HDV resolution.
In addition to the test patterns, I shot a number of other sequences with all three cameras, including a slow zoom, a ballet dancer, a jazz singer (the fabulous René Marie) and several other scenes, static and with motion, to try and isolate whether the motion from the Sony and Canon cameras was irregular. Playing back the video on a television and after capture to disk, I saw no difference in playback smoothness in any scenes.
What conclusions to draw? Tough to say. We're at the early stages of 24p HDV on the desktop and it's safe to say that any bizarre results could be software-specific. For example, the only way I could capture all three progressive video streams was using the Aspect HD presets from CineForm. Premiere Pro won't capture all three natively, and other prosumer NLEs will probably fare better with intermediary codecs in these circumstances as well.
I wish I had found the CineForm solution sooner, because all the JVC video that I captured natively in Premiere Pro took a long time to load and frequently crashed (on two different computers). All the CineForm video captured and played back smoothly, though the files are huge--a 2:30 (min:sec) file was about 1.8GB, about 3.3 times larger than the HDV file itself.
I will stick my neck out and say that the XL H1 does seem to produce higher-quality frames in progressive mode than the FX1, with a resolution slightly higher than the JVC. But more weighty conclusions will have to wait until I try capturing, encoding, and comparing in several applications, since CineForm and Premiere Pro, though they seemed to work perfectly, may not have been doing so. It's also worth noting that achieving a convincing film effect in event video involves much more than the frame rate or mode of the camera; shooting styles and editing techniques are at least as important. Here we're just doing image comparisons, plain and simple.
Tale of the Tape
What about the cameras overall? I'm reminded of a Vijay Singh quote: "It's not how good your good shots are, it's how bad your bad shots are." He was talking about golf, of course, but I think the same holds true in shooting video. While the JVC is absolutely capable of brilliant shots, and its 720p format may be better suited for conversion to SD DVD than 1080i, it didn't show through in real-world shots, and the risk of a "bad shot" is much, much greater. In a studio with retakes, that might not be a problem, but in live events, it's a potential nightmare.
In contrast, the Canon is much more novice-friendly with greater room for error. I absolutely adored the camera, and everyone who used it on the shoots we did with it loved it as well. In most tests, the quality was demonstrably better, if only visible in side-by-side comparisons.
Of course, then there's price. In truth, the unique qualities of the XL H1 that truly justify the price are those that I haven't evaluated, like Genlock sync, SMPTE timecode I/O, and SD and HD-SDI output. If these don't add value in your shop, it's tough to spend close to $9,000 on a camera when the FX1, at about one-third the price, produces nearly the same quality.
Special thanks to my crew on several of the shoots: Chuck Riedhammer, who saved my butt more times than I care to admit; Gary McClennan, the clean-up artist; and Scott Webster, the natural. I couldn't have done it without you boys.