How to get clean greenscreen key
This tutorials shows how to get a tight garbage matte traced around the object, get a good dvMatte Pro key, and an original overlay mask to retrain color and picture details. DV footage is notorious for its poor chrominance (color) resolution. dvMatte Pro blurs the chroma channel before pulling the key, removing most of the jaggedness.
Video tutorial
Garbage trace
Create a new project and import the green screen video clip. Drag it to the timeline.
Add the dvMatte Pro effect and pick a low and high color close to the object, and change the 'View'-option to 'Compsite'. Change the 'chroma soften' vertical to '4 px' for PAL footage and '1 px' for NTSC.
Now the base mask to create the garbage trace is ready. Select the video clip in the timeline and select 'Layer' and 'Auto-trace'. Pick the settings below, notably threshold '60%'. Click 'Preview' to get an idea how the trace will look. The trace will now be automatically be created, let it run.
In the timeline, enable the 'Transfer Control' pane and under 'TrckMat' pick the alpha matte for newly created auto-traced mask.
Expand the mask by selecting the auto-traced layer and adding the 'Simple Choker' effect. Set the 'ChokeMatte' to '-25 px'. Disable the dvMatte effect in the green screen footage to get an idea how far the matte expands.
Greenscreen key
Now, to get a good key from the green background, select the footage in the timeline and open the dvMatte Pro effect pane again. Disable dvMatte Pro or set view to 'foreground', zoom 200%, and re-pick the absolute highest color and lowest green color close to the object.
And in base matte, set black point to around '50' to get rid of the halo shadow.
Overlay
To get back the correct colors and picture details, add a new overlay layer.
Select both the auto-traced and green screen footage in the timeline and pick 'Duplicate' or Control+D. Rearrange the auto-traced layer to the top. Select the duplicated green screen footage and select 'none' for 'TrkMat' and then again select the alpha matte for the duplicated auto-traced layer. This is done to link the correct duplicated auto-traced mask.
Remove the dvMatte Pro effect in the duplicated video footage. And set the 'Simple Choker' effect in the duplicated auto-traced mask to '2 px'. Add a optional 'Fast Blur' effect to get a smooth edge, set the blur to '2 px'.
Voila! The key should be finished. The edge details should be preserved.