Objectives and Testing Rules
H.264 Codec Testing Objectives
The main goal of this report is the presentation of a comparative evaluation
of the quality of new H.264 codecs using objective measures of assessment.
The comparison was done using settings provided by the developers of each codec.
The main task of the comparison is to analyze different H.264 encoders for the task of transcoding video—e.g., compressing video for personal use. Speed requirements are given for a sufficiently fast PC; fast presets are analogous to real-time encoding for a typical home-use PC.
Speed/Quality tradeoff
Detailed descriptions of the speed/quality trade-off graphs can be found in
Appendix 7. Sometimes, codec results are not present in the particular graph owing to the
codec’s extremely poor performance. The codec’s RD curve has no intersection with the
reference’s RD curve.
The speed/quality trade-off graphs simultaneously show relative quality and encoding speed
for the encoders tested in this comparison. XviD is the reference codec, for which both quality
and speed are normalized to unity for all of the graphs. The terms “better” and “worse” are
used to compare codecs in the same manner as in previous portions of this comparison.
Please note that the method of averaging among all sequences assumes that all codecs produced results
for each sequence. When this is not the case, only existing results are taken into account.
Next figures show results for the Normal preset. The results differ depending on the chosen metric.
Y-SSIM: The three best codecs (no codec performs faster with higher quality) in terms of speed and
quality are XviD, Elecard and x264 on average, except for the "Ice Age" and "Indiana Jones" sequences.
The x264 encoder is better on average than DivX H.264 and MainConcept.
Almost all encoders except Microsoft Expression exhibited better results on average than did Theora.
Relative Quality Analysis
Next table shows relative bitrates for a fixed-quality output for all codecs.
Note that these tables do not include information about the speed of the encoder.
Note that each number in the tables below corresponds to some range of bitrates (see Appendix 7 for more details). Unfortunately, these ranges can differ significantly because of differences in the quality of compared encoders. This situation can lead to some inadequate results when three or more codecs are compared. This comparison technique will be improved in the future.
The leaders are x264 and MainConcept, depending on the quality metric (the bitrate difference is 18% for a fixed quality).
Table 3. Average bitrate ratio for a fixed quality—usage area “Movies,” Normal preset, Y-SSIM metric
|
DivX H.264 |
Elecard |
Expression |
MediaSDK |
MainConcept |
Theora |
x264 |
XviD |
| DivX H.264 |
100% |
124% |
127% |
118% |
87% |
138% |
76% |
128% |
| Elecard |
81% |
100% |
103% |
95% |
70% |
115% |
61% |
104% |
| Expression |
79% |
97% |
100% |
93% |
69% |
114% |
61% |
101% |
| Media SDK |
84% |
105% |
107% |
100% |
74% |
120% |
64% |
108% |
| MainConcept |
115% |
142% |
144% |
136% |
100% |
157% |
88% |
145% |
| Theora |
72% |
87% |
88% |
84% |
64% |
100% |
57% |
88% |
| x264 |
131% |
163% |
164% |
155% |
114% |
176% |
100% |
164% |
| XviD |
78% |
96% |
99% |
93% |
69% |
113% |
61% |
100% |
Next figure depicts the data from the table above.
Each line in the figures corresponds to one codec.
Values on the vertical axis are the average relative bitrates compared
with the codecs along the horizontal axis. A lower bitrate indicates better relative results.
Speed/Quality tradeoff
Detailed descriptions of the speed/quality trade-off graphs can be found in
Appendix 7. Sometimes, codec results are not present in the particular graph owing to the
codec’s extremely poor performance. The codec’s RD curve has no intersection with the
reference’s RD curve.
The speed/quality trade-off graphs simultaneously show relative quality and encoding speed
for the encoders tested in this comparison. XviD is the reference codec, for which both quality
and speed are normalized to unity for all of the graphs. The terms “better” and “worse” are
used to compare codecs in the same manner as in previous portions of this comparison.
Please note that the method of averaging among all sequences assumes that all codecs produced results
for each sequence. When this is not the case, only existing results are taken into account.
Overall Conclusions
Overall, the leader in this comparison is x264, followed by MainConcept and DivX H.264. The Theora encoder demonstrates the worst results among all codecs tested.
*The encoding speed of this codec does not fall in the required range owing to our chosen presets or to a lack of options.
The overall ranking of the codecs tested in this comparison is as follows:
- x264
- MainConcept
- DivX H.264
- MediaSDK
- Elecard
- XviD
- Theora
Microsoft Expression Encoder 3 could not be placed in this list because of its much longer encoding time compared with other encoders (except Theora).
Using the standard Theora interface, we could not find any multithreading options; Theora works only in single-thread mode. The leader in this comparison is x264—its quality difference (according to the SSIM metric) could be explained by the special encoding option ("--tune ssim"). Interestingly, using the PSNR metric for MainConcept yielded results comparable with or better than those of x264. This means that no encoder can achieve the best results for both SSIM and PSNR when using the same parameters.
The difference between the MainConcept and DivX H.264 encoders is not overly significant, so these encoders tied for second in this comparison. The developers of the Elecard encoder do not provide a High Speed preset, so its ranking is based solely on the results for the Normal and High Quality presets. The dicas encoder holds third or fourth place for some sequences.
This rank is based only on the encoders’ quality results. Encoding speed is not considered here.
The Graphics & Media Lab Video Group would like to express
its gratitude to the following companies for providing
the codecs and settings used in this report:
The Video Group would also like to thank these companies for
their help and technical support during the tests.
Add to Del.icio.us
Digg It
reddit
