DIgital MOtion Picture Restoration System for Film
Archives (DIMORF)[1]
A complex
solution for film scanning, processing and recording
M. Bölecz, L. Czúni, B. Gál, A. Hanis, L. Kovács, B. Kránicz, A. Licsár,
T. Szirányi
Department of Image Processing and Neurocomputing, University of
Veszprém, Hungary
and
I. Kas, Gy. Kovács, S. Manno
Computer and Automation Research Institute of the Hungarian Academy of
Sciences, Hungary
ABSTRACT
A new
semi-automatic digital restoration system is introduced, including all stages
of restoration, i.e. scanning, processing and recording. The featured
automatism is controlled by occasional operator interactions. All main
components are designed and are being manufactured to meet the special needs of
film archive specialists.
The film
scanner, as well as the laser recorder, reaches the resolution of 6K. The
scanner contains a line camera and is able to scan the whole surface of the
film including film perforation. The laser recorder, under construction at the
time of this proposal, uses a rotating mirror to reflect light onto the film
positioned on the inner surface of the recorder drum. The PC-based digital
processing unit includes a standalone framework and has a plug-in based modular
interface ready for forthcoming restoration algorithms. Built-in film
correction includes: stabilization, de-flicker, blotch detection, scratch
removal, de-noising, colour manipulation, de-fade effects. Interactive user
support and adaptive algorithms are designed to help semi-automatic operation
and thus reduce manpower costs.
INTRODUCTION
In the Hungarian National Film Archives (HNFA), similarly to
other film archives of the world, there are thousands of films to be saved from
final perdition. DIMORF is a product of efforts to create a relatively low cost
system capable to handle all degradations, especially those typical in the
HNFA. The proposed restoration system is based on a cooperation of the Computer
and Automation Research Institute of the Hungarian Academy of Sciences, the
Image Processing Laboratory of the University of Veszprém, the HNFA, and Cortex
Ltd. Our main objectives were to create a system that meets current and future
needs of film archivists:
1.
To handle all major types of film errors.
2.
To create an open interface for future algorithms.
3.
To reach 6K resolution as an upper limit for high-resolution
films.
4.
To make the application user friendly and to support the
work of operators with easily understandable digital report files.
5.
To enable future data-mining and intelligent process control
where XML data structures record all user interactions.
Special requirements of the HNFA, regarding the restoration
of a large number of archive films, are:
6.
Usually there is only one remainder copy of an artwork in
bad physical condition. This means that only one scanning procedure is
possible.
7.
Due to the long restoration time (at high resolution)
continuous operator intervention is not possible. The components of the systems
must operate parallel in semi-automatic mode, with occasional manual interventions
for fine-tuning.
8.
The life cycle of the restoration process can be greatly
varying but it is roughly according to the following schedule:
a)
Visual investigation to decide the degree of degradation and
the necessary pixel resolution since most archive films does not require 6K.
Library work;
Physical
cleaning and some repairing;
Determining
and setting the scanning parameters (lighting, physical scratch removal by
liquid-gate, application of infra-red scanning);
Scanning.
b)
Estimating and planning the necessary restoration tasks.
Defining main
cuts, marking main degradation places and features (approximate position, noise
level, scratches, etc.);
Starting
automatic restoration.
c)
Periodic manual interventions: supervision of film
reconstruction, fine tuning of parameters, ROI selection, defining further
batch filtering.
d)
Last step of the restoration cycle: final qualification
leads to new iterations or to the decision to finish the digital restoration
work and to transmit data to the recorder.
e)
Preparation for film writing: the method of frame
enlargement (blowup to 6K with special methods giving natural grainy sensation)
is to be determined, final grading according to the raw film material.
9. All steps of the restoration process are recorded in XML
files to achieve reversibility. With the help of these data the restoration can
be reproduced at different scales and further analysis is possible.
Many of the above requirements are related to the problem of
high costs of manual film restoration, the most relevant problem of the
restoration industry [2].
That is why the proposed algorithms and software solutions support an open,
expandable, and well-structured system driving towards built-in intelligence
and automatisms.
SYSTEM DESCRIPTION
Previous Work in Analogue
Film Copying
At the SZTAKI a film copy machine called “Film Saver” was created for
saving films of extremely bad physical condition [3]. It applies vacuum-wheel film driving, instead of
perforation needles, while the precise, CCD sensor controlled, repositioning of
the transfer optics compensates film shrinkage. In case of films in very bad
condition this device could be used for film saving, serving the input for
further digital restoration.
System Architecture
The system consists of 5 components described below.
1.
Very high-resolution film scanner.
The film scanner utilizes 3 units of 6K line
CCDs (manufactured by Kodak) with top sensitivities at 450, 550, and 650nm.
Light is supplied with a halide incandescent lamp through fiber optics. Pixel
data are transferred in three independent data channels to achieve reliable
very high bandwidth transfer at 6K resolutions. Besides the line sensors, a
low-resolution (PAL) camera is built in for fast online monitoring of the
scanning process. Scratch removal is supported with an infra sensor sensible to
light from 700nm to 1300nm. High resolution scanning of a 35mm frame is below
10sec and at a 3 times reduced resolution (the same device in a different operating
mode) it is below 3sec. Additional CCD sensors are responsible for the
detection of the position of perforations.
Figure 1. High resolution
film scanner
2.
Digital film restoration
workstation(s)
To achieve high compatibility and low cost, film restoration
algorithms are implemented on the Intel-based PC architecture running
MS-Windows operating system. Software applications support multithreading
technology thus multi-CPU platforms are proposed. The digital restoration
project and its operations are hierarchically organized into Projects, Tasks,
Jobs and Filters. This opens the gate for further extensions to multi-platform
processing with the help of Gigabit Ethernet network.
3. Sound processing unit
Sound is processed independently from images via proprietary digital
optical technology [7]. Conventional sound processing is also possible for
further sound reconstruction.
4. Film recorder
In the film writer a high-speed rotating mirror
projects modulated laser beam onto unexposed film positioned on the inner
surface of the recorder drum. The film is fixed in the inner arc (of the drum)
by vacuum to guarantee mechanical stability and accuracy. Laser light is
produced with Point Source lasers at 405, 532, and 640nm.
5. Storage
Storing high-resolution film data requires huge
amount of disk space. To keep costs low RAID controlled EIDE disks are used in
a dual CPU PC host. All devices are connected to this Terabyte capacity host
via Gigabit Ethernet network.
Since both hardware and software components are manufactured
by the members of consortium all parts are possibly jointly optimized.
State of work: the scanner and the digital restoration unit
are already in laboratory service while the film recorder is just being
manufactured at the time of submitting this paper.
DIGITAL RESTORATION
DIMORF has the ability to correct several types
of film error:
·
Film
vibration
·
Flickering
·
Blotch
removal
·
Noise
·
Scratches
·
Color
fading
Recent results of restoration algorithms have
been presented at [3] and [6]. Great impact has been made to
achieve automatic operations. For this purpose film analysis is supported by
cut detection and by film indexing based on color information and motion
activity [1]. Data representation in XML is
also aiming to create well-defined and controllable processes.
In the next sections automatic stabilization,
indexing, cartoon rendering is discussed in some details. The restoration
environment is also introduced at some level.
Automatic
stabilization
Eliminating the vibration of a degraded image sequence
causes difficulties in an automatic film restoration system. The vibration is
usually caused by improper film transportation during the recording, copying or
the digitization process. We have developed an automatic method for image
stabilization consisting of two main steps: estimating vibration then
correction by drifting the whole frame. Earlier stabilization algorithms are
unsuccessful in cases of multiple motions and human interaction is necessary to
achieve satisfactory results. Our algorithm is automatic, robust on noisy films
and avoids false results for most difficult sequences.
The observed vibration of films can be very complex since camera
ego-motion can also cause vibration having serious effects on the 2D projection
of 3D sceneries. There are cases where the whole image cannot be tested for
estimating the stabilization parameters, since the scene may contain multiple,
complex object motion. Consequently, recent systems need manual selection of a
base point or an object for adequate stabilization. This manual task is time
consuming so the aim of the proposed method is automatic vibration
stabilization without any manual interaction. This is achieved by means of
automatic ROF (Region of Fixation) selection in the image sequence. The motion
information of these selected ROFs will be applied for the stabilization of the
images.
We combined the phase correlation motion estimation method [5] with a top-bottom image splitting algorithm based on
motion activity. This motion estimation method is relatively insensitive to
fluctuations in image intensity (flicker and blotches are very typical for
archive films). For finding ROFs in a scene the images are divided into
sub-regions in a quad-tree manner. If the motion observed on whole image can be
well characterized with global motion then the proposed method uses only the
first level of the quad-tree (i.e. the whole image). In case of complex scenes
images are divided into sub-regions due to local objects motions. Analyzing the
motion trajectories of regions (the leaves of the quad-tree) we can find a ROF
or groups of ROFs which describe adequately the dominant motion of the whole
image. These ROFs are valid for a predefined length of app. 1 - 0.5 seconds
(depending on the constancy of the film). Next step is the filtering of the
motion of ROFs to estimate the ideal noise-free motion and then comes the
repositioning to obtain a stable sequence
In the example called “Chevy” the input sequence contains camera motion,
strong local motion and has definite vibration either. In Figure 2 (a) two
frames of the sequence can be compared visually (upper line frame: #13, bottom
line: frame #24). When the non-adaptive technique is applied (right images)
then the horizontal position of the right lamp of the car is rapidly altering.
That is the global stabilization technique fails while automatic (adaptive) ROF
selection stabilizes the sequence (middle images). Figure 2 (b) compares
horizontal motion in a graph.
(a) (b)
Figure 2. Left: input, adaptive and
non-adaptive filtered frames #13 and #24 of sequence “Chevy”. Right: the motion
of the car measured manually in direction X.
For better visual presentation of results please visit http://www.knt.vein.hu/~dimorf/demo where more details of the proposed method can be
found [6].
Using multi-scale processing the computational time could be
reduced. It means that the first few levels of the tree could be processed at
lower resolution with satisfactory results.
Testing on 2K
and 4K (horizontal resolution) image sequences, we found that if the detection
is done on 4:1 downscaled frames, the result of stabilization doesn’t differ
considerably from the stabilization done at the original scale (Figure 3 is a
typical result of such a comparison). It also turned out that this amount of
downscaling leads to reasonable gain in processing time: the example of Figure
3 shows time savings when processing a 1K 24bit image with an Intel P4@1.8GHz.
At 4:1 downscaling the computation time reduced from 7.3sec/frame to
0.9sec/frame. Further improvements in speed are expected after some code
optimization.
Figure 3 Left: Results of stabilization with different downscaled frames.
“Vibrating” stands for the original input, “stabilized” stands for the
differently downscaled stabilized outputs. Right: Relative stabilization times
with downscaled frames (with different ratios).
Film indexing
To support the semi-automatic configuration of filters the systems
builds up a knowledge database, which contains the parameters of manual user
interactions (such as fine tuning). These data can be correlated to the
different shots of a movie and thus the system can propose adaptive filter
browsers for supporting the operator’s decision making.
Image indexing is applied to representative frames (r-frames) of shots. Shots are detected
automatically by cut detection techniques. In the current stage two types of
content based indexing can be used for finding similar shots within or between
films. In both cases representative frames to be indexed are selected according
to the similarity of frames’ histogram to the histogram of the average frame of
the shot.
1. Indexing color signatures. In this case for the r-frame we choose the frame that has a
histogram closest to the histogram of the average frame of the shot. Similarity
can be measured with the Euclidean norm. 13 color signatures are extracted from
r-frames and indexed in database files.
2. Indexing motion vector information. The selection of
representative frames is the opposite of the previous technique. r-frames are those having histograms
less related to the average histogram of the shot. Then a few (app. 10-20)
frames, around each r-frame, are
analyzed for estimating motion. Motion is measured on 5 blocks (the four
quadrants of the frame and the central region) with the phase correlating
technique. Obtained motion vectors are indexed and can be used for retrieving
shots with characteristic motion (such as camera pan, tilt, zoom, or other
object motion).
Alternative
techniques for cartoons
NPR (Non-Photorealistic Rendering) painting techniques could
also be used with some filtering/restoration or classification goals mainly for
the processing of cartoons. For example, the stochastic painting technique [4]
has the ability to lessen errors produced by block-based coders, to smooth
surfaces, to filter out salt & pepper-like noise and other artifacts, while
preserving contours and main edges, producing a visually pleasant output.
Painting (i.e. image representation by stroke-sequences) can also be used for
indexing purposes [8].
Restoration environment
The overview of the digital restoration software environment
is in Figure 4 (a). Digital filters are responsible for the removal of film
errors. The implementation of a filtering algorithm must follow the specific
rules of a filter interface definition. This enables DIMORF to have future
filters as plug-ins simply added to the system. These filters are organized
into jobs which are grouped into
tasks. This enables the operator to define well-separated (and possibly)
parallel restoration processes. All information that can be described textually
(and does not exceed a certain amount of size) is stored in XML files. XML has
the advantage to describe information in well-defined structures (and it is
also notable that with simple transformations <<such as XSLT>>
information can be transformed into readable format).
Figure 4. (a) Main objects of the digital restoration environment in DIMORF.
(b) multi-level restoration.
Figure 4 (b) illustrates the use of XML data structures in
multi-level film restoration. For many types of degradation (f. e. color
fading, flickering) film analysis is accomplished on a low-resolution version
of the input sequence. Results of the analysis are recorded in XML files and
interpreted by the restoration algorithms for applying on the high-resolution
images.
CONCLUSIONS
AND FUTURE WORK
We have introduced a complete digital film restoration
system consisting of both hardware and software components. The DIMORF project
aims restoration at high-resolution with semi-automatic functions on the PC
platform to obtain a low cost solution for film archives. The film scanner and
recorder work at resolution of maximum 6K, which is adequate for the highest
quality requirements in future.
The digital restoration processes advocate multi-level
restoration and digital report files by XML description. Multi-level
restoration, where image analysis and synthesis is separated and can operate at
different scales, decreases computational costs tremendously when working on
high-resolution image sequences.
The color transmission of the scanner and the modeling of some possible
aging effects of films are examined. At the current stage we are investigating
the color transfer model of the film processing chain including the film
recorder under production. The developed system is capable of structured data
collection for future data-mining applications to increase system automation
and intelligence.
REFERENCES
[1] Hanis, A., Szirányi, T. Measuring the motion
similarity in video indexing, Proc. of 4th EURASIP, Zagreb, 2003.
[2] Kokaram, A,
et al. Robust and Automatic Digital Restoration Systems: Coping with Reality, Proc.
of IBC, 2002.
[3] Kovács, Gy. et al. DIMORF, Proc. of 2nd
and 3rd International Conference on Film Restoration,
Budapest, 2002 and 2003.
[4] Kovács, L. and Szirányi, T. Creating animations combining stochastic
paintbrush transformation and motion detection. Proc. of 16th ICPR,
Quebec, Canada, 2002, vol. 2, pp. 1090 to 2002.
[5] Kuglin, C. D. and Hines, D. C. The phase correlation image
alignment method. Proc. of International Conference on Cybernetics and
Society, IEEE, 1975. pp. 163 to 165.
[6] Licsár, A.,
Czúni, L. and Szirányi, T. Adaptive Stabilization of Vibration on Archive
Films. Proc. of CAIP, 2003.
[7] Bisztray, F.,
Erdélyi, G., Feketü, J. Manno, S. and Méder, I. Method and device for the
correction of errors of the injured or scuffed sound recording on sound-films.
Patent P0201132, Hungarian Patent Office, Budapest, 2002.
[8] Kato, Z., Ji, X., Szirányi, T., Tóth, Z., Czúni, L. Content-Based Image Retrieval Using Stochastic Paintbrush Transformation. Proc. of ICIP'2002, IEEE, Rochester, 2002.