The results of Work Package 2 on gathering user requirements have led to a table of goals, from which a set of requirements for image analysis have been constructed. Below is the original Goal list with inserted notes on the possible image analysis approaches. Some requirements are actually composed of sub-requirements, especially G4.

G1 Matching of similar images (includes "have you got this picture")

aspect ratio, histograms, colour regions, edge maps, multiresolution shape finder?

Also finding "Lost" images

Difficulty: 5

G2 Automatic search using synonyms

G3 Search based on the concept of style

depends what sort of "style", texture classifier, colour + texture classifier, neural networks to analyse image features

shape styles etc – need shape detectors.

Artist technique (?)

Difficulty: 8

G4 Search based on features oriented to the restoration framework

UV spotmeasures

histogram based threshold plus counter, binary shrink to count large regions only?

x-ray and reverse pic views of frames

line detectors, hough?, lp filter first?

craquelure classifier

edge/line detectors tuned to cracks, chain + classify? or texture of binary result? Snakes?

Search based on framework features

"butterfly", wood knots, planks, retouching detectors. These use rear-view images and require shape detectors.

Parquetage, stretchers, bars.

Difficulty 9

G5 Access information quickly and easily

Avoiding brute-force searching and image processing during queries. Fast indexing strategies required. May use Java Viseum image viewer for browsing large images.

Difficulty 5

G6 Search based on colour

Image with these colours? (pantone or Lab or sRGB)

convert to sRGB or Lab, look for colour regions at low res

3D colour histogram, similarity matrix

CIE/picked colour (plus damage): as above plus damage detector appropriate

Difficulty 5

G7 Query by low quality images (especially faxes)

scalespace versions of edge maps

Difficulty 7

G8 Query by sketch

scalespace edge map search, shapes searches?

Difficulty 9

G9 Query refinement

G10 Joint retrieval by content and by text

G11 Use of publishing products built on the Artiste system

G12 Detail finding

(eg finding which image a close-up is from): CCV , template matching?: edge map?(scalespace) searching

Difficulty 5

G13 Search using multilingual vocabulary

G14 Respect installation site privacy and security policy

G15 Produce a sustainable system after the end of the project

Document algorithms and code. Help system.

G16 Be consistent with partner’s pre-defined technical constraints.

 G17 (new!) shape of the painting - eg oval, round top etc. (also useful for restricting image matching to real areas)

G18 (new!) - if time allows! - Canvas Weave count. (possibly using X-ray image)

It is noticeable that there are vague requirements about speed, success rate and accuracy as well as human interface issues. It is expected that systems will initially be slower than expected but benefit eventually from faster hardware and software optimisation. It is also hoped that HCI issues will surface as users start to use the system, which is probably their first time with such a system.

We will identify the most commonly needed algorithms and implement these first, as many solutions are based on a combination of functions. Some are available in VIPS or MAVIS already, while others will need coding from scratch. They will be written in C/C++ under Unix for performance reasons, in such a way as to be portable to NT. A project web page will contain the latest software for download, together with help pages and an API.

Goals will be tackled in roughly this order: 1, 12, 6, 4, 7, 8, 3. This will depend on the success rate of algorithms as they are implemented as it is possible that a solution for one goal is found while tackling another.

Clearly multiresolution/multiscalar techniques will be involved due to the nature of some of the problems (eg find where this detail image is from) as well as the range of feature scales (eg small cracks on a 10kx10k image). From the list above certain types of algorithm are needed:

1. Simple aspect ratio/scale comparisons
2. Image statistics
3. Various monochrome and colour histograms
4. Segmentation by colour and shape
5. Image texture classifiers
6. General shape finders
7. Edge detectors and edge maps
8. Colour space conversions: sRGB, HSV, Lab, other RGB
9. Binary processing and morphology
10. General resampling, scaling, rotating
11. File format read/write/compress/etc.
12. Matching, graph matching, indexing, distance measures etc.

Many of these are found in any image processing package but the rest are more specialised or even problem specific variants.

As images are uploaded to the database they are passed to image processing functions which generate image feature vectors (IFVs). These will be stored in the database and comparison functions provided for searching using them. It seems that blobs will be used for the data or one of the built-in types (eg int arrays).

An API will be agreed on for the image processing functions which produce the IFVs. There should be a list of functions, descriptions of their strengths etc, data definitions (for the db fields). Then there is the list of associated "matchers" which produce scores given sets of test images.

To reduce compute-intensive operations the database will store image primitives which are higher-level and faster to compare if possible. For example histograms, averages, perhaps low-res edge maps, FFT, 3D colour histogram etc.

Starting with the simplest image processing and moving through to the latest, we expect a growing list of IFVs like:

aspect ratio, h/w, float

mean, mean of all pels, int

realscale eg mm/pel?

moments... various

FFT power spec, int[]int[]

texture... various, int[]

colour texture measures

spatial colour histograms

In ECS/IAM we will be cable to develop the image processing funtions and test them standalone or with MAVIS2. Then wrap them for the NCR database, or our next version of MAVIS using our database infrastructure.