register Homepage

ECLAP 2012 Conference Exposition and Demo Section

Air Drops - Computer vision-based video installation for creative and artistic experiences

Paolo Scoppola
  Abstract—An experience on the perception of space, color and sound. The viewer becomes the artist and paints abstract shapes on the screen by moving the hands in the air. Sounds from the nature are related to the movements of the brush and music is played to stimulate the imagination.
When the painting is finished, the image is recorded and becomes part of an online gallery.
The software behind the installation is not a mere mechanism that just allow a natural kind of painting, but is rather a language created by the artist/programmer to allow viewers become painters and express themselves.

The Digital Revolution


The arrival of a new technology in the world of communications has always caused significant social change.
It is enough to think about the birth of typography or the discovery of radio waves.
In this respect information technology has caused a veritable revolution that is still ongoing. But it is already evident how much mankind’s interaction with images and sound has already changed. In fact, in the latter part of the previous century technological development in the audiovisual sector was essentially focused on reproducing images and sounds with the best possible quality. The dream of reaching perfection was quickly revealed utopic but it had the merit of bringing to the fore attention on the perceptive experience. Then computers arrived and the focus shifted onto ease of use, but above all, thanks to Internet, onto the possibility of accessing an almost unlimited amount of information. In a very short time we have gone from the joy of stopping to listen to a CD or to leaf through a photographic book, to wanting to experience this kind of content anywhere, anytime on the run. This has led to dealing with images and sounds simply as information, removing meaning from content. It is possible to argue at length on the matter, but it is undeniable that information technology has taken on a significant role in the world of media. One wonders if this development has taken the only possible direction or whether there are other possible scenarios. Currently digital technology is utilized principally to maximize the production and exchange of information, reducing it to a mere consumer item.


But an alternative is possible, such as developing new ways of interacting with audiovisual content.
This is the case with interactive installations, which propose as their main theme the sensory experience and the value of the cognitive process. These types of installations benefit from the latest technologies to create environments in which images and sounds are altered in real time by the movement of the viewer. Let’s imagine a projector screen on which the content is determined by the movements of whoever stands before it. The viewer seeks to comprehend the relationship between gesture and the altering of screen content, directing his attention to reflect on what he is perceiving. Consequently he would find himself observing and participating with, how others next to him deal with the same experience. It becomes evident that this perceptive experience is completely different, and that it is in fact raising our awareness. The distinguishing feature of this type of installation is not to be found in the context of the spectacular, but rather in the fact that the viewer becomes aware of the connection between his own body and the audiovisual content. It effectively goes beyond the concept of fruition to question the very value of the cognitive process. What will happen when the ordinary ‘consumer’ of images, begins to question what they really experience whenever they observe an image or hear a sound? And what if this increased awareness of cognitive processes would bring about a greater happiness?

Computer Art and Popular Culture

This project seeks to investigate digital technology and it’s impact on the perceptive experience, and to confront digital art with what is commonly thought of as ‘popular art’ by going from a technological and conceptual environment to an everyday world that is filled with instinctive and emotional aspects. The intent of the project is to bring the installation to a general audience that lies beyond the electronic arts circuit. Therefore as the aim of the project regards the theme of perception, the work wishes to communicate with the audience in an immediate way, without requiring the aid of cultural references. History shows us how painting, music and other forms of artistic expression have always managed to communicate transversally, enriching themselves by connecting with common people. In this respect Air Drops [1] wants to be accessible to an audience located outside the paradigms of the digital art world, which is not partial to a particular technology but nonetheless perceives an inherent need for expression.


Essentially the installation consists of a computer with some special input/output devices and software that uses  computer vision algorithms for real-time editing of multimedia content. Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images in order to produce numerical or symbolic information, in the form of decisions [2]. In this case we use a video camera that faces the viewer in front of the screen and computer vision software that records the streaming video and recognizes the position of the viewer’s hand in the space. Then collected geometrical information is used to move the brush on the screen, create images and modulate the volume. All these contents are generated or modified in real-time, that is, they are not simply a reproduction of contents stored in the computer’s memory [3]. The interest in this kind of installation, beyond the use of specific technologies involved, lies in the fact that what the viewer sees and hears is the immediate result of his gestures combined with the execution of the software. We discuss now how starting from writing the code we get the painting on the screen and how different languages are involved in this process. In this chapter we refer to the artist as the author of the installation and to the painter as the viewer that interacts with the installation.

A.    The Computer’s Language

Generally, software is created by writing code in some programming language or by using visual programming tools [4]. Through this writing, the artist makes all the instruments with which the visitor will realize his painting, like the system to track the hand or the tools to make brush strokes and choose color. So, at the root of what is seen on the screen, there is the computer language and the way in which the artist uses it.

A portion of the code (visual representation) that controls the brush color.

B.    The Artist’s Language


The computer language allows the artist to create the painting tools and it’s clear that the possibilities are almost endless as it is only a matter of writing code. This leaves only the need to choose which tools to offer the painter, that is, we need to create a language of gestures, signs and colors with which the painter can express himself. This is the central point of the installation because this is where we determine what will be the live experience. If the interaction is simple and intuitive, the painter focuses only on the creative aspect, otherwise he remains concentrated on the actions required to overcome the difficulties of use. The sound plays a key role in this process and is present in two distinct forms. On one hand there is a list of songs composed by the artist to help the painter to concentrate and stimulate creativity. Tracks can be selected with a gesture in order to find the one that best matches his mood. On the other, there are a number of natural sounds (water, fire, wind) associated with hand gestures, which vary in intensity depending on the speed of the movement. This is intended to give a suggestive perception of the motion of the hand and in this way, to stimulate the viewer’s gestures.

C.    The Painter’s Language

The viewer approaches the screen and starts moving one hand. In a short time (maximum one minute) he understands the relationship between his movements and the result on the screen. Instinctively, the viewer produces a series of signs with specific traits and colors, which vary from person to person, like a sort of language. By watching the web gallery[1] we see how different the paintings can be, and anyway we see that the result is almost always well defined and pleasing from an aesthetic point of view. This is the result of the combination of three languages: a formal language, the way computers work,and the way the artist with creative tools and methods of interaction finally creates his own expressive language.


How it works


Air Drops is based on rENGINE [5], a software for interactive video installations that the artist has developed for realizing his works. rENGINE takes the images from the camera and sounds from the microphone (not used in Air Drops). It analyzes these data and retrieves numerical information like user position, audio volumes and many others. Then it uses the information to decide how to modify preexistent or how to produce new content and sends it to devices such as video projectors, screens and audio speakers. Air Drops takes advantage from the use of a depth camera, that is a camera that produces black and white images, where each pixel value stands for the distance between the element of scene represented by the pixel and the camera itself. This allows easily separating an object from the background and keep track of its position in a three dimensional space. We use the horizontal and vertical position of the hand to move the brush on the screen and we use the depth position to do many other things, like change the brush thickness or simply lift the brush from the virtual canvas and move in an other point. About the content production, it’s well known today that the computers performance allow developing very complex real-time application, e.g. the 3D video games. Anyway, in this case we avoid showing stunning images, because the painter has to be gratified by its own creativity and not surprised by what the computer is able to do. This is the reason why on the screen there are very simple image elements. The painter has to control the elements, not the opposite.
For the same reason, both the hardware and the software are completely hidden from the viewers. rENGINE, in fact, does not require the viewer to engage with any physical device, allowing a natural and instinctive interaction with the images and sounds. At the same time, the graphical user interface [6] is extremely simple and the least invasive possible. As a result, visitors do not realize they are using a software or do not feel the sensation that their creative experience is mediated through the use of a computer.

Installation hardware


Live experience

The installation is not a physical object; it's more precisely a combination of different elements. The exhibit space plays a critical role, because it determines how much strong can be the cathartic effect of images and sounds. An empty space with low lights conditions works at best, because lets the painter to be concentrated only on the creative process. But it's important to avoid a complete darkness because the painter can lose the perception of space and so, the relation between the own body and the images on the screen. The second element is the audience. In any interactive installation there is always a group of person that do something while the others stay on watching, but the work of the artist has to communicate to both of the them. So the viewers that don't interact should be entertained by the show, especially in the case of Air Drops, where just a person at time can interact with the installation. Here, the action itself of the painter is what entertains the others, exactly like it happens when somebody appears on the stage and gives a performance. A third element is the animator, that usually it's the artist. This person represents for the audience a sort of door through which approaching to the creative experience. The words that the animator says and his way of communicating serve to make understand how to interpret what the eyes see and the ears hear in the exhibit space. Last but not least, we have the painter that becomes part of the representation.  Beyond the software, the music and the exhibit space, what appears on the screen is produced by his/her mind combined with the gestures.  Finally each painter is also a guide for those persons who are waiting for making their paintings and this creates an other kind of interaction that happens exclusively between visitors.


Color painting Black and white painting
Romascienza 2011, Rome Miela Theater, Trieste 2011



The aim of this project is not to create a tool for painters, especially because it’s intended for public events, where most of the viewers don’t have any painting skills. On the other side, the images that appear in the web gallery, even if they have almost always a pleasant and well-defined aspect, do not aspire to be art works. The artistic value lies in the performance of the painter, when he/she starts to interact with the screen and the music which creates an intimate atmosphere and stimulates the viewer’s creativity. This leads to a pleasant discovery of colors, shapes, sounds and notes. The image on the screen contains very simple elements that nonetheless attract the painter’s attention, because they come directly from his/her gestures and therefore take on a completely different value. Finally the desire of doing something nice drives the painter to create shapes until he/she feels that nothing more has to be added to the screen. The painter feels gratification for having made something pleasant and decides to save the painting on the computer. Therefore the value of this interactive installation doesn’t consist of the possibility to do something more complex than just viewing a movie or hearing music. The major point of interest becomes the cognitive process, the possibility to approach the concepts of image and sound in a completely different manner. Air Drops has been presented in several events during 2011 and has been successfully experienced by children, adults, and the elderly.




main Conference page           Call for papers of Workshops            Call for Papers

Register                  Program                        Keynote Speakers            Speaker Information      

            Exposition Information                             Conference Community                                                                               
Program and Organizing Committee 
International Attendees                             Hotel & Travel                         Venue & Location  


Your rating: None