Rosa Alice Branco
(PHD) Associate Professor/ ID+ (Instituto de Investigação em Design Media e Cultura) / ESAD (Escola Superior de Artes e Design) Porto, Portugal 
rabranco@gmail.com   

Hugo Branco
(Master in Digital Arts at IUA – Pompeu Fabra University, Barcelona) / ESAD, Caldas da Rainha, Portugal
hugobranco@gmail.com

Abstract

Whenever one creates an object, a system of thought, an equation – with full consciousness of one’s objectives – one can never know for sure which will be the full nomad extent of the provoked phenomenon.

In this sense, we hereby propose an itinerary which has entertainment as its starting point, then taking us through Design, Cognitive Sciences, Psychology and back to Design, Sciences and Life.

In the present case, we take a tool developed as a videogame controller and – through an experimental process – the extrapolation of this tool for the field of Sound Design.

This device belongs to the BCI (Brain Computer Interface) class, and was originally developed to substitute the use of the mouse/joystick in videogames. Nevertheless, with the development of specific software it is now practicable to employ this device for controlling any music production software, which means the possibility of modulating sound with the mind.

By the means of three sensors placed on the user’s forehead, the ‘NIA’ (Neural Impulse Actuator) reads the electrical impulses generated by the Alpha and Beta waves of the brain and translates them into command messages.

Therefore, this investigation becomes also an attempt to understand the connection between the brain and its object. This leads us directly into the field of Cognitive Sciences and Psychology, since we’re asking the brain to operate an innovative performance regarding our intuition. The plasticity of the brain is a main issue in this process, for it allows unexpected possibilities in what concerns learning and changing throughout life.

If we know how to deal with our brain to control the frequency of its waves, how to use the different capacities of each hemisphere, how to develop and restrain some areas, we might in fact approach a state of consciousness where rationality also integrates highly developed instincts and intuitions.

Back to Design, mental training can – on one hand – allow us to better control these new tools in the context of experimental processes in Design, as it can – on the other – lead us to a more adequate view and practice of Design: one that manifests our commitment to the world.

Entertainment

In the present case, we take a new tool developed as a videogame controller and – through an experimental process – the extrapolation of this tool for the field of Sound Design (in which Sound Design is an example from which one can foresee further uses of this tool in other fields of Design).

This device, called NIA – Neural Impulse Actuator – belongs to the BCI (Brain Computer Interface) class, and was originally developed to substitute the use of the mouse/joystick in videogames. It has also been used to include people with limited mobility that can’t actually use their hands to play games, since with the NIA one can control the game without needing keyboard strokes or mouse clicks, using only brainwaves and/or facial expression. 

(Sound) Design

Entertainment taken as starting point, one can now move on and cross borders: what if, with the NIA, we can apply brainwaves and facial expression to creative uses, such as, let’s say, Sound Design? Actually, with the development of specific software it is now practicable to employ this device for controlling music production software, which means the innovative possibility of modulating sound with the mind.

By the means of three sensors placed on the user’s forehead, the ‘NIA’ (Neural Impulse Actuator) reads the electrical impulses generated by the Alpha and Beta waves of the brain and translates them into command messages: these command messages are then translated by our developed software into MIDI or OSC messages, which can be read by any music production software such as MAXMSP, Pure Data, Reactor or Ableton Live. [1]

Cognitive Sciences and Psychology

Due to the nature of our study, this investigation becomes also an attempt to understand the connection between the brain and its object. This leads us directly into the field of Cognitive Sciences and Psychology, since we’re asking the brain to operate an innovative performance regarding our intuition. The plasticity of the brain is a main issue in this process, for it allows unexpected possibilities in what concerns learning and changing throughout life.

Brain plasticity: the tool for changes

Neuroplasticity can be understood as the brain’s capacity to reorganize when submitted to changes, which can be, in fact, related to several practices.

The concept of neuroplasticity is somehow recent. Not long ago, the immutable structure of the brain in its adult state was an accepted fact: neurons died without others being born to replace them. Today it is proved the neurogenesis exists, though the neurons are born in a much smaller amount then the ones that die. Brain plasticity is nevertheless much more relevant in what concerns the amount and quality of synapses – functional connections between neurons – that can multiply throughout all our life, as easily as they can break, since the brain follows a law of economy which one can translate by the expression: “use it or lose it”.

On the other hand, it is known that learning is possible not only through new connections, but also by reinforcing the existent ones. This function’s fundamental part is the hippocampus, a core zone of memory, since its primary function is to transform short term into long term memory, with multiple and consistent connection cables.

Another great expression arises here: “Neurons that fire together wire together”.

The most interesting issue about brain plasticity is that some changes in brain circuits occur very fast and can even include our attitude towards life. In fact, if one stimulates one’s left prefrontal area, and inhibit the excessive activity of the correspondent area in the right hemisphere, will result in switching from a more pessimist and aggressive state to a more optimist and conciliatory one. Therefore, brain plasticity allows (and is allowed by) brain modulation as well as the awareness of this modulation.

Regarding the amplitude of this plasticity, one question immediately arises: which is the relation between brain plasticity and genetic determination? As a matter of fact, genetic determination is as meaningful as environmental and psychological determinations, articulated through brain plasticity. What we have here is a 

“modulation of the expression of a genotype (the genetic constitution of an individual) by environmental or cultural factors, beyond the concept of interaction” [2] (Ansermet and Magistretti 2004, p. 24).

This means that if there are many, nowadays, who consider that brain plasticity is the founder of a new paradigm, that’s because it opens the possibility of thinking about the transformation of brain circuits through epigenetic changes (external to the genome) like cultural factors, educational and environmental variables, etc. The environment stimulates and the brain reacts, through changes in neuronal circuitry, in the frequency of the mind waves, and changes in some actions and behaviors.

 The role of the trace in the creation of automatisms

Experience leaves a footprint in memory, a mark, a “trace” (in psychoanalysis) that modulates behaviors. The brain has very fine mechanisms to store these pieces of information and to manage their recall whenever necessary. It is clear that the hippocampus plays a fundamental role in this process, since it holds responsibility for turning short term into long term memory, which can last a whole life.

Sometimes, when we already possess the appropriate savoir-faire for a specific situation, the footprint gets spontaneously imprinted. These are the cases where continuous experience creates automatisms. 

Trace, in fact, being a place of reinforcement and reinforced throughout life, is in many circumstances unconscious, in others just interiorized, which allows us to take action through instincts in a progressively more adequate way, on one hand, or a progressively more creative way, in the other. [3]

In cases of extreme persistence and efficiency in learning (great reinforcement of the trace), the modulation can be of such order that the function of a component of the brain can be altered like in the case of the chess champion, Susan Polgar.

The nomadic case of Susan Polgar

Susan Polgar spent her childhood and adolescence playing chess and studying catalogs of famous games, at least six hours a day: it was astonishing to see a little girl wining against famous chess players. Later, in adulthood, she developed a skill that allowed her to win a chess game in sixty seconds only. But how does she manage to play – and win – without time for thinking?

About this incredible performance, Susan Polgar stated that she plays using only her intuition, to which scientists have responded with a comparison between this process and some kind of direct line from brain to hand.

Since Susan Polgar accepted to submit herself to tests concerning new brain imaging, one was able to better understand the process used in her 60 seconds games:  it was verified by functional magnetic resonance that the fusiform gyrus, responsible for the recognizing of faces, acted on her also as an recognition operator for chess configurations, allowing decisions to be as fast as our most intuitive responses, like recognizing a face or escaping a collision trajectory. [4]

From this case we can draw an essential conclusion regarding both brain plasticity and intuition: neuroplasticity can operate the nomadic passage of reasoning to intuition and automatisms.

Other nomadisms concerning the brain

In case of injure, or in the specific case of loss of the kinesthetic sense – responsible for feeling one’s body and for the pressure and effort necessary to undertake actions – most people become immobilized. Nevertheless in some famous cases, due to the persistence and intelligence of the situation, some patients could act autonomously without ever arriving to recover feeling of the body. In fact, they developed automatisms that made it possible for them to acquire a “second nature” without nevertheless feeling their own body ever again.

These automatisms are only made possible by the plasticity of a brain that reorganizes itself, after persistent reinforcement of the new feedbacks for action. These new excitements of the neuronal device allow the individual to move forward from perception to creative action. Therefore, neuroplasticity is always modulating the brain, like an architect that lets his technical knowledge be guided and achieved by the inspiration of the context: place, landscape, regional materials, weather, and even a tree that grows alone in the horizon of the vision. The modulation of the brain goes from imprinting experience – its persistence and new variables – to the construction of a singular imaginary. [5]

Plasticity of emotions

This process alone is enough to explain why we act and develop adequate behaviors, like cutting the cake at one’s wedding, making a toast, kissing, (Ansermet and Magistretti 2004. p. 44). . [6] But this process doesn’t explain, nevertheless, why the middle sister loses her appetite when she remembers her older brother’s wedding, to which she had attended with her husband, who in the meantime left her

When a stimulus from the external world is associated to a perception, it is also associated to a somatic, body state, which originates the perception of an emotion. It’s highly understandable that anxiety for example, couldn’t exist without the accelerated breath and heart beat, sometimes cold sweat, stomachache, trembling legs. [7]

Also according to Ansermet and Magistretti, the amygdala – the organ that plays the main role in cases such as fear and anxiety 

“connects perception and its imprint to the unfolding of somatic reactions, and, at the same time, it sends information to the procedural memory through the pre-frontal cortex” (2004, p. 185) [8].

It is know that António Damásio’s great thesis lies on the fact that the pre-frontal area is responsible for emotions and the decision making ability (Damásio.1995 and 2010). [9] Thus, it shouldn’t surprise us that the unconscious traces – the ones directly imprinted in the amygdala– constitute the unconscious reality that may also constitute new stimulus (Ansermet and Magistretti 2004, p. 185) [10] . 

The more the brain is modulated by mechanisms of synaptic plasticity, the more there is the possibility of creative action. 

Back to (sound) design: introducing the NIA

BCIs were originally developed regarding health purposes such as assisting, augmenting, or repairing human cognitive or sensory-motor functions. It was nevertheless impracticable to ignore the infinite possibilities and incredible potential concerning creative and experimental uses of this kind of device.

Aiming to put in practice our study of the relations between brain and object in the 

context of experimental processes in Design, we found a Brain-Computer Interface called NIA (Neural Impulse Actuator), a device which seemed simultaneously accessible, affordable and simple enough to use as starting point for our tests on the subject.

The NIA belongs to the non-invasive BCIs category, meaning that the sensors that translate brain activity into information aren’t implanted inside the grey matter of the brain (invasive), neither are they implanted inside the skull outside the brain (partially invasive), but rather outside the skull. This kind of device is easier to wear and safer to use, though in comparison to the others it produces poor signal resolution – since the skull dampens signals, dispersing and blurring the electromagnetic waves created by the neurons. It seemed nevertheless much more practicable for this kind of context.

The NIA makes use of a headband placed on the user’s forehead, featuring three sensors that will read the frequencies of the user’s brainwaves, as well as eye movement and facial tension. Though there are several other kinds of brainwaves, the NIA only works with alpha and beta waves, which are respectively 9-14Hz and 15-40Hz. [11]

Input 1: brainwaves

Brainwaves display electrical activity emanating from the brain. Let’s consider four different types of waves, according to amplitude, frequency and level of brain activity.

Beta waves are the fastest (15 to 40 cycles a second) and so they have relatively low amplitude. They are generated when the brain is aroused and actively engaged in mental activities. They correspond to states of outward awareness and are also generated by fear, anger, worry, hunger or surprise: beta waves are characteristic of a strongly engaged mind.

With 9 to 14 cycles a second, alpha waves represent non-arousal. They are slower then beta waves and higher in amplitude. Corresponding to a lesser engagement and arousal, alpha waves are generated during non-drowsy relaxation, like tranquil states of consciousness, pleasant inward awareness, body/mind integration, and meditation. Someone who has completed a task and sits down to relax, or takes time out to reflect is usually in an alpha state.

Theta brainwaves are of even greater amplitude and slower frequency (5 to 8 cycles a second). They are usually generated when performing tasks that become so automatic that one can mentally disengage from them – like driving on a freeway and noticing that one can’t recall the last five minutes. The continuous and repetitive nature of the action draws them from outward and focal to inward and peripheral attention, generally leading to states of recall, creativity, imagery and visualization, free-flowing thought, future planning and inspiration. Theta waves are also dominant during dreaming and REM (Rapid Eye Movement sleep: a normal stage of sleep characterized by the rapid movement of the eyes, where most vividly recalled dreams occur) states.

The fourth kind of brainwaves, delta, are of the greatest amplitude and slowest frequency (1,5 to 4 cycles a second). As far as we know, they correspond only to deep dreamless sleep. When active dreaming takes place, delta brainwave frequencies increase into the frequency of theta brainwaves. [12]

All four brainwaves are common to the human species, and though a brainwave state may predominate at a given time or situation, depending on the activity level of the individual, the remaining three brain states are also present at all times.

The knowledge of brainwaves and their corresponding states, together with an active modulation of the brain according to these principles, can optimize one’s ability to make use of the specific characteristics of those states, and enhance the brain’s plasticity over time, allowing us to be more mentally productive across a wide range of activities, such as being intensely focused, relaxed or creative.

Input 2: eye movements and facial tension

Both muscular activity and eye movement – which constitute, along with alpha and beta brainwaves, active inputs to the NIA – are also determined by emotions. For instance, the dilation and retraction movement of the pupil of the eye, usually attributed to the amount of light that reaches the eye, is also determined by the emotional state of the individual.

Output: command messages

The electrical impulses generated by Alpha and Beta waves, (frequencies to which the NIA is receptive) along with eye movement and facial tension will be translated by the NIA into command messages: originally developed for gamers to substitute/aid the use of the mouse/joystick in videogames, the development of specific open-source software made it possible to transform these command messages into MIDI messages.

Defined in 1982, MIDI is an industry-standard protocol that enables electronic musical instruments (synthesizers, drum machines), computers and other electronic equipment (MIDI controllers, sound cards, samplers) to communicate and synchronize with each other. Being able to convert brain signals ,via NIA, into MIDI messages, means that it is now practicable to employ this device for controlling – amongst other things – any music production software, which means the possibility of modulating sound with the mind. [12]

Experimental use of the nia in sound design for video

Taking Sound Design as a starting point and test zone for our research, let’s consider, for example, designing sound for a video work: having already prepared a basic set of sounds, rhythms or melodies, the sound designer can then use the ‘NIA’ (in substitution of the mouse and keyboard or the knobs and buttons of a MIDI controller) to work on the remaining creative choices – like the variation of sound levels over the time, or the employment of diverse sound effects. [13]

Self sound design in real time

Through this instinctive and connection between image and body reactions, in this experimental context, Sound Design for video emerges as a result of an intuitive process, highly variable from individual to individual, since it finally depends on the emotions that result in organic actions related to the brain, the eyes and the muscles. Furthermore, it is also variable concerning the same individual, depending on how many times he has watched that specific video, the circumstances in which he’s watching it, his state of mind, etc.

One can imagine an experiment in which the Sound Design work is done simultaneously to the visualization of the video (without stopping or rewinding). In this context, our brainwaves, eye movement and facial tension are surely influenced by the visualization of the video in real time, so, in a certain sense, we are letting the video itself (via the reactions caused to our brain, translated in electrical signals, captured by the ‘NIA’ and applied to the music production software) decide on a certain number of creative choices. Could this not be regarded as a kind of ‘self Sound Design in real time’? 

further developments

Enhancements to this investigation can now take place through the extrapolation of this particular experimental process into other areas of Design and creativity. One can also achieve better results by testing the use of other devices which can more effectively put our ever learning intuition and instincts to a progressively more direct use concerning the creative process.

To succeed in this task, not only the right technologies are required but also a persistent and effective study of the brain, together with intense mental training. A trained, clear, receptive brain, can much more effectively receive outside input, converting it into the output command messages that will determine the result of the creative act, guided by the use of intuition.

Enlightenment

It’s somehow not usual to find words such ‘Enlightenment” in a Design paper. In fact, in spiritual terms, “Enlightenment” refers to a spiritual revelation or deep insight into the meaning and purpose of all things, a profound spiritual understanding: a fundamentally changed consciousness whereby everything is perceived as a unity. [14]

In a secular context though, the word often means the full comprehension of a situation, and takes the broader sense of wisdom, understanding and clarity of perception. The concept refers to the 17h and 18^th centuries’ European intellectual movement known as the Age of Enlightenment or Age of Reason, in which major philosophical developments related to scientific rationality took place.

Concerning our present study, “Enlightenment” is seen here as a progressive synchronization of the subject with the world, though a process in which rationality also integrates highly developed instincts and intuitions.

If we know how to deal with our brain in order to control the frequency of its waves, how to use the different capacities of each hemisphere, how to develop and restrain some areas, we might in fact approach a deeper ‘comprehension of the situation’: a state of consciousness close to what we can call “Enlightenment”, one that grants us better perception of the world and its present needs.

Back again to design and life

Back again to Design, mental training can allow us to better control these new tools in the context of experimental processes in Design (it is proven that meditation provokes physical changes in the brain – which we now know are allowed by neuroplasticity).

The further we develop the right technologies and interfaces for receiving and converting brain data into command messages, and the better prepared our brains are – allowing us to achieve deeper control over the use of our intuition and instincts (like in the case of Susan Polgar) – the better we will be able to depend solely on our brains in order to control software and also hardware in virtual and real environments. The experimental, creative and practical applications for this kind of brain-computer interaction both in the field of Design and in other fields of human activity, are almost infinite. [15]

Mental training, together with neuroplasticity, allows changes in brain circuits which can even include our attitude towards life. Through the progressive synchronization with the world we like to call “Enlightenment”, this process leads us to a much more adequate view and practice of Design: one that manifests and reflects our deep comprehension and commitment to the world.

References

[1] Cf. Kowalski, C 2008, OCZ’s Neural Impulse Actuator – The flying car of control schemes, viewed 23 October 2010, http://techreport.com/articles.x/14957

 [2] [4] [6] [8] [10] Ansermet, F and Magistretti, P 2004, À chacun son cerveau, Paris, Odile Jacob. 

[3] Cf. Gibson, J 1986, The ecological approach to visual perception, Lawrence Erlbaum, New Jersey and London.

 [4] Cf. Polgar, S, 2007, Make me a genius, viewed 20 October 2010,

http://susanpolgar.blogspot.com/2007/08/national-geographic-my-brilliant-brain.html

[5] Cf. Doidge, N 2008,  Les étonnants pouvoirs de transformation du cerveau. Belfond, Paris.

[7] Marendaz, C 2009, Du regard à l’émotion : la vision, le cerveau, l’affectif, Le Pommier, Paris.

 [9] Damásio, A 2010, O livro da consciência: a construção do cérebro consciente, Temas e Debates, Lisboa. Damásio, A 1995, O erro de Descartes: emoção, razão e cérebro humano, Publicações Europa – América, Lisboa.

[11] Eric C, Schalk G, Roland J, Rouse A and Moran  D 2009, Evolution of brain-computer interfaces: going beyond classic motor physiology, viewed 06 October 2010,<http://thejns.org/doi/full/10.3171/2009.4.FOCUS0979>

[12] Vásquez,M 2009, Digital creativity, positioning & promotion,viewed4 October 2010,<http://www.revistawide.com.br/index.php/find-design-latino-americano/>

[13] Dehaene,S and Petit, C 2009, Parole et musique, Odile Jacob, Paris.

[14] Ricard, M. 2005, Em defesa da felicidade, Pergaminho, Lisboa.

[15]Hall, E 1986, A dimensão oculta, Relógio d’ Água, Lisboa.