The video game industry creates a huge and steadily growing market that as a whole is worth about USD 70 billion in 2012. Many signals from leading hardware manufacturers and game developers indicate that in the nearest future video games will be enriched by affect-aware capabilities that can dramatically improve and change human-computer interaction. The first symptom of these groundbreaking changes is the popularity of Microsoft Kinect sensor which has become the fastest selling consumer electronics device ever.

The main objective of this project is to develop new emotion recognition algorithms that could be used in affective and affect-aware software, especially video games. Separate algorithms will be created for different information channels: audio, physiological signals, video and scene depth. Also a decision-level multimodal classifier will be developed to increase recognition efficiency and trustfulness of each single channel classifiers. A valuable outcome of the project will also be a reference multimodal database of emotions’ recordings including scene depth information.

The main novelty of this project is aiding optical information with scene depth information for reliable recognition of facial expressions. 3D scene information should significantly improve face localization and matching algorithms when using the active appearance model approach.

Another objective of the project is creation of a complete multi-modal affect recognition system to provide automatic and continuous affect recognition of video game players. This will comprise a complete framework that would enable cooperation between developed affect recognition system and any video game by application programming interface defined within the project.

To validate results of the project a prototype affect-aware video game will be developed that would adopt its parameters, such as difficulty level, pace or humor basing on the current emotional state of a player and challenges in the game.

This project is funded from Norway Grants in the Polish-Norwegian Research Programme operated by the National Centre for Research and Development