BCI Controllers

There are a number of different control schemes you can use to integrate BCI into your game. We have even developed some tools to help get you started so you can focus on designing your game, without having to program all the BCI bits yourself!

The P300 Event

The P300 event is a pattern that happens in your brain activity when you are focused on something and a sudden, unexpected change occurs (this is called the "oddball paradigm"). This pattern shows up as a positive spike in amplitude approximately 300ms after the unexpected change occurred (hence the name P300). This event appears in a known region of the brain, that we can measure using electrodes placed on the scalp over that area.

To use this event as a control scheme for a BCI-enabled game, we visually flash objects on a grid on the screen. The user simply needs to focus on a specific object in the grid, and when that object flashes, the brain detects the flash as an unexpected change and thus the P300 response appears. The BCI monitors brain activity for that P300 event, and associates it to object that flashed at the same time point the P300 appeared in the brain activity. In that way, the user is able to select an object in the grid simply by visually focusing on it.

We have created an adjustable tool in Unity that will set up this grid, execute the random flashing sequence and take care of the brain activity processing in the background. You can take this tool and adjust the size and shape of the grid, change the types of objects that are flashing, and even implement auditory feedback to incorporate P300 into your game however you want. The only thing that matters is that the oddball paradigm is kept intact - that the stimulus gets randomly presented while the user focuses on one of the possible selections! You can find our updated P300 BCI tool here: Unity P300 Tool.

Making Use of Mental Commands

Mental commands refer to a simplistic BCI control scheme where a person simply imagines some vivid imagery, like moving an object in a direction. These are used by some commercial BCI headsets, and are meant to help distinguish intended actions by the user by taking a 'snapshot' of what the brain activity looks like during any given command. Typically, the first mental command that needs to be trained/identified is a rest or 'neutral' command. This allows the computer to know what the 'non-action' state of the brain looks like, and gives a ground baseline for the computer to compare against. Then when you train an active mental command, say imagining pushing an object away from you, the BCI headset can now distinguish between the two snapshots to decide if you were actively imagining your mental command or not (neutral).

Typically, as more mental commands are trained it becomes more difficult for the BCI systems to reliably separte each command. We therefore are recommending for game developers to think about how to build games that play well with a limited number of actions if they are building for mental command BCIs. Previously this meant playing games using just a single (or maybe two) buttons!

A big consideration when thinking about using mental commands in a BCI game is that, at the moment, these commands are triggered by surpassing a threshold. Functionally this means that the BCI command is registered more like a mouse click than a button press - there is not a 'hold' option for triggering the mental command from the BCI side. So when thinking about game design with mental commands you either want to use the mental command to activate instant actions (click an option, move a set distance etc.) or design your game to require successive inputs that build up in order to trigger the action of interest. An example of how this could be used is similar to the typical "repeatedly press A/X/F" to lift a heavy object or open a chest, like in God of War or The Last of Us. Instead of resetting the user's count to 0 on a failed press perhaps create a 'bank' of presses that slowly gains until the action initiates. In this manner the mental command BCI rapid triggering can be simulated as a longer 'hold' action if that's of interest. This could also work with the neutral/rest command!

Another thing to consider when using mental commands is that there is a built in delay between when the mental command is given and when the BCI is able to identify and send the command. This delay is usually somewhere between 150 - 500 milliseconds depending on the system, computational resources, processing etc. Therefore its important to consider how to naturally allow grace for this delay to not affect the feel of the game. Using the above banking method or time-warping mechanics represent exmaples of how to overcome this natural delay barrier for players.

A New Challenge: SSVEP

SSVEP stands for steady-state visual evoked potential. This refers to a pattern in your brain activity that is elicited when you attend to a light or stimulus flickering at a specific frequency. If look at (attend) to a flashing stimulus for a sustained period of time, your brain activity will sync up and oscillate at the same frequency as that stimulus. In this way, a BCI can tell which flashing stimulus you are attending to be searching for what frequency is dominant in your brain activity.

We have a "work in progress" version of an SSVEP tool for python, making use of the pygame game engine. This tool allows you to create and place SSVEP flashing stimuli throughout your game, and generates the flashing pattern and timing for you. We encourage those up for a challenge to try out this tool and incorporate SSVEP in a python-based game!