2.1 Principle of the EBCM
We witnessed impressive advancements in the programmable metasurfaces [25, 32, 33]. On the other hand, BCI has been captivated a popular notion of enabling brain signals to directly deliver the physical actions. With non-invasive electrodes [5, 6], the P300 potentials can be identified from the brain signals under a certain time sequence of visual stimuli. In this work, we intuitively combine the P300 coding characteristics with the programable information metasurface, and experimentally demonstrate a robust P300-based EBCM to showcase the robust control of EM wireless information (e.g., synthesis, manipulation, and encoding/decoding) with only inputs from the brain signals and the designed functions, as shown in Fig. 1a.
A displayer is placed in front of the operator to show the graphical user interface (GUI), and there is a virtual button matrix, as exhibited in Fig. 1b. Different buttons correspond to different coding-pattern operations of EBCM. Each trial corresponds to one command sending, and the buttons start to flash successively in a random order for about 5 rounds, each of which contains one flash for each button. The flash sequence is randomly generated before each trial. Figure 1c presents the flashing sequence of 40 buttons (40 rows) in the beam-scanning scheme, where the yellow squares indicate the starting point of the stimulus. The vertical axis represents the number of buttons from 1 to 40, while the horizontal axis represents the 160 sequential stimulus flashes marked by yellow bars in Fig. 1c. Each marked block in Fig. 1c represents a duration of 30 ms and each flash lasts for 100 ms, which will span slightly more than 3 blocks. In Fig. 1c, we further show a zoom-in view of three flashes to illustrate the stimulus time sequences of different buttons more clearly.
The operator’s attention is focused on the button corresponding to the command that he/she would like to issue (i.e., the target). When the target flashes, due to the oddball effect, a positive potential may be detected from EEG after approximately 300 ms, dubbed P300 potential [34]. Such a P300-based BCI has been experimentally proved effective for the brain to directly control the external devices [5, 35]. To present the brain signals for two stimulus types (target and nontarget), we exhibit the measured EEG signals in Fig. 1d, where the red and blue curves correspond to the signals for target and nontarget stimulus, respectively. In each subfigure of Fig. 1d, the signals corresponding to both target and nontarget buttons are averaged across multiple flashes. For each button flash in a trial, a segment of EEG signal from 0 to 600 ms after the flash onset is extracted and corrected with a baseline extracted from 200 ms before the flash onset. Next, we exploit this sequential stimulus to directly control the field programmable gate array (FPGA) to execute the related instructions in the operators’ mind, and successfully realize the paradigm shift from the brain signal to EM scattering manipulations. As a proof-of-concept demonstration of EBCM, we present wireless text communications between two operators by the minds. By designing specific rules between EEG and EM signals, the transmitted text in mind is accurately received and decoded in experiments. To exhibit more functions realized by the same EBCM, we also demonstrate three applications, which include (1) EM-beam scanning; (2) multiple scattering-beam switching; and (3) metasurface pattern encoding. More details of the EBCM platform are provided in Additional file 1: Supplementary Note S1 [36].
2.2 Wireless text communication by mind based on EBCM
To fully exhibit the application of EBCM, we design and experimentally demonstrate wireless text communications by mind, as illustrated in Fig. 2a. A text GUI is provided for the BCI operator (see Additional file 1: Supplementary Note S2 [36]), in which the visual button is encoded directly as a specific coding sequence composed of ‘0’ and ‘1’, related to two coding patterns. Here we employ a single-beam pattern with high gain and a scattering-reduction pattern for amplitude discrimination, respectively corresponding to ‘1’ (high amplitude) and ‘0’ (low amplitude). As a proof of prototype, we show the text wireless transmissions by mind from one operator to another within our EBCM communication system. The operator A, as the text transmitter, sends the letters by visually staring at the character button on the GUI of EBCM. When the target letter is decoded from the EEG signals, a coding sequence based on the ASCII codes is implemented on FPGA to switch the time-varying patterns.
In the encoding process, since the buttons representing the related text characters have the corresponding ASCII codes, the selected button is directly translated to the binary ASCII codes with the frame header “11111111110000”, as illustrated in Fig. 2c. Then according to the final code, the metasurface reflects the high or low intensity to the space. In the decoding process, we firstly collect the spatial EM energy using the receiving channel, including a microstrip antenna embedded beside the metasurface, as shown in Fig. 2e, as well as a low-noise amplifier (LNA) and a high-speed analog-to-digital convertor (ADC) controlled by FPGA. The collected data stream is a series frame set, which represents the sampled intensity at the acquisition rate of 10 MHz. We use the decoding algorithm to locate the position of the frame header to determine the starting point of the data frame, as illustrated in Fig. 2f. Then the sampled data are transferred into the binary ASCII codes, and we display the text in the GUI (provided in Additional file 1: Supplementary Note S2).
Four text sequences are sent and received successfully by mind using the EBCM platform, including “HELLO WORLD”, “HI, SEU”, “HI, SCUT”, and “BCI METASURFACE”. A recorded video is provided in Additional file 2: Supplementary Movie S1. The average inputting time of each character is about 5 s using the P300-based BCI by a skillful BCI operator. Since the programmable metasurface can achieve the “0/1” code transmitting speed of at least 1 Mbps, the maximum character transmitting speed for the metasurface is about 5 × 104 characters per second (20 bits each sequence). Hence the final text-transmitting speed is about 12 characters per minute. It is worth mentioning that the P300-based BCIs yield great accuracies and robustness among various noninvasive BCIs [5, 6]. It is possible to improve the text input speed by applying some quick-spelling paradigms [7, 37].
2.3 Experiment implementation and results of wireless communications
The detailed illustration of the communication system is depicted in Fig. 3a, in which the transmitting and receiving parts are marked in orange and green. In the transmitting part, the EEG signals are firstly detected and processed with the BCI devices and translated into the corresponding control signals of FPGA. The control signals follow the signal coding principle of the corresponding interface shown in Fig. 2b–d. The FPGA executes the coding pattern arrangements and drives the PIN diodes to the desired states. In the receiving part, the microstrip antenna (MSA) beside the metasurface obtains the EM signals from the transmitter and sends it into the LNA and detector. The detector samples the analog amplitude, which is further converted to the digital codes for FPGA. The presented process is unidirectional but the communication system of EBCM is bidirectional since the transmitting and receiving fronts are respectively the metasurface and MSA, as illustrated in Fig. 3b, which enables simultaneous transmission and reception of EBCMs by the same operator. When one metasurface sends a signal, the MSA next to the other metasurface receives and demodulates it. A similar process can also be performed in reverse simultaneously.
The experimental scenario is exhibited in Fig. 3c, in which Operator A executes the text transmitting task and Operator B receives and reads the text. The distance between transmitting and receiving metasurfaces is about 1.3 m, in which the transmitting metasurface is excited by a broadband antenna with a distance of 0.3 m, and the receiver is an antenna integrated near the receiving metasurface, which is connected to an LNA and a high-speed detector, as well as an ADC, controlled by another FPGA. The received and demodulated letters and text are finally displayed on the designed GUI. The experiment process of the wireless text communications is recorded in Additional file 2: Movie S1.
Figure 4a illustrates the processed EEG responses of the channel OZ to two stimulus types (target and nontarget) when the subject is spelling the word “HELLO”. In each subfigure corresponding to the spelling of one character, for each of the two stimulus types, the event-related potential (ERP) waveforms are extracted by the time-locked average of the EEG signals across all flashes of the target or one of the nontarget buttons in a trial. Compared to the nontarget data, high amplitude is clearly observed at about 300 ms after the stimulus. According to the EEG signals, EBCM produces the amplitude-modulated EM signals of different letters using the ASCII code.
To fully demonstrate the wireless communications, we further provide 12 segments of the measured EM signals, including the letters ‘H, E, L, O, B, C, I, S, M, E, T, A,’ as showed in Fig. 4b–d, where the high and low amplitudes respectively mean ‘1’ and ‘0’ in the ASCII code. The presented data are collected by an EM detector and then normalized. Each detection generates an amplitude pulse and numerous amplitude pulses to compose the presented data, where the high and low amplitudes respectively mean the codes 1 and 0. The ASCII codes of these letters are clearly observed according to Fig. 4b–d. Please note that the EEG signal waveform of the same test subject at different times and different states are not the same even when test the same letter. The EEG detection based on the P300 is aimed at the test subject having a clear peak at 300 ms after being stimulated, to prove that the corresponding key is triggered.
2.4 More functions of wavefront syntheses by EBCM
To further demonstrate more functions of wavefront syntheses by the brain signals, we design three typical applications including the visual-beam scanning, multiple EM modulations, and coding pattern input. Here, we establish a demonstration prototype of EBCM, in which the metasurface is replaced by the light emitting diode (LED) version instead of PIN diode, as shown in Fig. 5a. Since the coding patterns on the metasurface directly determine the EM functions, we embed LEDs to intuitively visualize the pattern control in the EBCM verification system (see Additional file 1: Supplementary Note S3). In the visual-beam-scanning scheme, we wish that the operator can direct the EM beam scanning accordingly using EBCM by simply staring at various directions in the sky, as depicted in Fig. 5b. We design a beam-scanning GUI with a sky background in Additional file 1: Fig. S7, and the detailed descriptions are provided in Additional file 1: Supplementary Note S4. We record a video of the visually-controlled beam scanning towards multiple directions, as presented in Additional file 3: Supplementary Movie S2. We envision that this scheme can be further integrated with the augmented reality (AR) technique and finds more applications in adaptive mind-text wireless communications and intelligent radar detections.
To realize multiple EM modulations by EBCM, we design a specific interface to show that the operator can directly drive the EBCM for diverse EM functions (see Additional file 1: Supplementary Note S4), including beam deflections, orbital-angular momentum (OAM) beam generations, and radar cross section (RCS) controls, as shown in Fig. 5c. The simulated results of these functions are presented in Fig. 5d–m. For example, we illustrate the result of vertically-reflected single beam generated by the uniform phase pattern in Fig. 5d, and the results of three deflecting angles in Fig. 5e–g, where the simulated data clearly indicate the scattering directions of 15°, 30°, and 45°, suggesting great consistency with the measured directions marked in the red cross. More results are given in Additional file 1: Supplementary Note S5. In the RCS level control, four RCS buttons from ‘01’ to ‘04’ will generate the scattered fields presented in Fig. 5i–l, showing the scattering levels of − 15 dB, − 12 dB, − 9 dB, and − 6 dB, respectively. For the OAM-beam generation, we exhibit two scattered fields of two OAM modes (+1 and +2) in Fig. 5h and m, in which the central amplitude null is clearly observed.
For experimental verification of the multiple EM modulations by EBCM, we recorded the controlling process of three representative EM functions, as shown in Additional file 4: Supplementary Movie S3. The related coding patterns of 10 deflecting directions are given in Additional file 1: Supplementary Note S6. The measured results presented in Additional file 1: Supplementary Note S7 [36] show good agreements with the simulations in visual-beam scanning and multiple EM modulations. The related 30-channel EEG signals are also listed in Additional file 1: Supplementary Note S8 [36], where the P300 waves are successfully captured from the EEG signals when the target button flashes. In addition, we perform the metasurface pattern coding scheme using EBCM. We show that the 2-bit phase coding patterns of the metasurface can be directly input from the brain signals to generate various EM functions. More details are illustrated in Additional file 1: Supplementary Note S9 and Additional file 5: Movie S4.