Integral imaging near-eye 3D display using a nanoimprint metalens array

Integral imaging (II) display, one of the most critical true‑3D display technologies, has received increasing research recently. Significantly, an achromatic metalens array has realized a broadband metalens‑array‑based II (meta‑II). However, the past micro‑scale metalens arrays were incompatible with commercial micro‑displays; furthermore, the elemental image array (EIA) rendering is always slow. The two hinders in device and algorithm prevent meta‑II from being used for practical video‑rate near‑eye displays (NEDs). This research demonstrates a meta‑II NED combin‑ ing a commercial micro‑display and a metalens array. The large‑area nanoimprint technology fabricates the metalens array, and a novel real‑time rendering algorithm is proposed to generate the EIA. The hardware and software efforts solve the bottlenecks of video‑rate meta‑II displays. We also build a see‑through prototype based on our meta‑II NED, demonstrating the feasibility of augmented reality. Our work explores the potential of video‑rate meta‑II displays, which we expect can be valuable for future virtual and augmented reality.


Introduction
True-3D display technologies [1], including integral imaging (II) display [2][3][4], holographic display [5][6][7], volumetric 3D display [8], super multi-view display [9][10][11][12], etc., can satisfy the three-dimensional visual experience of human to the real scene.In particular, the II display is considered one of the most promising true-3D display technologies with the advantages of compact volume, full parallax, quasi-continuous viewpoint, and convenient full-color display.Compared with the current 3D display technologies based on binocular parallax [13][14][15], II display is not faced with the vergence-accommodation conflict (VAC), thus greatly alleviating visual fatigue.In the past few years, numerous efforts have been put into the performance improvement for both glasses-free II displays [16][17][18][19][20][21][22] and II near-eye displays (NEDs) [23][24][25][26][27][28][29][30].While most II displays currently adopt a microlens array (MLA) to modulate light, an II display using a novel nano component, metalens array, was proposed [22] as the "meta-II display." The meta-II display can take advantage of the metalens array's flat nature and tremendous flexibility in light field manipulation, thus opening opportunities for improving the II display's resolution, field of view, depth of field, etc.In contrast, the problems are challenging for a regular MLA-based II display.Despite the promising perspective of meta-II displays, video-rate meta-II displays for practical use have been rarely reported for two reasons.First, previously, only micron-scale metalens arrays were available due to the fabrication difficulty of large-area metalens arrays.Correspondingly, commercial micro-displays at a millimeter scale were unavailable, but a fine mask had to be used as a static picture generation unit [22].Secondly, real-time image rendering is always challenging for II displays because of the high computational complexity of calculating the elemental image array (EIA).Therefore, the challenges in nano-device and algorithms both hinder practical meta-II displays.
We notice that deep ultraviolet lithography technology [31][32][33][34] has been used to realize low-cost, large-diameter, and large-area metalens in recent years, and nanoimprint technologies [35][36][37][38][39] have also been proposed to fast reproduce metalens samples.Such advances in metalens mass fabrication have significantly boosted the interest in metalens-based virtual reality (VR) and augmented reality (AR) display [40][41][42][43][44][45] and potentially enabled largearea meta-II.On the algorithm side, almost all existing EIA generation methods are based on viewpoints, which repeatedly perform geometric projections to induce high computational complexity.Though previous studies have proposed strategies such as parallel computing [46][47][48] and sparse viewpoints [49][50][51] to accelerate the rendering, the hardware complexity is elevated, or the rendering accuracy is sacrificed.That is to say, current acceleration strategies have to encounter a tradeoff between computational complexity, hardware complexity, and rendering accuracy.In particular, advanced computing hardware is unacceptable in wearable near-eye devices considering its power consumption and portability.Therefore, a new rendering method breaking the tradeoff is needed to achieve desired video-rate meta-II displays besides largearea metalens devices.
Here, we demonstrate a novel meta-II NED by combining a commercial micro-display and a large-area metalens array, as shown in Fig. 1a.Our metalens array is designed with the high-refractive-index nanoimprint glue and experimentally fabricated by the large-area nanoimprint technology.Next, a new rendering method is proposed to fast generate the EIA by exploiting the invariant voxel-pixel mapping in an II display.As a result, true-3D display is verified through monocular focus cues and motion parallaxes in both simulation and experiment.An average frame rate as high as 67 FPS is achieved to support real-time rendering.Moreover, based on the meta-II display module, we build a see-through system by merging 3D images with surrounding objects, showing the broader potential of the meta-II display for AR.

Optical architecture of the meta-II NED
Figure 1a illustrates the optical architecture of our meta-II NED.Three main components are shown in the top right corner: a micro-display, a 3D printing holder, and a metalens array.The micro-display is crucial to provide high-resolution pictures.Here, we adopt a 0.39-inch Si-OLED micro-display (BOE Technology Group Co., Ltd., B039FH8A0) with a high pixel density of 5644 PPI (i.e., pixel pitch: 4.6 μm).The following metalens array design and EIA rendering algorithm comply with the microdisplay's specifications.In order to assemble the microdisplay and the metalens array, we specially design a Fig. 1 Diagrammatic drawing of meta-II NED. a The 3D AR effect of the meta-II NED with the major components of the metalens array and the micro-display.The light from the micro-display enters the human eye through the metalens array and a beam splitter's reflection, while the ambient light can also be seen through the beam splitter.The virtual 3D images (number "3" and letter "D") are reconstructed to coincide with the chess pieces ("Rook" and "Pawn"), respectively.b The photograph of the assembled meta-II micro-display panel.The micro-display and the metalens array are well aligned and assembled on both sides of the 3D-printed holder 3D-printed holder.The micro-display and the metalens array can be well aligned and assembled on both sides of the holder so as to form an easy-to-use NED.To implement a see-through system, we use a beam splitter to merge 3D virtual images with the real scene, as the AR system Fig. 1a shows.Figure 1b shows our meta-II NED module, where the metalens array is aligned with the micro-display.The entire NED module in Fig. 1b weighs only 7.23 g for a seamless user experience.

Nanoimprint metalens array
In terms of the metalens array fabrication, nanoimprint technology is used to develop our metalens array.Compared to electron beam lithography, nanoimprint technology can quickly replicate many metalens array samples, especially large-area samples.Since the refractive index of nanoimprint adhesive is generally lower than 2.0, a high nanopillar is needed to realize the 2π phase interval coverage, leading to a large depth-diameter ratio of the nanopillar and increasing the difficulty of stamping, as discussed in the attached Additional file 1: Sect.S1.Here, balancing the difficulty of nanoimprint fabrication and the realization of phase interval coverage, we select the imprinting adhesive with a refractive index of 1.9 as the metalens material, and the thickness of the nanopillars is set as 500 nm.After an optimization design, the lattice constant of 416 nm is selected with the rectangle lattice arrangement for the nanopillar grating.In the nanoimprint process, a residual adhesive layer will inevitably remain on the substrate.However, it can be seen from the discussion in Additional file 1: Sect.S1 that the thickness of the residual adhesive layer little influences the overall phase, so it is not taken into consideration here.As a demonstration in this paper, we choose the appropriate size of the metalens array as 1840 μm×1840 μm with 4 × 4 metalenses; that is, the single metalens aperture is 460 μm×460 μm.Considering that the micro-display's pixel pitch is 4.6 μm, one single metalens corresponds to 100 × 100 pixels in the micro-display.Hence, the number of viewpoints of our meta-II NED is 100 × 100, and the effective pixel resolution of the micro-display is 400 × 400.Thanks to the rapid replication advantage of nanoimprint technology, four identical metalens array samples shown in Fig. 2a are fabricated.The detailed nanoimprint fabrication processes are described in the Additional file 1: Sects.S2 and S3.
Figure 2b shows the metalens array's microscopic image, where no obvious defects can be seen, and the entire sample is intact.Figure 2c, d show the top view and side view of the local electron microscope image of the metalens array.In summary, high-quality nanoimprint metalens arrays have been achieved.
After fabricating the metalens array samples, the optical field scanning experiments were carried out.The experimental optical path is shown in Additional file 1: Fig. S4, and the corresponding results are collected in Fig. 2e-h.Figure 2e shows the normalized light intensity distributions of all 16 metalenses in the array in the yz plane at 547 nm.It can be seen that all metalenses converge the light approximately on the same focal plane, and all have only one focal point with a specific range of depth of focus (DOF).The average focal length of the metalens array is 5.8 mm, and the average DOF is 392 μm at 547 nm.Furthermore, the distributions of light intensity of all metalenses are similar in the yz plane, indicating a good uniformity of the metalens array.Hence, we can extract the optical field distributions of a single metalens to represent the optical field characteristics of the entire array.Figure 2f-h respectively show the focal plane, the cross profile along the x direction, and the corresponding modulation transfer function (MTF) of a single metalens at 547 nm.In Fig. 2g, the black scatter points represent the experimental data, while the green curve is the data curve after the Airy fitting.Thus, the focal spot's full width at half maximum (FWHM) is 6.6 μm.In the MTF curves in Fig. 2h, the black dotted line is the diffractionlimited MTF, and the solid green line is the calculated MTF.Since the aperture of all metalenses is square, the diagonal aperture replaces the side length in calculating the diffraction limit.As seen, the focusing performance of the metalens is nearly diffraction-limited.

Real-time EIA rendering method
After obtaining the large-scale metalens array, the conventional slow EIA rendering method becomes the pivotal factor that impedes the desired video-rate meta-II display.The conventional rendering imitates II photography [2].That is, an EIA is generated by capturing elemental images virtually with a camera array, which consumes much time to project all viewpoints to elemental images.Here, we propose a new rendering method to satisfy realtime without sacrificing computational complexity and accuracy.As shown in the middle of Fig. 3a, a voxel in an II display is generated by optically integrating homogeneous pixels.And these homogeneous pixels are mapped to the voxel through raytracing.Note that voxels on a depth plane are invariantly determined by system parameters (lens pitch, object distance, etc.) and statically mapped to pixels.Hence, we can exploit the static mapping prior to the EIA rendering, acquire voxels on specified depth planes, and save the voxel-pixel mapping as a look-up table (LUT), as shown in the left of Fig. 3a.Each cell in the LUT records the coordinates of all homogeneous pixels that form a voxel.In this manner, merely look-up operations are executed during the EIA rendering, bringing an ultra-fast EIA generation.The processes of our method can be elaborated as follows.
The preprocessing contains two steps: (a) Voxel-pixel mapping acquisition.Use a lens model (or the pinhole model) to trace the chief rays of all voxels for each metalens, then obtain the voxel- in a LUT.A 3D scene is formed by multiple depth planes whose number affects the LUT size.Usually, five to ten depth planes are ample, considering the depth resolution of human eyes [52].At this point, each LUT's size is approximately several megabytes, entirely affordable for modern electronic devices.The modest memory demand indicates a reasonable LUT size.
Secondly, with the LUT, our EIA rendering includes two procedures: (c) Image resampling.Resample and rasterize an input 3D scene to voxels determined by the system parameters, as shown in the right of Fig. 3a.Black voxels can be ignored, indicating the method is more appealing for black-ground pictures (e.g., AR systems).The resampling is merely performed once to be a negligible overhead of the whole rendering.(d) Look-up operation.Assign rasterized data of each voxel to its corresponding homogeneous pixels according to the LUT, and output the final EIA for the II display, as shown in the left top of Fig. 3a.Because geometric projections in conventional viewpoint-based methods are replaced with ultrafast look-up operations, the rendering speed can be accelerated for several orders of magnitude.
Using the Meta-II system with the 4-by-4 metalens array, we continuously executed the EIA rendering for 30 pictures.Our environment and rendering performance are as follows.The result shows that our method's average frame rate exceeds the requirement of 60 FPS for video-level performance.More importantly, we use quite an entry-level personal computer but do not pay a cost for more complicated hardware.
Average runtime of 30 pictures: 15 ms.

Verification of true-3D display
We conducted a meta-II display in the real mode (real images generated) to verify motion parallaxes and monocular focus cues, which are essential characteristics of true-3D display compared with the binocular parallax-based conventional 3D display.Consider a typical situation: the number "3" is located on the central depth plane (36 mm in this system), and the letter "D" is 10 mm behind it, as shown in Additional file 1: Fig. S7a.The EIA is calculated for the scene using the real-time generation method above, then inputted into the micro-display in simulation (LightTools) and experiment, where the distance between the metalens array and the micro-display is 5 mm.
First, Fig. 3b-d show simulated images at the viewing angles of − 1 º, 0 º, and 1 º, respectively, while fixing the receiver's accommodation at the depth plane of the number "3". Figure 3e-g show the corresponding experimental results, which are consistent with the simulation.As a result, when the viewing angle is 0°, as Fig. 3c, f show, the number "3" is in the middle of "D." After rotating the viewing angle to − 1º in Fig. 3b, e, "3" is close to the edge of "D"; in contrast, it approaches the arc of "D" under the viewing angle of 1 º, as Fig. 3d, g show.The evident change in relative positions between the objects at different depths demonstrates effective motion parallaxes.Next, the monocular focus cus is considered.As the receiver in the simulation and the camera in the experiment both focus on the depth plane of "3, " the number "3" is sharp, while the letter "D" is relatively blurred.This result reveals that we create two optical depth planes, through which the number "3" is in focus and the letter "D" is out of focus.In this manner, the monocular focus cue is also proved.More verification for the motion parallax and focus cue is provided in the Additional file 1, e.g., the experimental blue and red images at different viewing angles in Additional file 1: Fig. S8 and the corresponding videos with green, blue, and red images in Additional file 1: Movies S1-S3, respectively.

See-through AR prototype
Our nanoimprint-based large-area metalens array and the real-time EIA rendering method have enabled a practical meta-II display.This section adopts it as the engine to implement a see-through AR prototype with a true-3D feature.By reconstructing virtual 3D images far behind the micro-display, the AR prototype works in the virtual mode so that a beam splitter can merge the real world and the virtual image at adjustable depths for users, as shown in Additional file 1: Fig. S7b.Note that the EIA generation method is the same for the real and virtual modes by setting desired image depths.In this prototype, the distance between the metalens array and the microdisplay is 5.74 mm for green images.The reconstructed depth planes of the number "3" and the letter "D" are 80 and 300 mm from the metalens array, respectively.We use a cellphone camera to imitate human eyes and capture images.The distance between the eye and the metalens array is about 20 mm.The light from the microdisplay enters the camera through the metalens array and the beam splitter's reflection, while the ambient light transmits through the beam splitter to enter the camera, as shown in Fig. 1a.Thus, the camera can capture merged virtual 3D images and the surroundings to verify the AR effect.
Figure 4 shows AR effects with different colors, captured in the experiment in a dark ambience.In Fig. 4a, when the camera focuses on the chess piece "Rook" in the foreground, the number "3" and the chess piece "Rook" are both clear, showing the depths of both are the same.Meantime, the letter "D" is blurry, as expected, because it is reconstructed at a farther depth.When the camera focuses on the chess piece "Pawn" in the rear, as Fig. 4b shows, the letter "D" becomes sharp, while the number "3" in the middle is unrecognizable.That is to say, the reconstructed depth of the letter "D" coincides with the chess piece "Pawn." Compared with conventional AR systems, e.g., those based on total-internal-reflection waveguides, our meta-II can achieve true-3D through the monocular focus cue verified above, as shown in Fig. 4a and b.With the help of the monocular focus cue, the VAC can be alleviated by matching it with binocular parallax-induced convergence.In Fig. 4c-f, the reconstructed blue and red 3D images are similar, further verifying the AR capacity of our meta-II.The related videos are provided in Additional file 1: Movies S4-S6.
Furthermore, we adopt rings and dice patterns to examine the resolution, as shown in Fig. 5. First, the similar focus cue is verified.When the camera focuses on the black chess in the middle, the rings and the dice are clear, while the real objects in the foreground and background are blurry.More importantly, eight bright/dark donut pairs can be distinguished in the rings in Fig. 5a besides the central bright spot.Each donut pair represents two voxels, so the rings contain 33 individual voxels.The outermost diameter of the rings is 14.2 mm, and the distance between the rings and the metalens array is 355 mm for a field of view of 2.29°.Thus, the angular resolution of our meta-II AR system is 33/2.29°=14.4PPD (pixels per degree), equivalent to the Oculus Quest.The clear image of the dice in Fig. 5b also demonstrates the ability to display complex patterns.In general, the critical parameters of our mera-II AR system can be listed as follow, FOV: 2.29°; depth of field: 100-375 mm; angular resolution: 14.4 PPD.
Fig. 4 The AR prototype based on the meta-II display.Red, green, and blue images (real images) are presented on a dark background.a The image is captured by focusing on the green number "3" and the chess piece "Rook" with details enlarged in the red frame at the right.Note that the number "3" and the chess piece "Rook" are clear, while the letter "D" and the chess piece "Pawn" behind are blurry.b The image is captured by focusing on the green letter "D" and the chess piece "Pawn." At this point, the letter "D" and the chess piece "Pawn" become clear, while the chess piece "Rook" and the number "3" in the front become blurry.Similar results of (c, d) blue and (e, f) red images

Discussion and conclusion
Combining a metalens array, a commercial micro-display, and a real-time EIA rendering method, a novel meta-II NED is archived.High-precision large-area nanoimprint technology is used to fabricate the metalens array with high-refractive-index material of nanoimprint glue.We also propose a new voxel-based EIA rendering method to support real-time by upgrading the conventional viewpoint-based method.As a result, a true-3D display capacity is verified.By merging 3D images with surrounding objects, we also implement an AR prototype with true-3D display.In addition to the design of the large F-number metalens array described above, we also show the design of the small F-number metalens array and its corresponding integral imaging simulation in Sect.S6 of the Additional file 1.Note that the design flexibility of metalens arrays is highly valuable for next-generation near-eye displays regarding several long-standing issues of integral imaging.For example, extended depth of field is vital for true-3D NEDs to present images from the person space to the vista space, whereas the conventional microlens array induces a very limited depth of field.In contrast, a metalens array can be easily designed as a polarization multiplexing element with varying focal lengths to different polarization directions, allowing depth of field extension.Another issue is the narrow FOV.Free-form surface optics, an effective aberration correction scheme to increase the FOV, is difficult for traditional microlens arrays to achieve.However, our meta-II provides a promising solution for further study: freeform phase profiles that precisely compensate for the field-dependent aberration can be recorded in a slim metalens array.More importantly, both extened-depth-of-field meta-II and FOV-expanded meta-II suffer from no cost in computational complexity and system volume compared with the meta-II proposed in the current manuscript.Finally, we believe the design flexibility of metalens arrays, the lowcost nanoimprint fabrication with the feasibility of mass production and our real-time rendering method can promote video-rate meta-II NED for future VR and AR.
the wavelength of 547 nm.

Fig. 2 Fig. 2 (
Fig. 2 The nanoimprint metalens arrays and the measurement results.a Optical photo of our nanoimprint metalens arrays.b The microscope photo.c Top-view SEM image of a portion of the metalens array.d Side-view SEM image with high magnification.e Normalized measured intensity distributions in the y-z plane of all 16 metalenses in the array.f The normalized measured focal plane intensity distribution of a single metalens.g The x-direction cross sections of the measured intensity profiles in (f).The solid green curve denotes the Airy fit of measured data (black square point).The green text gives the full widths at half-maximum (FWHM) of the fitting measured data.h Calculated MTF (solid green curve) of a single metalens.The dashed line represents the corresponding diffraction limit (See figure on next page.)

Fig. 3
Fig. 3 Sketch map of the real-time EIA rendering method and the verification of true-3D display.a The proposed EIA rendering method mainly includes acquiring the voxel-pixel mapping, LUT construction, image resampling, and look-up operations.b-d Simulated 3D images with motion parallax and focus cue in the meta-II system.e-g The corresponding experimental results.The images at different view angles have evident differences, demonstrating an effective motion parallax

Fig. S2 .
Optical and SEM images of the silicon master mold of metalens array.Fig. S3.Schematic illustration of nanoimprint fabrication of metalens array.Fig. S4.Measurement setup for characterizing the performances of metalens array.Fig. S5.Measured intensity distributions in the yz plane of a single metalens.Fig. S6.Measured intensity distributions in the focal planes of a single metalens.Fig. S7.Optical path diagrams for the 3D II display.Fig. S8.Experimental results of the 3D parallax effect.Fig. S9.The defocusing AR effects of the meta-II NED system.Fig. S10.Simulated results of a small F-number metalens array.Fig. S11.The eyebox size simulations of different F-number metalens arrays.Movies S1.The results of 3D parallax effect under the green light.Movies S2.The results of 3D parallax effect under the blue light.Movies S3.The results of 3D parallax effect under the red light.Movies S4.The results of AR effect under the green light.Movies S5.The results of AR effect under the blue light.Movies S6.The results of AR effect under the red light.