Computational coherent Raman scattering imaging: breaking physical barriers by fusion of advanced instrumentation and data science

Abstract

Coherent Raman scattering (CRS) microscopy is a chemical imaging modality that provides contrast based on intrinsic biomolecular vibrations. To date, endeavors on instrumentation have advanced CRS into a powerful analytical tool for studies of cell functions and in situ clinical diagnosis. Nevertheless, the small cross-section of Raman scattering sets up a physical boundary for the design space of a CRS system, which trades off speed, signal fidelity and spectral bandwidth. The synergistic combination of instrumentation and computational approaches offers a way to break the trade-off. In this review, we first introduce coherent Raman scattering and recent instrumentation developments, then discuss current computational CRS imaging methods, including compressive micro-spectroscopy, computational volumetric imaging, as well as machine learning algorithms that improve system performance and decipher chemical information. We foresee a constant permeation of computational concepts and algorithms to push the capability boundary of CRS microscopy.

1 Introduction

1.1 Brief history of CRS

The Raman scattering, first reported by Dr. C.V. Raman in 1928 [1], is an inelastic scattering process that exchanges energies between photons and molecules to carry information on molecular vibrations. Since the development of lasers in the 1960s, Raman micro-spectroscopy has become an indispensable analytical tool in biology and medical surgery mainly due to its two "frees": label-free and water background-free. These benefits enable us to study living samples without endogenous perturbation. In addition, Raman peaks have a much narrower spectral bandwidth than fluorescent dyes' emission spectrum, which enables the simultaneous study of various metabolic species in the same environment.

Despite its significant advantages, one fundamental drawback of Raman scattering lies in its severely limited cross-sections. A typical Raman cross-section is on the level of 10^–30 cm [2] per molecule, which results in a very long signal integration time from seconds to minutes per focal spot. Such limited speed makes it impractical to perform pixel-by-pixel imaging of dynamic systems. To break the fundamental cross-section limits, nonlinear optical process has been introduced to coherently enhance the Raman signals [2]. With two synchronized ultrafast lasers, coherent Raman signals arise in the forms of coherent anti-Stokes Raman scattering (CARS) and stimulated Raman scattering (SRS). In CRS, two laser fields, namely pump (ω_p) and Stokes (ω_S), synchronously interact with the target molecule. When the beating frequency (ω_p − ω_S) matches the Raman vibrational mode ω_v, a coherently amplified energy transfer process occurs, which annihilates the pump photon (stimulated Raman loss, SRL), converts it to the Stokes beam (stimulated Raman gain, SRG), and generates photons at a new frequency at ω_p − ω_S + ω_p (CARS) (Fig. 1a). The SRL and SRG are together termed SRS. More detailed descriptions can be found in recently published CRS books [3, 4].

Fig. 1

Overview of CRS microscopy. a Energy transfer diagram of CRS. b A typical CRS setup. OM optical modulator, D dichroic, SU scanning unit, O objective, C condenser, F filter, PD photodiode, PMT photo-multiplier tube, Lock-in Lock-in amplifier. Hyperspectral CRS by c frequency tuning or pulse shaping, d multiplex, and e spectral focusing

CARS and SRS were first observed back in the 1960s [5, 6], yet the burgeon of CRS for biomedical imaging started in the 2000s with the development of ultrafast lasers. The first CARS microscope was reported in 1982 through a non-collinear geometry [7]. In 1999, Xie and coworkers introduced synchronized high-repetition-rate ultrafast laser pairs in a co-propagating and tight focusing geometry [8]. Since CARS signal arises in a new frequency, highly sensitive photodetectors such as photomultiplier tube (PMT) and single-photon avalanche photodiode (SPAD) are used. One major issue with CARS is the non-resonant background, which distorts the CARS spectrum and overwhelms the CARS peak under low concentrations, limiting the chemical specificity and sensitivity. One route to address the non-resonant background is through analytical post-processing. Vartiainen et. al. pioneered phase retrieval methods based on maximum entropy model to fit and extract Raman line-shapes from raw CARS spectra [9, 10]. Cicerone group developed time-domain Kramers–Kronig (TDKK) [11] for phase retrieval and proved its equivalence with maximum entropy [12]. Experimentally, the non-resonant background can be circumvented by SRS, which directly probes the imaginary part of the Raman susceptibility that matches the spontaneous Raman line shapes. In 2007, a broadband SRS microscope with a 1 kHz laser was developed [13]. Next year, Freudiger et al. reported a high-speed high-sensitivity SRS system based on a high-repetition-rate narrowband laser [14]. In this pioneering work, the SRS signal was detected by modulating one beam and measuring the periodic intensity fluctuations on the other beam through a lock-in amplifier. A high-saturation silicon photodiode (PD) is preferable since the measured SRS signal is a weak modulation over an intense laser field. Unlike CARS, SRS has spectral line shapes consistent with spontaneous Raman counterparts, and its intensity scales linearly with the molecular concentration, which facilitates the identification and quantification of target molecules. Yet, SRS is not completely background-free, as transient absorption, cross-phase modulation, photothermal lensing and two-photon absorption can contribute to the overall modulated signals [15], complicating the analysis of SRS at low molecular concentrations. A typical CRS microscope is illustrated in Fig. 1b, in which synchronized pump and Stokes beams are combined and sent to a laser-scanning microscope. CARS signal is filtered and detected by a PMT, while the SRS signal is detected on the unmodulated probe beam (SRL on the pump is detected in the case of Fig. 1b) via a PD and a lock-in amplifier.

1.2 Instrumentation advances toward pushing the physical limits of CRS imaging

In early developments, CRS microscopes employed narrowband lasers to target a single Raman band [14, 16]. Such a single-color scheme can enable high-speed imaging of known species with distinct spectral peaks, such as lipid and myelin in living samples [17, 18]. Yet, its ability to study unknown and spectrally overlapped species is severely limited. To expand the spectral bandwidth, hyperspectral CRS has been developed to generate a spectrum at each pixel. Hyperspectral imaging can be implemented through various instrumentation approaches. One simple way is through wavelength tuning of narrowband laser to obtain CRS images sequentially over a spectral window [19, 20] (Fig. 1c). Yet the imaging speed is limited due to the slow frequency tuning speed of the laser. An alternative approach employs pulse shaping of femtosecond pulses. A mechanical stage [21] or scanning mirror [22] can be used as the pulse shaper, which is faster and more robust compared with laser tuning. The third scheme is multiplex CRS (Fig. 1d), in which a pair of broadband and narrowband lasers are used to excite all Raman peaks simultaneously. The signals are then collected by an array detector in a parallel fashion [23,24,25] to avoid spectral distortion by sample movement. The last method, termed spectral focusing (Fig. 1e), involves two broadband femtosecond pulses which are both chirped to disperse frequency components in the time domain [26,27,28]. Likewise, a mechanical delay scanner is used to tune the temporal overlap between the two pulses which translates to the change in the vibrational frequency. Compared with pulse shaping, spectral focusing offers an advantage in photon efficiency as all energies of the femtosecond pulses are used. Another key aspect of a hyperspectral CRS system is the coverage range of the spectral window, which affects the resolving power. To that end, Cicerone and coworkers implemented an ultra-broadband supercontinuum laser source to achieve hyperspectral CARS with 500–3500 cm⁻¹ spectral coverage [29], and its advantage in sophisticated compositional analysis of lipid particles in Caenorhabditis elegans is demonstrated [30] (Fig. 2a). Figueroa et al. reported a broadband hyperspectral SRS system, which incorporated a fiber amplifier on a commercial femtosecond laser system [31], expanding the spectral coverage from 200 to 600 cm⁻¹.

Fig. 2

Instrumentation advances toward pushing the physical limits of CRS microscopy. a Ultra-broadband hyperspectral CARS imaging of Caenorhabditis elegans. CARS spectra from five different lipid particle regions are shown in the 600–3100 cm⁻¹ spectral region. Scale bar, 10 μm. b Video-rate 4-color SRS imaging of Euglena gracillis. Chemical maps of lipids, paramylon, chlorophyll and protein + nucleic acid are generated. Scale bar, 10 μm. c Multiplex SRS and its application to flow cytometry. A spectrum-time window SRS flow data in 1.8 ms showed 8 PMMA and 5 PS beads. SL slit, G Grating, Cyl cylindrical lens, M mirror. TAMP tuned amplifier. PD photodiode. d Ultrafast stimulated Raman scattering microscopy based on collinear multiple beams. Top inset: AOD input waveform to generate laser combs. Bottom inset, AOD diffraction angles and images of the 46-channel laser combs for both beams. Selected frames from 2-kHz label-free imaging of the polymerization process at 3043 cm⁻¹ are shown. Scale bar, 5 μm. AOD acousto-optic deflector, RF radio frequency; TD time domain, FD frequency domain; PA photodiode array, F filter. a–d are adapted from references [30], [33], [39] and [41], respectively

Imaging speed is fundamental in determining whether the microscope can faithfully capture dynamic or high-throughput events. In fact, the development of CRS microscopy is driven by the broad interest in improving the speed of spontaneous Raman microscopy. Using high-speed resonant or polygon mirrors, CRS has achieved video rate with single color [17, 19]. Ultrafast multicolor SRS imaging was reported using laser pulse trains with periodic wavelength switching [32]. To perform high-speed hyperspectral CRS, an additional challenge in high-speed spectral acquisition arises. Ozeki et al. reported a tunable bandpass filter based on a high-speed galvo scanner and demonstrated high throughput compositional imaging of tissue [29] and live cells [33] (Fig. 2b). High-speed hyperspectral SRS has also been implemented in the spectral focusing scheme. Through the use of galvo [34], resonant [35] or polygon scanners [36] to tune the temporal delay, microsecond-level spectral acquisition speed has been achieved. Ultrafast delay-line tuning is also reported in Fourier-transform CARS (FT-CARS) using polygon [37] or resonant mirror [38]. Changing from a single element detector to an array detector can drastically improve the speed performance. Cheng and coworkers reported multiplexed hyperspectral SRS using a 32-channel tune-amplifier (TAMP) array that achieved 5 µs per spectrum [39] (Fig. 2c). Besides spectral acquisition, such one-dimensional detector arrays have also been used to simultaneously record a line illumination in space, enabling speed-demanding applications such as stimulated Raman imaging flow cytometry [40] and ultrafast imaging of chemical kinetics at two-kilohertz frame rate [41] (Fig. 2d). Further parallelizing the detection to wide field is challenging due to the insufficient laser power.

Sensitivity is the foundation that defines all aspects of the system performance. Theoretical analysis and experimental validation have demonstrated that CARS and SRS share a similar millimolar sensitivity at microsecond-scale dwell time [42]. For example, based on signals from C-H stretching vibration, a typical CARS system can reach the detection limit of 70 mM (mM) dimethyl sulfoxide (DMSO) at 10 μs pixel dwell time [43], while SRS can reach 21 mM DMSO detection limit at 83 μs [35], corresponding to a detection limit of 60 mM DMSO under 10 μs dwell time. To push the detection limit below millimolar, various schemes have been proposed. Plasmonic enhancement with nanostructures has been reported on both CARS [44] and SRS [45] with single-molecule sensitivity. With minimal requirement on laser powers, plasmonic CARS could be implemented with a wide-field configuration [46]. This opens doors to imaging at unprecedented speeds if combined with state-of-the-art high-speed cameras. Near-field approaches have also been applied to boost the sensitivity of CRS. An atomic force tip can induce local field enhancement and measure CRS-generated changes optically [47] or mechanically [48], achieving a near-single-molecule detection limit. Near field also significantly improves the spatial resolution to ~ 10 nm, enabling the study of fine structures beyond the reach of far-field optical microscopes. Min et al. proposed electronic pre-resonance SRS to enhance sensitivity by tuning the excitation wavelength to optimal conditions that both receive electronic resonance enhancement and maintain chemical specificity [49]. The scheme has achieved sub-micromolar sensitivity on chromophores. More recently, SRS has been coupled with fluorescence detection to reach single-molecule sensitivity [50]. With fluorescence detection, far-field super-resolution SRS has been achieved through stimulated emission depletion [51].

1.3 Unmixing of hyperspectral CRS images

CRS has enabled high-speed chemical imaging on biological samples based on intrinsic Raman peaks. However, biological samples are sophisticated microsystems that consist of various metabolites which often have spectral overlaps, especially in the strong yet crowded carbon-hydrogen (CH) region. This hinders the quantitation and identification of chemicals in cells and tissues using narrowband single-color CRS. Over the past years, significant endeavors have been made to develop hyperspectral CRS that produces a Raman spectrum at each pixel. Hyperspectral image offers the potential for deciphering the information on chemical compositions and abundance in a complex environment. However, due to the high dimensionality of the raw image, such information is not readily available. Algorithms are required to identify major pure components and decompose concentration maps. Parallel with instrumentation developments in hyperspectral CRS, various hyperspectral image unmixing methods have been reported. Depending on whether prior information is given on the composition of pure components, we categorize them into either supervised or unsupervised methods.

In unsupervised unmixing methods, the algorithm first performs data mining on the spectral domain to identify pure components. Since the raw spectra are high-dimensional data points and are correlative in many spectral bands, feature extraction and dimension reduction are commonly used. Principal component analysis (PCA) is a widely used dimension reduction method that aims at finding lower-dimensional orthogonal projections that maximize data variance. An example of PCA for component identification is demonstrated in hyperspectral SRS flow cytometry [39]. The authors used a 32-channel resonant tuned amplifier array to record SRS spectra from the samples passing through a flow chamber. The resulting data was a 2D matrix with a collection of spectra, which was projected by PCA to a lower dimension space for pure components identification using clustering analysis. Spectral phasor [52] is another prevalent dimension reduction method, which takes the real and the imaginary parts of the first harmonic of the Fourier-transformed spectral data. It transforms the data into a polar coordinate system where the position and width of the original peak correspond to the radius and polar angle, respectively. More importantly, combining two peaks results in a position on a line between the two individual peaks. The linearity greatly facilitates downstream clustering analysis. Spectral phasor has been widely adopted for fluorescence lifetime imaging [53] and pump-probe micro-spectroscopy [54], where the decay curves are transformed into points falling on a half circle in the phasor domain. Fu et al. reported spectral phasor for hyperspectral SRS image segmentation [55]. The performance under a sophisticated environment was validated using hyperspectral SRS images of mammalian cells (Fig. 3a–c), facilitating the segmentation of seven subcellular organelles. One limitation of the spectral phasor is the lack of robustness to noise. To address the issue, Zhang et al. introduced Markov Random Field (MRF) to phasor analysis [56]. MRF is a forward term that encourages adjacent pixel correlations of each class to suppress spurious salt-and-pepper noise. Segmentation quality improved by incorporating the prior with the forward model calculating the maximum a posteriori (MAP) estimation of the label maps.

Fig. 3

Hyperspectral CRS image analysis methods. a Maximum intensity projection of a hyperspectral CH SRS image on mammalian cells. b Phasor plot of the hyperspectral image. Yellow boxes indicate manual segmentation. c Color-coded maps of cellular components based on phasor segmentation. d Flowchart of hyperspectral CRS image unmixing using MCR-ALS. e Concentration maps and f spectra after MCR-ALS analysis of a hyperspectral SRS image on Caenorhabditis elegans in the fingerprint region. LROs, lysosome-related organelles. a–c are from reference [55], d is from reference [21] and e–f are from reference [58]

PCA and spectral phasor are dimension reduction tools that enhance features for better identification of spectral components. Downstream data segmentation or decomposition algorithms are required to generate label or concentration maps. Alternatively, nonnegative matrix factorization (NMF) aims at directly decomposing the hyperspectral data matrix as the multiplication of two submatrices, including the spectral profiles of pure components and the concentration maps. One challenge of NMF is the non-convexity of its inverse problem with two unknowns. Multivariate curve resolution (MCR) solves the problem by performing alternating least-squares (ALS) fitting on one variable while keeping the other fixed, and vice versa [57] (Fig. 3d). In 2013, Zhang et al. applied MCR-ALS to hyperspectral SRS analysis and demonstrated unmixing of cancer cell images into lipid, protein/nucleotide and water in the CH region [21]. Because it does not require prior information on chemical components, MCR has been widely adopted for discovery-driven biological research. An example of MCR-ALS analysis of fingerprint SRS imaging on Caenorhabditis elegans is shown in Fig. 3e, f in which four components were distinguished and mapped without prior knowledge [58]. Similar to phasor, MCR does not incorporate information on spatial and spectral feature correlations that could be used as priors to improve algorithm robustness. Lin et al. added neighboring pixel correlations to both spectral profiles and concentration maps through generalized Gaussian Markov Random Field (ggMRF) [59]. This regularized MCR has achieved better performance under noisy conditions or with missing values. Factorization into Susceptibilities and Concentrations of Chemical Components (FSC³) is another NMF-based algorithm for unsupervised quantitative chemical imaging [60]. Developed initially for hyperspectral CARS unmixing, the algorithm is also applicable to SRS with modifications to pre-processing procedures. FSC³ solves the optimization of two variables using an Iterative fast block principal pivoting algorithm [61] and adds a physical constraint that ensemble concentrations sum up to one. The physical constraint enhances downstream quantitation of absolute concentrations. An improved version, weighted FCS³, was proposed in 2015 [62]. In this work, the authors added weights to emphasize the contribution of spectral errors over random noise and systematic errors at compound aggregated positions. Weighted FCS³ is advantageous in processing sparse chemical species whose contributions would be overlooked if we use global optimization of the entire image. Ozeki et al. proposed independent component analysis (ICA) [22], another NMF which assumes measured spectral data is a linear summation of independent components with maximum non-Gaussianity. After PCA analysis of the raw hyperspectral data, it operates with principal component images to find an unmixing matrix that maximizes the skewness. Each independent component image is then reconstructed to map a pure component.

We have discussed unsupervised methods that decipher the chemical composition of unknown systems. However, even with advances in algorithms, such a task is still inherently challenging, as the output is prone to the influence of non-resonant background, noise corruption and spectral crosstalk. In some cases, prior information on the chemical composition can be obtained from modalities such as mass spectroscopy and fluorescent labeling. Hyperspectral CRS aims to provide label-free, multiplexed chemical imaging that is not achievable otherwise. With spectral information on pure components, hyperspectral CRS images can be decomposed in a robust and efficient manner. The simple least-square fitting has been applied to unmix hyperspectral SRS images into chemical maps [20]. However, its performance is limited under high spectral crosstalk or low SNR. Lin et al. addressed the issue by incorporating a pixel-wise l1 norm regularization to promote sparsity on the number of contributing species at each pixel [36]. Such "chemical sparsity" is based on the observation that at each pixel, only a few components have significant concentrations.

1.4 Computational methods to break the design space trade-offs

As said above, instrumentation innovations have pushed CRS imaging to the speed of up to 2 kHz frame rate, spectral coverage of up to 3500 cm⁻¹, and spectral acquisition speed of up to 5 µs per spectrum. However, these conditions cannot be realized simultaneously due to the physical limit determined by the sensitivity limit of CRS. For example, further increasing the speed will deteriorate the setup's signal-to-noise ratio (SNR), rendering it inapplicable to biomedical applications. Under the constraint of photodamage, this trade-off can be conveyed as a design space—a hyperplane intersecting with three axes representing speed, spectral bandwidth and SNR. Optimization on instrumentation enables the system to reach an optimal condition point on the hyperplane, yet going beyond it remains challenging.

One appealing way to overcome the barrier is through computational methods. In conventional CRS microscopy, the focus is on the design of instruments that generates data. Computational methods go one step further by treating the raw data not as the results but as measurements that can be further processed to provide signals with more desirable information. A typical computational imaging problem can be described as an inverse problem, in which a forward model is used to describe the physical process that generates experimental measurements from signals (e.g., the addition of measurement noise, the modulation from signal to measurement). Inversion of the forward process then yields the desired signals from measurements. When the inversion of the forward model does not yield optimal results, prior models (or regularizations) are incorporated to stabilize the inversion problem. These prior models assume certain properties of the signal based on existing knowledge (e.g., image smoothness, signal sparsity, etc.). One straightforward example is denoising [63, 64], which models the noise distribution and adds prior knowledge of spatial correlations to extract signals from noisy measurements. Algorithms can also synergistically guide the design of instrumentation to facilitate reconstruction. Fourier ptychography [65] and structured illumination microscopy [66] are examples that surpass the physical limits of the space-bandwidth product or spatial resolution.

In the following sections, we highlight recent advances in computational CRS imaging. First, we review compressive methods to improve the speed without loss of information by utilizing sparsity in the spatial and spectral domains. We then discuss computational methods for volumetric CRS imaging, including digital holography, computed tomography and spatial frequency encoding. Lastly, we survey applications of deep learning in CRS micro-spectroscopy, including denoising, background removal, chemical label segmentation, etc.

2 Compressive CRS micro-spectroscopy

As CRS is evolving towards high spectral bandwidth and high speed, challenges arise in both system performance and data handling. On the one hand, trade-offs in the design space inevitably lead to compromises between parameters. On the other hand, data recording and processing becomes overwhelming as frame rate and spectral bandwidth increase. Compressive sensing, a concept in signal processing, is a promising approach to address the issues. Compressive sensing aims at acquiring data below the Nyquist sampling rate while preserving the same information as conventional sampling. This is possible because the data is sparse, and measurements are random and dispersed in the sample space [67]. Images are highly sparse signals as the "information rate" is often much lower than the raw data bandwidth. Namely, images contain many low spatial frequency signals that can be concisely expressed with a proper transformation (e.g., wavelet transform). The sparsity condition is even more entrenched for hyperspectral images as spectra also contain highly compressive features such as broad Raman peaks. With fewer sampled points, the system can achieve a higher image speed. Alternatively, each sampled point can share a longer signal integration time which leads to an increased SNR. Due to its nonlinear nature, tight-focusing laser scanning is generally required for CRS, which makes the most prevalent compressive imaging approach (wide-field spatial random multiplexing + single-pixel detection) unviable or underperformed. Therefore, sampling approaches without loss of power density on the sample are required.

The first compressive strategy focuses on the spectral domain since random multiplexing in the spectral domain does not conflict with the tight-focusing condition. In 2017, Berto et al. reported a compressive spectral SRS scheme [68]. As shown in Fig. 4a, a narrowband pump and a broadband Stokes were combined and interacted with the sample, generating a spectrum of Raman peaks simultaneously. To record the spectrum with a single photodiode, a digital micromirror device (DMD) was placed after the grating at the collection side. The DMD served as a programmable spectral filter. A spectrum could either be recorded via conventional raster scan or compressive multiplexing with a Hadamard compressive basis (Fig. 4b). Compressive data reconstruction with l1 norm regularization was implemented for compressive multiplexing measurements. Compared to raster scanning, compressive multiplexing achieved a 30% compression rate with ~ 60% fidelity. The second compressive CRS was implemented by Takizawa et al. in FT-CARS spectroscopy [69], in which the spectral information was recorded by taking the Fourier transform of a time-domain interferogram. As depicted in Fig. 4c, a resonant mirror was used to scan the temporal delay to generate the interferogram. As a prerequisite for compressive sensing, the reconstructed signal (i.e., the target spectrum) must be sparse. Raman spectra over a broad window (e.g., the whole fingerprint region) are generally sparse since they consist of narrow Raman peaks. Due to the nonlinear scanning speed of the resonant mirror, random measurement was elegantly achieved by temporally uniform sparse sampling below the Nyquist rate. Reconstruction was implemented using standard compressive sensing algorithms with l1-norm regularization on the signal. Figure 4d demonstrates the results of compressive sensing as compared with interpolation under the same sampling rate and full sampling. At 30% compression rate, compressive sensing maintained high fidelity, free of spurious artifacts as shown in interpolation.

Fig. 4

Compressive CRS micro-spectroscopy. a Setup of compressive spectral SRS. b Programmable spectral filter using DMD, which can be either programmed for conventional raster scanning or multiplex compressive sensing using a Hadamard basis. Quantification of spectral fidelity between full sampling and compressive sensing at different compression rates is shown. c Setup of compressive FT-CARS. d Comparison between compressive sensing, sparse sampling interpolation and fully sampled spectrum. e Setup of compressive hyperspectral SRS based on matrix completion. f Laser scanning and frequency tuning using 3D triangular Lissajous trajectory. Sampled pixels from three spectrally adjacent frames are projected and color-indexed. g Model-based matrix factorization algorithm to decompose sparsely sampled hyperspectral SRS image into concentration maps and spectra of pure components. a–b are from reference [68], c–d are from reference [69] and e–g are from reference [59]

Compressive CRS can also be implemented jointly in the spatial and spectral dimensions. Conventional spatial compressive sensing requires each measurement to be a random multiplexing of the entire field of view. This is challenging for a laser-scanning system like CRS as each spatial measurement can only be contributed from a diffraction-limited focal spot. Lin et al. proposed a compressive hyperspectral SRS scheme based on matrix completion [59]. Closely related to compressive sensing, matrix completion claims that a low-rank matrix can recover the complete information from a subset of randomly sampled entries [70]. Hyperspectral images are naturally low-rank matrices since the number of spectral frames is much greater than the number of pure components. More importantly, for the matrix completion problem, each measurement is a single matrix entry instead of random ensembles of the entire view, making it compatible with the laser-scanning scenario. The authors adopted spectral focusing SRS with a galvo-based delay tuner (Fig. 4e) such that all three axes could be tuned at similar speeds. To achieve random sampling over the entire hyperspectral image stack with microsecond pixel dwell time, a three-dimensional scanning scheme based on a triangular Lissajous trajectory was proposed. Scanning frequencies of the three scanning axes were designed to deviate slightly from each other, achieving a 3D pseudo-random trajectory which randomly sampled across the hyperspectral stack, as shown in Fig. 4f. A model-based matrix factorization algorithm was implemented to decompose the sub-sampled hyperspectral image into the multiplication of two submatrices: pure components spectra and concentration maps. Prior models on both submatrices were added to promote correlations on adjacent entries. As depicted in Fig. 4g, a compression ratio of 20% was achieved on freely moving biological samples, reaching a speed of 0.8 s per hyperspectral stack.

Since compressive CRS primarily involves the spectral domain, having prior knowledge of the spectral information can lead to supervised compression schemes that are more robust and efficient. In 2011, Freudiger and coworkers published a spectrally tailored excitation-stimulated Raman scattering (STE-SRS) scheme [71]. In this work, a broadband pump beam was spectrally dispersed onto a spatial light modulator (SLM) and was combined with a narrowband Stokes beam to excite all the Raman bands allowed by the SLM. With information on the Raman spectra of all the species, the authors reduced measurements from tens of frames to two for each component. Two SLM patterns were applied to maximize the transmission from target peaks and suppress crosstalks from overlapping components. A similar implementation was reported by Bae et al. with a different approach to SLM pattern design and signal recovery [72]. For compressive hyperspectral CARS, a two-step approach was reported by Masia and coworkers [73], where a spatially under-sampled, spectrally fully-sampled image was first acquired to calculate the sparse spectral bins able to reconstruct the full spectral information. In the second step, spatially fully-sampled images with few spectral frames were acquired, achieving a 25 times speed enhancement. For narrowband CRS systems, each measurement is confined to a certain Raman band. In this case, selective targeting of significant peaks can be performed via feature selection algorithms such as the Least Absolute Shrinkage and Selection Operator (LASSO) regression [74].

3 Computational volumetric CRS imaging

Similar to multiphoton microscopy, CRS is inherently a nonlinear process with axial sectioning capability. In a standard CRS system, pump and Stokes beams are collinearly focused on the sample as a Gaussian shape and form a planar image through laser raster scanning. For volumetric imaging using such a conventional scheme, one needs to perform axial scanning of either the objective or the sample. Nevertheless, mechanical scanning suffers from high data throughput and low speed, which is unsuitable for large volume or in-vivo 3D imaging. Computational volumetric imaging, such as digital holography [75], light field microscopy [76] and optical diffraction tomography [77] provide scanless 3D information of the sample, but they typically require wide-field illumination, which is detrimental to nonlinear optical processes such as CRS. For SRS, wide field is further undesirable, as existing cameras do not have sufficiently high dynamic range and saturation power to detect tiny fluctuations on a strong laser field.

Regardless of the limitations of wide-field CRS, digital holographic CARS has been reported [78] for 3D imaging. As depicted in Fig. 5a, the authors applied wide-field illumination of the combined pump and Stokes on the sample to generate CARS images at 2ω_p − ω_S. Meanwhile, a reference beam at the same 2ω_p − ω_S frequency interfered with the CARS signal at the camera plane, creating an off-axis hologram that contains both the amplitude and phase of the complex CARS field. The raw hologram was then separated into amplitude and phase in the Fourier domain based on differences in spatial frequencies. The phase was then mapped into sample thickness using digital propagation equations. An example of the raw hologram, intensity, phase, and digitally propagated CARS field at different depths for polystyrene microbeads in water are shown in Fig. 5b. Further improvement has been reported for reconstruction of sparse signals, which adopted compressive reconstruction to reduce out-of-focus background [79]. CARS holography enables fast single-shot volumetric imaging without scanning with a large depth of field. However, the sensitivity and detection depth are incommensurate with laser scanning CRS, which hinders its further applications on biological samples.

Fig. 5

Computational volumetric CRS imaging. a Setup of wide-field CARS holography. b From top left to bottom right, raw CARS hologram, amplitude, phase and digitally propagated CARS field at three different depths (z = 7.92 µm, 2.92 µm and -15.6 µm) of polystyrene microbeads in water. c Schematic of stimulated Raman projection (SRP) tomography with a rotational stage. SRP volumetric imaging of a single adipose cell is demonstrated. d tilt-angle stimulated Raman projection tomography (TSRPT). Tilt-angle beams incident at four different angles are focused by the objective. Volumetric reconstruction from four tilt-angle SRS projection images of PS beads is shown. e Principles of stimulated Raman scattering tomography (SRST) with frequency encoding. f SRST and conventional SRS imaging of mouse ear skin at different depth sections are illustrated. a, b are from reference [78], c, d are from references [80] and [81], respectively, e, f are from reference [85]

An alternative approach to performing volumetric imaging is tomography. By collecting projection images from different angles, a 3D volume can be rendered with isotropic spatial resolution. Chen et al. proposed Bessel-beam-based volumetric stimulated Raman projection (SRP) tomographic imaging [80]. To extend the collection volume, both pump and Stokes beams were transformed from Gaussian to Bessel shape to maintain focus over an extended range. Using a two-dimensional beam scanner and single-element detector, a projection image was recorded. As depicted in Fig. 5c, with a rotational sample stage, a series of projection images were recorded from different incident angles. A filtered back projection algorithm was applied to reconstruct volumetric SRS images with isotropic spatial resolution in all directions. SRP volumetric imaging of a single adipose cell demonstrates its depth-resolving performance and low background level. The scheme is ideal for studying 3D structures of cells as it provides isotropic spatial resolution. However, due to the need for sample rotation in a capillary glass, it is too cumbersome to perform volumetric imaging on living cells or large tissue samples. Lin et al. proposed a simplified approach, tilt-angle stimulated Raman projection tomography (TSRPT) [81], which acquires projection images from different azimuth angles by tilting the under-filled incident beams at the back pupil (Fig. 5d). Tilting and scanning were achieved by two scanning mirror pairs, avoiding mechanical movements of the sample. A vector-field-based back-projection algorithm was applied to reconstruct a volume. Volumetric rendering with four tilting angle projections on PS beads is shown in Fig. 5d. Compared with SRP, the system has higher throughput as it requires fewer projection images. However, the limited projection angle reduces the axial resolution and creates missing-cone issues.

In projection tomography, the purpose of sample rotation or beam tilting is to decode depth information from individual projection angles. Alternatively, such an encoding–decoding process can be performed in the frequency domain. Frequency domain encoding has been reported to capture spectral information in SRS without a spectrometer [82, 83]. Spatial frequency modulation was also used to record lateral line-scan profiles in CARS with a single detector [84]. In a recent work by Li et al. [85], spatial frequency encoding was performed in the axial direction to achieve high-speed volumetric SRS imaging. As in Fig. 5e, the authors engineered the Stokes to Bessel beam to extend the imaging volume. Meanwhile, the pump was modulated by an SLM to form Bessel "light beads" with different beating frequencies. By varying the spatial frequency to record a series of projection images using laser scanning and a single detector, a depth-resolved SRS image was reconstructed using a robust inverse fast Fourier transform. A comparison between conventional SRS and the new scheme is shown in Fig. 5f. Owning to the self-reconstructing nature of Bessel beams, a two-fold enhancement in imaging depth is achieved. Like TSRPT, the system is free of sample rotation, making it accessible for live cell and tissue imaging.

4 Deep-learning CRS microscopy

For computational imaging schemes, it is essential to establish the forward model of the system and develop algorithms to solve the inverse problems for reconstruction. However, characterizing the forward model operation matrix is both labor-intensive and prone to system perturbations. Plus, the inverse problems are often ill-posed with multiple solutions, requiring additional regularization terms based on assumptions of the underlying images, such as smoothness or sparsity. Deep learning is a revolutionary method that can directly learn to perform sophisticated tasks on image reconstruction using adequately annotated training data [86]. Knowledge of the specific tasks and properties of the data are embedded in the training of a multi-layered neural network, with minimal requirements for the human design of the system. Deep learning has been widely used in optical microscopy, such as image denoising [87], cross-modality image transformations [88], fluorescence prediction from transmission images [89, 90], image-based cell profiling [91, 92], speed enhancement [93] and PSF engineering [94] in super-resolution localization microscopy. In this section, we focus on CRS and discuss various deep learning applications that either improve the fundamental performance of CRS or strengthen the downstream CRS image analysis.

Denoising is an image restoration procedure that aims at decoupling noise from an image to enhance signal fidelity. Statistical denoising methods, such as total variation [95], block-matching 3D filtering [63], non-local means [64], solve the problem by modeling noise distribution and signal structure. However, the models are not universal in different scenarios, and tedious hyper-parameters tuning is often necessary. In CRS microscopy, system noise is mainly contributed by laser noise, electronic noise and shot noise. Laser noise follows a 1/f trend and can be minimized through modulating at a higher frequency in SRS. Electronic noise, i.e., Johnson-Nyquist noise, is related to the input impedance and is irrelevant to laser power. The third and major source of noise is the shot noise which is proportional to the square root of the input laser power. It statistically follows Poisson distribution but can be approximated by Gaussian at high laser power conditions. For denoising a hyperspectral CRS image, one needs to further consider different noise levels of spectral frames due to inhomogeneous laser powers. Treatment for this problem was demonstrated by Liao et al., in which a spectrally varying total-variation denoiser first estimated noise levels at all frames and then performed total variation minimization [96]. Deep learning has shown the potential in denoising microscopic images of various modalities, outperforming traditional denoising algorithms [87]. Several research groups have reported using deep learning to recover the SNR of CRS images in different contexts. Using the same U-Net structure as in ref [87], Manifold et al. demonstrated the denoising of 2D SRS images of cells and tissues under low exposure powers [97]. CNN denoising of 2D endoscopic CARS images has been reported by Yamato and coworkers [98]. As illustrated in Fig. 6a, Lin et at. reported a Spatial-Spectral Residual Net (SS-ResNet) for denoising of hyperspectral images [36] and developed a downstream hyperspectral image unmixing scheme. Inspired by work in the video-processing community [99], the authors implemented a pseudo-3D convolution kernel using the combination of a 2D spatial and a 1D spectral kernel. Because SS-ResNet has a smaller model size compared to 3D CNN, a very deep network can be implemented to enhance the denoising performance. With an ultrafast hyperspectral SRS setup, the authors achieved high-speed hyperspectral SRS in the fingerprint region. After denoising, the image fidelity on spatial and spectral dimensions was validated by spectral unmixing the hyperspectral images into three chemical maps (Fig. 6b). Abdolghader et al. treated hyperspectral SRS images as a collection of spectra that can be processed by a convolutional autoencoder with 1D kernels [100]. Using noisy images as both input and output, unsupervised image denoising was reported but with a sacrifice in spectral fidelity. The authors further applied a k-means clustering algorithm after denoising to yield chemical segmentation maps in an automatic and unsupervised manner. Denoising of hyperspectral CARS images with a 1D spectral network has also been achieved [101] for high-speed fingerprint CARS imaging.

Fig. 6

Deep learning for CRS image denoising and segmentation. a Principles of SS-ResNet image restoration and spectral unmixing. GT, ground truth. b Comparison between raw, SS-ResNet denoising and GT. The hyperspectral images are decomposed into chemical maps of protein, cholesterol and fatty acid. Scale bars, 20 µm. c Structure and workflow of training and validation of ResNet34 for the segmentation of SRS histology images. d SRS histological images of neoplastic (top) and normal (bottom) larynx tissue and the network classification results. a, b are from reference [36] and c, d are from reference [106]

As a high-speed label-free chemical imaging scheme that operates under ambient light, SRS has emerged as a promising candidate for real-time intraoperative histopathology [102]. However, performing real-time decision-making with SRS is a challenging task for pathologists due to the following challenges. First, the histoarchitectural contrast in SRS originates from intrinsic Raman peaks of the tissue, which differs from conventional hematoxylin-and-eosin (H&E) staining contrast familiarized by pathologists. Second, the high volume of SRS images during surgery can overwhelm the pathologists during operation, making it challenging to provide intraoperative consultation. Supervised methods for medical image segmentation and classification are effective tools to alleviate human labor. Traditional statistical classifiers can automatically find the optimal decision boundary in the high-dimensional feature space. Yet, extraction of discriminative features from medical images requires input from experienced researchers, which lacks robustness for intraoperative decision-making. Deep learning is an ideal solution for robust biomedical image segmentation [103], as feature extraction is implemented automatically during the learning process. SRS histology has been closely integrated with deep learning over the past years. In 2017, Orringer et al. reported a portable fiber-laser-based SRS microscope for intraoperative imaging of unprocessed surgical brain tumor specimens [104]. The setup generates 2-color SRS images (2850 cm⁻¹ & 2930–2850 cm⁻¹) that simulate the contrast of H&E staining. A multi-layer perceptron was used for classifying SRS images into non-lesional, low-grade glial, high-grade glial, or non-glial tumors. The performance was validated through leave-one-out cross-validation. The classifier achieved 100% accuracy in distinguishing lesional from non-lesional and 90% in differentiating glial from non-glial among lesional specimens. In 2020, with the same portable SRS histology setup for intraoperative brain tumor diagnosis, the authors applied CNN for the prediction and achieved 94.6% overall accuracy, which is comparable to pathologist-based interpretation using conventional H&E staining [105]. Zhang et al. reported two-color SRS (2850 cm⁻¹ + 2930 cm⁻¹) to highlight protein and lipid and added second harmonic generation (SHG) to reveal collagen [106]. As demonstrated in Fig. 6c, a residual network with a CNN kernel was trained to identify neoplastic regions from normal regions with the three-color SRS histology images. Figure 6d illustrates SRS histology imaging of laryngeal SCC tissue and normal tissue, followed by the network prediction on either neoplastic (red) or normal (grey). An accuracy of 90% was achieved on the dataset, and further testing on 33 independent specimens yielded near-perfect diagnostic concordance between SRS-based prediction and standard histology readings. Deep learning segmentation of SRS images also facilitated discovery-driven biomedical research. Feizpour et. al. reported visualization of drug uptake in the skin with SRS and deep-learning-based image segmentation [107]. Parallel with SRS, deep learning for classification and clinical decision-making has been combined with CARS. Broadband CARS spectral classification has been reported by Manescu et. al. using a tandem artificial neural network with a decision rule model [108]. The rule assumes each target class is a linear combination of input spectra, achieving high prediction accuracy with interpretable results. Diagnosis of cervical cancer using single-color CARS microscopic images of pap smears of patient samples has been demonstrated with CNN [109]. A cross-validation with Raman microspectroscopy classification was performed, demonstrating the high prediction accuracy of the method. Weng et al. applied transfer learning on a pretrained CNN using CARS images of lung cancer for automated differential diagnosis, achieving 89.2% accuracy in identifying normal and cancerous tissue [110].

Deep learning has also been used for CARS non-resonant background removal. In CARS micro-spectroscopy, one major drawback is the dispersive line shape due to the interference between the resonant and non-resonant components of ${\chi }^{(3)}$, leading to a peak shift and a dip at higher wavenumbers. As mentioned earlier, numerical phase retrieval algorithms have been developed to extract Raman line shapes from raw CARS spectra, including time-domain Kramers–Kronig (TDKK) transform [11] and maximum entropy method (MEM) [9, 10]. However, they require knowledge of the non-resonant background which needs separate measurements, and the computation speed is insufficient for applications with real-time display requirements. Several groups reported deep learning phase retrieval [111,112,113]. Figure 7a demonstrates an example where a 1D network for CARS non-resonant background removal was used. Successful training could be achieved through simulated CARS spectra owing to the robust generalizability of neural networks. Deep-learning-based background removal is advantageous in that it requires no experimental measurement of the non-resonant background and has a fast processing speed of milliseconds. A similar approach has also been demonstrated to remove cross-phase modulation background in spectroscopic SRS [114] using the peak width differences between the Raman band and the background.

Fig. 7

Deep learning for CRS-specific applications. a A 1D neural network for non-resonant CARS background removal. Examples of simulated raw CARS spectra (blue), true Im(χ_R ⁽³⁾) (green) and network-predicted Im(χ_R ⁽³⁾) (red) are shown. CL convolutional layers, FC fully connected layers. b Schematic of the U-within-U network. c Left, hyperspectral SRS image (maximum intensity projection) of an unlabeled live lung cancer cell. Right top row, network predicted fluorescence labels of nuclei, mitochondria and endoplasmic reticulum. Right bottom row, fluorescence images after staining. Scale bar, 25 μm. d Schematic of single-shot femtosecond SRS mapping of intracellular organelles using deep learning. Orange arrows indicate training set generation, green arrows represent training validation, and blue arrows stand for testing. (e) Illustration of U-net prediction of two-color picosecond SRS images (2845 & 2930 cm⁻¹) using single-shot femtosecond SRS. Example prediction results on gastric tissue are shown. Scale bars, 50 µm. a is from reference [112], panels b, c are from reference [116], d is from reference [56] and e is from reference [118]

Besides outperforming traditional algorithms in areas such as image segmentation and denoising, deep learning can tackle problems where analytical modeling is impossible due to our limited perception of underlying physical processes. Examples include the prediction of fluorescent labels from a stack of bright-field transmission images [89, 90], super-resolution localization image prediction using wide-field fluorescent images [93] and fusion of low and high-resolution images to break the space-bandwidth-product [115]. Manifold et al. proposed using hyperspectral SRS images to generate fluorescent-labeled images [116]. A U-within-U network was developed for hyperspectral image processing to produce label-free multiplexed fluorescent images without physical constraints on fluorescent emission overlap (Fig. 7b, c). Zhang et al. reported using deep learning to map intracellular organelles using single-shot femtosecond SRS images [56]. As shown in Fig. 7d, in femtosecond SRS, since both pump and Stokes are broadband lasers, the chemical contrast in one single shot is contributed by all Raman bands in the spectral window. Due to the variation in chemical composition, major intracellular organelles such as lipids, nuclei and cytoplasm exhibit intensity differences. Together with morphological signatures, segmentation of organelles can be achieved with femtosecond SRS despite the lack of spectral profiles. Nevertheless, the task is challenging for analytical segmentation algorithms since extensive feature selection is required to utilize spatial and intensity signatures for optimizing the performance. In this work, a neural network with DenseNet [117] structure was applied for segmentation. To obtain training data, the authors used spectral-focusing-based hyperspectral SRS imaging on the target samples. On the one hand, simulated femtosecond SRS was generated through the summation of the stack along the spectral dimension. On the other hand, organelle maps were calculated through phasor analysis of the hyperspectral data. A trained network successfully mapped organelle maps of lipid, nucleus, cytoplasm and endoplasmic reticulum (ER) with femtosecond SRS, which enabled high-speed imaging of lipid motility and lipid-ER interaction. Similarly, Liu et al. used single-shot femtosecond SRS images as input to train a U-Net generating two-color picosecond SRS images at 2845 and 2930 cm⁻¹(Fig. 7e) [118]. Another diagnostic network was used in conjunction with the chemical mapping network to provide instant diagnosis of gastroscopic biopsy with > 96% accuracy.

5 Outlook

In this review, as summarized in Table 1, we have introduced various computational methods that have been used to push the boundary of CRS chemical microscopy in the aspects of compressive sensing, volumetric imaging, denoising, segmentation and others. Meanwhile, one needs to pay attention to the applicable range of computational algorithms to avoid erroneous interpretations of the measurements. For conventional statistical computational imaging, it is crucial to evaluate whether the forward model can appropriately describe the underlying physical process. This involves the statistical distribution of measurement noise, the image convolution kernel of the imaging system, etc. Rigorous experiments should be taken to characterize the forward model and calibrate model parameters. When prior models/regularizations are used, a comprehensive understanding of the signal is necessary. For example, whether the signal is sparse, whether the signal is smooth, etc. Hyperparameter tuning for the prior models is crucial for yielding correct results and may require iterative updates and validations. For deep learning applications, although the task of sophisticated modeling on the inverse problem is alleviated, an appropriate selection of network structures and sufficiently large training and validation datasets are necessary.

Table 1 Summary of computational CRS methods

Looking into the future, we expect instrumentation advances will continue to increase the data throughput on temporal, spatial and spectral dimensions, which provides more features on data structures such as sparsity and correlation. Meanwhile, new computational methods can be harnessed to break the design space trade-offs and provide enriched chemical compositions for biomedical research. With rapid advances in computational optical microscopy, we expect more ideas to infiltrate CRS. However, since most computational methods focus on wide-field implementations, the translation into CRS microscopy is nontrivial. Extensive modeling, system design and algorithm development need to be performed to ensure applicability to CRS imaging. In the future, with improvements in detection sensitivity, computational methods will play an even more important role as existing methods remain viable to boost the newly established design space, and new methods may arise to achieve breakthroughs in aspects such as field of view, imaging depth, and spatial resolution.