GLINT-RU: Gated Lightweight Intelligent Recurrent Units for Sequential Recommender Systems

Abstract.

In the rapidly evolving field of artificial intelligence, transformer-based models have gained significant attention in the context of Sequential Recommender Systems (SRSs), demonstrating remarkable proficiency in capturing user-item interactions. However, such attention-based frameworks result in substantial computational overhead and extended inference time. To address this problem, this paper proposes a novel efficient sequential recommendation framework GLINT-RU that leverages dense selective Gated Recurrent Units (GRU) module to accelerate the inference speed, which is a pioneering work to further exploit the potential of efficient GRU modules in SRSs. The GRU module lies at the heart of GLINT-RU, playing a crucial role in substantially reducing both inference time and GPU memory usage. Through the integration of a dense selective gate, our framework adeptly captures both long-term and short-term item dependencies, enabling the adaptive generation of item scores. GLINT-RU further integrates a mixing block, enriching it with global user-item interaction information to bolster recommendation quality. Moreover, we design a gated Multi-layer Perceptron (MLP) for our framework where the information is deeply filtered. Extensive experiments on three datasets are conducted to highlight the effectiveness and efficiency of GLINT-RU. Our GLINT-RU achieves exceptional inference speed and prediction accuracy, outperforming existing baselines based on Recurrent Neural Network (RNN), Transformer, MLP and State Space Model (SSM). These results establish a new standard in sequential recommendation, highlighting the potential of GLINT-RU as a renewing approach in the realm of recommender systems. The implementation code is available at https://github.com/szhang-cityu/GLINT-RU.

1. INTRODUCTION

In this era of data exploding, Sequential Recommender Systems (SRSs)  (Zhang et al., 2019; Yue et al., 2024; Liu et al., 2023a; Hidasi et al., 2015; Kang and McAuley, 2018; Liu et al., 2024; Li et al., 2022) have gained much attention in capturing users’ preferences within a large amount of sequential interaction data. GRU4Rec  (Hidasi et al., 2015) as one of the earliest session-based recommendation models, employs stacked Gated Recurrent Units (GRU) for item-to-item recommendations. However, the RNN-based methods (Hidasi et al., 2015; Li et al., 2017) are fading from the recommendation realm due to their relatively lower accuracy. In recent years, transformer-based SRSs  (Kang and McAuley, 2018; Sun et al., 2019) have become increasingly popular for the powerful multi-head attention mechanism  (Vaswani et al., 2017). They exhibit remarkable ability in capturing sequential interactions and delivering accurate predictions (Li et al., 2023). However, despite their effectiveness, current attention-based models suffer from substantial computational demands and extended inference time, which is caused by the dot product operation in the attention mechanism  (Wang et al., 2020; Liu et al., 2023a).

To tackle the issue of the high computational cost of attention-based SRSs, much research has been done to improve the attention mechanism to the linear computational complexity. LinRec  (Liu et al., 2023a) changes the order of dot product between query and key matrices by designing a special map**, dramatically reducing the inference time. LightSAN  (Fan et al., 2021) projects historical interactions into interest representations with shorter length, thereby reducing the computational complexity of transformers to a linear scale. Mamba4Rec  (Liu et al., 2024) outperforms the existing attention mechanism by utilizing one efficient mamba block based on a selective SSM variant (i.e., Mamba (Gu and Dao, 2023)), within which a structured state tensor is used to address long-range dependencies. SMLP4Rec  (Gao et al., 2024) is an efficient pure Multi-layer Perceptron (MLP) framework that achieves fast inference by resha** input sequence tensors, demonstrating the potential of MLP-based models  (Li et al., 2022; Zhou et al., 2022; Long et al., 2024) for sequential recommendation tasks.

However, to achieve high performance, the transformer-based recommendation requires deeply stacked transformers. Even with a linearized attention mechanism, the inference speed of such large transformer architecture may not meet the desired efficiency standards  (Zhang et al., 2024). Furthermore, the efficiency of the MLP architecture is adversely affected by the resha** of large tensors, which is notably time-consuming. GRU4Rec  (Hidasi et al., 2015) is also an efficient framework for recommendation tasks, but it exhibits limitations in learning complex user behaviors. In this paper we aim to further improve the efficiency of recommendations, and tackle the issue of low accuracy of traditional RNN-based frameworks.

To further reduce resource consumption, accelerate inference speed, and enhance the model performance, we propose a novel efficient SRS framework named Gated Lightweight IntelligeNT Recurrent Units (GLINT-RU). This paper employs various gate mechanisms in appropriate positions to fully perceive the data environment and filter or select information automatically  (Zhang et al., 2024; Chang et al., 2023; De et al., 2024). We introduce an expert mixing block that captures the long-/short-term dependencies via GRU, and utilize linear attention to capture the global interaction information between users and items (Jacobs et al., 1991; Masoudnia and Ebrahimpour, 2014). This strategy not only supplements the sequential information, but also improves the inference speed due to the linear computational complexity of paralleled GRU and linear attention mechanism. Additionally, we implement a dense selective GRU, which selects the output of GRU adaptively, and considers the connections among adjacent items. It leverages the gate mechanism to select crossed features, and extracts short-duration patterns to refine the model’s understanding of user behavior dynamics. Moreover, a gated MLP block is utilized to select on the outputs of the expert mixing block, deeply filtering the information based on the data environment.

We summarize the major contributions of our work as:

• In this paper, we introduce GLINT-RU, a novel and efficient recommendation framework that achieves remarkable inference speed with a streamlined architecture that only requires a single layer, and exhibits state-of-the-art performance. It is a advanced model that captures complex user-item interactions from mixing blocks, which substantially improves the performance of traditional sequential recommendation models.

• We introduce a dense selective GRU module, which not only incorporates connections between adjacent items, but also empowers the model with the capability to selectively learn sequential information. The integration of this advanced GRU module into the model markedly elevates its performance, establishing a new standard for efficient recommender systems.

• We implement a gated MLP block, allowing for deep filtering of dense information by automatically perceiving the data environment. This mechanism enables our model to be more flexible and adaptive to complex sequential user behaviors.

2. PRELIMINARIES

In this section we will briefly introduce our recommendation task, and then introduce the basic efficient GRU and linear attention modules used in our framework.

2.1. Problem Statement

For a sequential recommendation task, we have a set of users $\mathcal{U}=\{u_{1},u_{2},\dots,u_{|\mathcal{U}|}\}$ who have historical interactions with a set of items $\mathcal{V}=\{v_{1},v_{2},\dots,v_{|\mathcal{V}|}\}$. Among these users, the $i$-th user has a preferred item sequence denoted as $s_{i}=[v_{1}^{(i)},v_{2}^{(i)},\dots$, $v_{n_{i}}^{(i)}]$, where $n_{i}$ is the length of the item list that the $i$-th user interacts with. Our goal is to design an efficient framework and predict the next item that the user is most likely to interact.

2.2. Gated Recurrent Units

As an essential part of GLINT-RU framework, the GRU  (Chung et al., 2014) module contributes to the recommendation task by capturing the long-term dependencies among the items in the sequence and dynamically adjust its memory content across different parts of the sequence  (Dey and Salem, 2017; Shen et al., 2018). The update mechanism of a GRU cell can be formalized as follows:

(1) $$\boldsymbol{z}_{t} =\sigma(\boldsymbol{W}_{z}\cdot[\boldsymbol{h}_{t-1},\boldsymbol{x}_{t}]+\boldsymbol{b}_{z}),$$

$$\boldsymbol{r}_{t} =\sigma(\boldsymbol{W}_{r}\cdot[\boldsymbol{h}_{t-1},\boldsymbol{x}_{t}]+\boldsymbol{b}_{r}),$$

$$\tilde{\boldsymbol{h}}_{t} =\tanh(\boldsymbol{W}\cdot[\boldsymbol{r}_{t}*\boldsymbol{h}_{t-1},\boldsymbol{x}_{t}]+\boldsymbol{b}),$$

$$\boldsymbol{h}_{t} =\boldsymbol{z}_{t}*\boldsymbol{h}_{t-1}+(1-\boldsymbol{z}_{t})*\tilde{\boldsymbol{h}}_{t},$$

where $\sigma(\cdot)$ is the sigmoid activation function, $\boldsymbol{x}_{t}$ is the input of GRU module in $t$-th time step, $\boldsymbol{h}_{t}$ represents the $t$-th hidden states, $\boldsymbol{z}_{t}$ and $\boldsymbol{r}_{t}$ are the update gate and the reset gate, respectively. $\boldsymbol{b}_{z},\boldsymbol{b}_{r},\boldsymbol{b}$ are bias, $\boldsymbol{W}_{z},\boldsymbol{W}_{r},\boldsymbol{W}$ are trainable weight matrices. As is shown in Eq.(1), GRU uses the update gate to control the retained information volume from previous hidden states in the current time step, while the reset gate controls the information that should be forgotten.

The GRU (Gated Recurrent Unit) module, equipped with update and reset gates in sequential GRU Cells, is adept at capturing the relationships among the items throughout the sequence, while maintaining a relatively low computational complexity. However, the sequential information in GRU cannot be interacted, and each hidden state is primarily encoded from preceding elements, which restricts the representational capacity of GRU-based recommendation models, particularly in capturing complex item dependencies across the entire sequence.

2.3. Linear Attention Mechanism

Attention mechanism as a powerful core of the transformer structure, exhibits performance in learning sequence interactions in recommendation tasks. However, the high computational cost of dot product between query matrix $\boldsymbol{Q}$ and key matrix $\boldsymbol{K}$ substantially lower the inference speed of transformer-based SRSs especially when the sequence length $N$ is much larger than hidden size $d$ (Shen et al., 2021; Wang et al., 2020). To tackle this issue, the linear attention mechanism  (Liu et al., 2023a) designs an special map** function to change the order of the dot product and reduce the computational complexity to $\mathcal{O}(Nd^{2})$. The linear attention mechanism can be written as:

(2) $$\boldsymbol{A}^{\prime}(\boldsymbol{Q,K,V})=\mathcal{X}_{1}\left(elu(\boldsymbol{Q})\right)\left(\mathcal{X}_{2}\left(elu(\boldsymbol{K})\right)^{\mathrm{T}}\boldsymbol{V}\right),$$

where $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$ are row-wise and column-wise $L_{2}$ normalization map**s, respectively, $\boldsymbol{Q,K,V}$ are learnable query, key and value matrices and $\boldsymbol{A}^{\prime}$ is the output attention score. This approach mitigates the issue that the softmax layer concentrates on the scores of merely a few positions, enlarging the information capacity of attention mechanism  (Liu et al., 2023a). By implementing linear attention, our GLINT-RU framework is capable of learning interactions between items in long sequences.

3. METHODOLOGY

Figure 1. Framework of proposed GLINT-RU. The core part of this framework is the Dense Selective GRU, where hidden states are deeply selected and aggregated. Gate mechanisms including expert mixing gate, dense selective gate, and MLP gate are all employed in appropriate positions to deeply filter information.

3.1. Framework Overview

Many existing recommendation frameworks depend on transformer structure  (Sun et al., 2019; Kang and McAuley, 2018; Liu et al., 2023a), which incurs substantial computational overhead and low inference speed. Restricted by the large computational complexity of stacked transformers, linear attention-based models have approached a plateau in terms of minimizing inference time and resource consumption. Uniquely, we propose an advanced recommendation framework that integrates the linear attention mechanism and efficient dense selective GRU module, which further reduce the computational cost compared with stacked linear transformers and SSM-based models. Additionally, this dense selective GRU module also enables our framework to understand complex long-/short-term user behaviors, and substantially reduce the computational cost and inference time. Figure 1 shows the structure for our GLINT-RU framework.

GLINT-RU framework integrates an expert mixing block for mixing sequential information from the dense selective GRU expert and the linear attention expert, and a gated MLP block for further learning and filtering complex user behaviors.

In the expert mixing block, the dense selective GRU module is employed to capture the long-term dependencies and local connections among the items, and selectively learn the sequential information. In addition, the linear attention expert is responsible for modeling item interactions from the user. By combining these two powerful expert modules, our GLINT-RU is capable of adaptively learning dependencies and relevant items from the sequence, which gains deeper insights into the complex user behaviors.

After the user-item interactions are selectively learned by mixing block, the item scores are conveyed to the gated MLP block, where the information is filtered according to the data environment. The framework employs various gates in appropriate positions to deeply filter information, improving the model’s flexibility and the ability to perceive and select information.

3.2. Item Embedding Layer

For sequential recommendation tasks, information on items interacted by users should be encoded to tensors through the embedding layer  (Zhao et al., 2023). We denote the length of input user-item interactions as $N$, and embedding size as $d$. For a interaction sequence $s_{i}=[v_{1},v_{2},\dots,v_{n},\dots v_{n_{i}}]$, the $n$-th item $v_{n}\in\mathbb{R}^{D_{n}}$ can be projected into the representation $\boldsymbol{e}_{n}$ by the following formulation:

(3) $$\boldsymbol{e}_{n}=\boldsymbol{W}_{n}v_{n},$$

where $\boldsymbol{W}_{n}\in\mathbb{R}^{d\times D_{n}}$ is trainable weighted matrix. The embedding layer outputs the encoded item sequence in a tensor:

(4) $$\boldsymbol{E}=[\boldsymbol{e}_{1},\boldsymbol{e}_{2},\cdots,\boldsymbol{e}_{N}]^{\mathrm{T}}.$$

In traditional transformer-based models, positional embeddings are typically necessary because the attention mechanism lacks the inherent capability to encode temporal information  (Kang and McAuley, 2018). Uniquely, in this paper we employ the GRU module to capture sequential behaviors, which has already taken temporal information into consideration. Therefore, we decide not to add the positional embedding layer into the framework.

3.3. Dense Selective GRU

Existing GRU cell learns sequential data by conveying information from preceding cells. Although this mechanism has the superiority of capturing the long-term dependencies in the sequence, it predominantly focuses on information from previous items, while potentially overlooking the valuable context information provided by adjacent items, which are often closely related in real-world applications. To address these challenges and extract local temporal features, we introduce the dense selective GRU as the core component of the GLINT-RU framework. This innovation aims to enhance the model’s flexibility and ability to capture complex long-/short-term patterns in user behavior. By implementing dense selective GRU, the computational complexity can be further reduced, and the recommendation accuracy of GRU-based framework can be substantially improved.

3.3.1. Dense GRU module

Therefore, to enable each GRU cell to be exposed to local temporal features of user behaviour, we introduce a temporal convolution layer, where adjacent item information is adaptively fused before being fed into GRU module:

(5) $$\boldsymbol{L} =\boldsymbol{X}\boldsymbol{W}_{0}+\boldsymbol{b}_{0},$$

$$\boldsymbol{C} =\mathrm{TemporalConv1d}(\boldsymbol{L})$$

where $\boldsymbol{X}=[x_{1},x_{2},\dots,\boldsymbol{x}_{N}]^{\mathrm{T}}$ is the input tensor with $d$ feature dimensions, $\boldsymbol{L}$ is the output of the linear layer, and $\boldsymbol{C}=[\boldsymbol{c}_{1},\boldsymbol{c}_{2},\dots,\\ \boldsymbol{c}_{N}]^{\mathrm{T}}$ is the output of the convolution operation $\mathrm{TemporalConv1d(\cdot)}$ with $N$ steps. $\boldsymbol{W}_{0}$ and $\boldsymbol{b}_{0}$ are weight matrix and bias, respectively. The size of the convolution kernel is set as $k$. According to Eq.(1), the GRU cell generates the output based on the input and the hidden state at the previous time step:

(6) $$\boldsymbol{h}_{n}=\mathrm{GRUCell}(\boldsymbol{c}_{n},\boldsymbol{h}_{n-1}),$$

where $\mathrm{GRUCell}(\cdot)$ is the basic unit in Eq.(1) that processes a current hidden state. Then we generate new features by linearly combining the outputs of the GRU shown in Figure 1, capturing complex patterns of user preferences. These novel feature representations can be more effectively utilized for predicting behaviors and preferences of users. The feature-crossing process can be written as:

(7) $$\Phi(\boldsymbol{H})=\boldsymbol{H}\boldsymbol{W}_{H}+\boldsymbol{b}_{H},$$

where $\boldsymbol{H}=[\boldsymbol{h}_{1},\boldsymbol{h}_{2},\dots,\boldsymbol{h}_{N}]^{\mathrm{T}}$ is the output of GRU, $\boldsymbol{W}_{H}$ is the learnable weight matrix and $b_{H}$ are bias. Although each input state $\boldsymbol{c}_{t}$ incorporates information from both preceding and subsequent items, each output hidden state is still determined by the hidden state of the preceding time step. Therefore, to capture the context information of the output sequential hidden states, we implement a temporal convolution on the crossed features. This convolution layer extracts local temporal features to understand user behavior dynamics, and enhance the predictive accuracy of our model:

(8) $$\boldsymbol{Y}=\mathrm{TemporalConv1d}(\mathcal{G})$$

where $\mathcal{G}$ is the Selective Gate function, and $\boldsymbol{Y}=[\boldsymbol{y}_{1},\boldsymbol{y}_{2},\dots,\boldsymbol{y}_{N}]^{\mathrm{T}}$ is the output matrix from the dense selective GRU module . The two convolution layers together with GRU cells improve the density of the sequential information, enabling each hidden state to be learned from behaviors of more input time steps.

3.3.2. Selective Gate

To filter the hidden information of the GRU module, we design a selective gate where outputs of the feature crossing layer are selected based on the input state of the GRU. The selective gate weights are generated by a two tiny linear layers with SiLU activation function  (Elfwing et al., 2018), and we use them to select useful hidden states and filter information:

(9) $$\boldsymbol{\delta}_{1}(\boldsymbol{C}) =\boldsymbol{C}\boldsymbol{W}_{\delta}^{(1)}+b_{\delta}^{(1)},$$

$$\mathcal{G}(\boldsymbol{H})=\boldsymbol{\Omega}(\boldsymbol{\delta}_{1}(\boldsymbol{C}),\boldsymbol{H}) =(\mathrm{SiLU}(\boldsymbol{\delta}_{1}(\boldsymbol{C}))\boldsymbol{W}_{\Omega}^{(1)}+\boldsymbol{b}_{\Omega}^{(1)})\otimes\Phi(\boldsymbol{H}),$$

where $\boldsymbol{C}=[\boldsymbol{c}_{1},\boldsymbol{c}_{2},\dots,\boldsymbol{c}_{N}]^{\mathrm{T}}$ also serves as the input of the GRU module, $\boldsymbol{W}_{\delta}^{(1)}$ , $\boldsymbol{W}_{\Omega}^{(1)}$ are weight matrices, $\boldsymbol{b}_{\delta}^{(1)},\boldsymbol{b}_{\Omega}^{(1)}$ are bias. With this gate mechanism, our GRU-based model could become more flexible with the ability to perceive the data environment.

3.4. Expert Mixing Block

Although a single GRU module can capture long-term dependencies for recommendation tasks, it lacks the ability to model global connections within the item sequence. By introducing the linear attention mechanism and mixing these two powerful experts, the computational complexity will be substantially reduced compared with transformer-based models, and more information about item connections and dependencies will be considered. Moreover, the two employed experts are parallel in our framework, which further improves the model efficiency.

In real applications, the conditions of the data might vary a lot. The GRU is naturally suited for sequential data, demonstrating effective performance in sequential recommendation tasks that exhibit strong temporal dependencies, while attention focuses on relevant items in the sequence dynamically. To adapt to complex data conditions, we attribute appropriate weights to the two experts by a mixing gate:

(10) $$\boldsymbol{M} =\alpha_{1}^{(t)}\boldsymbol{A}^{\prime}(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V})+\alpha_{2}^{(t)}\boldsymbol{Y},$$

$$\alpha_{i}^{(t)} =\mathrm{softmax}(\alpha_{i}^{(t-1)})=\frac{\alpha_{i}^{(t-1)}}{e^{\alpha_{1}^{(t-1)}}+e^{\alpha_{2}^{(t-1)}}},\ \ i=1,2,$$

where $\alpha_{1}^{(t)},\alpha_{2}^{(t)}$ are trainable mixing parameters at $t$-th training iteration. Then we filter this output by introducing another data-aware gate which selects the outputs based on the original input data batch:

(11) $$\boldsymbol{\delta}_{2}(\boldsymbol{X}) =\boldsymbol{X}\boldsymbol{W}_{\delta}^{(2)}+b_{\delta}^{(2)},$$

$$\boldsymbol{Z}=\boldsymbol{\Omega}_{2}(\boldsymbol{\delta}_{2}(\boldsymbol{X}),\boldsymbol{M}) =(\mathrm{GeLU}(\boldsymbol{\delta}_{2}(\boldsymbol{X})))\otimes\boldsymbol{M},$$

where $\boldsymbol{X}$ is the input of expert mixing block, $\boldsymbol{W}_{\delta}^{(2)}$ is the weight matrix of linear layer, $\boldsymbol{b}_{\delta}^{(2)}$ are bias.

3.5. Gated MLP Block

Most existing frameworks, especially transformer-based models, utilize a two-layer feed forward network to capture the nonlinear relationships among features before giving predictions. To further enhance the performance of the model and augment useful features from the expert mixing block, we introduce the gated MLP block, which employs a gate mechanism again to deeply filter the information and generate item representations for predictions.

(12) $$\boldsymbol{\delta}_{3}(\boldsymbol{Z}) =\boldsymbol{Z}\boldsymbol{W}_{\delta}^{(3)}+b_{\delta}^{(3)},$$

$$\boldsymbol{P}=\boldsymbol{\Omega}_{2}(\boldsymbol{\delta}_{3}(\boldsymbol{Z}),\boldsymbol{Z}) =(\mathrm{GeLU}(\boldsymbol{\delta}_{3}(\boldsymbol{Z})))\otimes\boldsymbol{Z},$$

$$\boldsymbol{R} =\boldsymbol{P}\boldsymbol{W}_{o}+\boldsymbol{b}_{o}$$

where $\boldsymbol{Z}$ is the output of expert mixing block, $\boldsymbol{P}$ denotes the output of gated linear layer, $\boldsymbol{R}$ represents the item representation, and $\boldsymbol{W}_{\delta}^{(3)},\boldsymbol{W}_{o}$ are weight matrices, $\boldsymbol{b}_{\delta}^{(3)},\boldsymbol{b}_{o}$ are bias. The recommendation scores are generated by item representations and embeddings, followed by final prediction probability $\hat{y}_{i}$ of the $i$-th item:

(13) $$\hat{y}_{i}=\mathrm{softmax}(\boldsymbol{R}_{i}(\boldsymbol{e}_{i})^{\mathrm{T}}),$$

where $\boldsymbol{R}_{i}$ is the representation of $i$-th item. Finally, our GLINT-RU give top-$k$ items with the highest scores for recommendation tasks.

3.6. Complexity Analysis

In this subsection, we will analyze the time complexity of our GLINT-RU, and illustrate why GLINT-RU has inherent superiority over other popular SRS models in model efficiency.

Assuming that the sequence length is set as $N$, the embedding size is $d$ and the kernel sizes for GLINT-RU is $k_{1}$, the time complexity of GLINT-RU is $\mathcal{O}((2k_{1}+12)Nd^{2})$. The complexity is calculated throughout the network, from the embedding layer to the prediction layer. GLINT-RU is more efficient than other models in the following aspects: Firstly, our model shows significantly low time complexity, as GLINT-RU is a highly paralleled mixed network with only one layer to achieve high performance. Our framework utilizes paralleled expert networks, and employs efficient GRU module to capture long-term dependencies, outperforming all the other models in theoretical complexity. Secondly, traditional transformer-based model SASRec  (Kang and McAuley, 2018) suffer from large computational complexity, especially When the sequence length $N$ is large. LinRec  (Liu et al., 2023a) changes the dot product of attention mechanism, substantially reduce the computational cost. However, Linrec still requires stacked transformers to achieve high performance. These transformer-based models require at least two layers to achieve satisfactory accuracy, while our GLINT-RU achieves outstanding performance with only one layer. Thirdly, Mamba4Rec utilizes a hardware-aware mechanism to accelerate the training and inference speed  (Gu and Dao, 2023), but its improvement in recommendation performance and efficiency is not stable, which will be further verified via our experiments.

4. EXPERIMENTS

In this section, we conduct extensive experiments to show the effectiveness and efficiency of our GLINT-RU Framework. After we introduce our implementation detail, the experiment results will be analyzed in details. The experiments in this section are set to answer the following research questions:

• RQ1: How does GLINT-RU framework perform compared with other state-of-the-art SRS baseline models?

• RQ2: To what extent does GLINT-RU improve model efficiency compared with other state-of-the-art SRS frameworks?

• RQ3: How do dense selective GRU and linear attention mechanism contribute to GLINT-RU?

• RQ4: How does the hyperparameter setting affect the performance of GLINT-RU?

4.1. Datasets and Evaluation Metrics

We evaluate GLINT-RU based on three benchmark datasets ML-1M  , Amazon-Beauty and Amazon video Games .

Statistical information of the three datasets is shown in Table 1. We adopt Recall, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG) as the evaluation metrics for our experiments. The interactions are grouped by users chronologically, and the datasets are split by the leave-one-out strategy (Liu et al., 2023a).

4.2. Baselines

To demonstrate the effectiveness and efficiency of GLINT-RU, we make comparisons between GLINT-RU and state-of-the-art effective and efficient RNN-, attention-, MLP- and SSM-based baselines.

Traditional Models (1) GRU4Rec (Hidasi et al., 2015): utilizes GRUs to capture sequential dependencies within user interaction data for session-based recommendations. (2) BERT4Rec  (Sun et al., 2019): adapts the Bidirectional Encoder Representations from Transformers (BERT) architecture to model user behaviors for personalized recommendation. (3) SASRec (Kang and McAuley, 2018): captures long-term and short-term user preferences by applying a multi-head attention mechanism to learn representations of user interaction sequences.

Efficient Models (1) LinRec (Liu et al., 2023a): reduces the computational costs substantially by changing the dot product of attention mechanism in the transformer-based models. We select SASRec as the backbone of LinRec. (2) SMLP4Rec  (Gao et al., 2024) uses a tri-directional fusion scheme to learn correlations on sequence, channel and feature dimensions efficiently. (3) Mamba4Rec  (Liu et al., 2024): explores the potential of selective SSMs for efficient sequential recommendation.

Table 1. Statistical Information of Adopted Datasets.

Datasets	# Users	# Items	# Interactions	Avg.Length	Sparsity
ML-1M	6041	3707	1,000,209	165.60	95.53%
Beauty	22,364	12,102	198,502	8.88	99.93%
Video Games	24,304	10,673	231,780	9.54	99.91%

Table 2. Overall performance comparison between GLINT-RU and baselines.

Models	ML-1M			Amazon-Beauty			Amazon-Video-Games
	Recall@10	MRR@10	NDCG@10	Recall@10	MRR@10	NDCG@10	Recall@10	MRR@10	NDCG@10
GRU4Rec	0.6954	0.4055	0.4748	0.3851	0.1891	0.2351	0.6028	0.2929	0.3660
BERT4Rec	0.7119	0.4041	0.4776	0.3478	0.1584	0.2027	0.5490	0.2541	0.2916
SASRec	0.7205	0.4251	0.4958	0.4332	0.2325	0.2798	0.6459	0.3404	0.4128
LinRec	0.7184	0.4316	0.5002	0.4270	0.2314	0.2775	0.6384	0.3355	0.4073
SMLP4Rec	0.6753	0.3870	0.4558	0.4457	0.2408	0.2891	0.6480	0.3484	0.4195
Mamba4Rec	0.7238	0.4368	0.5054	0.4233	0.2213	0.2689	0.6488	0.3389	0.4123
GLINT-RU	0.7379∗	0.4517∗	0.5202∗	0.4472∗	0.2498∗	0.2964∗	0.6573∗	0.3549∗	0.4266∗
Improv.	1.95%	3.30%	2.93%	0.34%	3.74%	2.53%	1.31%	1.87%	1.69%

• Recommendation performance of GLINT-RU and existing state-of-the-art benchmark SRS models have been shown. $``\ast"$ indicates the improvements are statistically significant (i.e., two-sided t-test with $p$ ¡ 0.05) over baselines). The best results are bolded, and the second-best are underlined.

4.3. Implementation

In this subsection, we introduce the implementation details of the GLINT-RU. We use Adam optimizer  (Kingma and Ba, 2014) with the learning rate 0.001 for our training process. Both the train and evaluation batch size are set as 2048. The hidden size is set as 128 for ML-1M, and 64 for Amazon Beauty and Video Games. As is shown in Table 1, the average length of ML-1M , Beauty and Video Games are 165, 8.88 and 9.54, so we set the maximum sequence length as 200 for ML-1M, and 100 for the two amazon datasets. We adopt the dropout rate 0.5 for amazon datasets considering their high level of sparsity, compared with 0.2 for ML-1M. Other implementation details follow the settings of original papers (Liu et al., 2024; Gao et al., 2024; Liu et al., 2023a).

4.4. Overall Performance (RQ1)

In this subsection, we compare the performance of GLINT-RU with both traditional recommendation frameworks and state-of-the-art efficient models. The results, as shown in Table 2, demonstrate the effectiveness of GLINT-RU on metrics Recall@10, MRR@10, and NDCG@10 in bold. According to the above table, it is evident that GLINT-RU defeats all the selected transformer-, RNN-, MLP- and SSM-based baselines. We improve the performance upper bound of efficient recommendation models by $0.34\%\sim 3.74\%$.

Traditional RNN-based models, like GRU4Rec, might have difficulty in dealing with complex user behaviors. Although it can capture the long-term dependencies from long sequences, it struggles to learn effectively from extremely sparse datasets. Traditional attention-based methods like SASRec have great performance on the three datasets, but it is quite inefficient due to the high computational complexity of the attention mechanism.

Efficient model LinRec changes the softmax operation, and takes attention scores from more positions into consideration, improving the performance on long-term sequential recommendation tasks compared with its backbone SASRec. Both SMLP4Rec and Mamba4Rec achieve impressive performance on the three datasets. However, Mamba4Rec demonstrates enhanced proficiency in modeling long-term sequence, but exhibits low performance in relatively short sequences. Conversely, the SMLP4Rec shows superior performance in short sequence recommendations while being less effective with longer sequences. In addition, SMLP4Rec requires additional features to enhance its performance. Uniquely, GLINT-RU integrates the advantages of linear attention mechanism and dense selective GRU module, adaptively extracting dependencies and most relevant behaviors from users. The gate mechanism employed in GLINT-RU substantially enhances its ability to filter the information based on the dynamic data environment and mix the experts according to the data adaptively.

In summary, GLINT-RU as a novel efficient framework, shows its superiority over state-of-the-art baselines. This underscores the potential of dense selective GRU and models with hybrid modules as more powerful tools for recommendation tasks.

Table 3. Efficiency comparison.

Datasets	Model	Infer.	Training	GPU Memory
ML-1M	GLINT-RU	31ms	86s/epoch	8.81G
	Mamba4Rec	41ms	108s/epoch	7.72G
	SASRec	55ms	172s/epoch	21.51GB
	BERT4Rec	88ms	285s/epoch	21.51GB
	LinRec	37ms	101s/epoch	11.67GB
	SMLP4REc	51ms	151s/epoch	16.13GB
	Improv.	16.22%	17.44%	-
Beauty	GLINT-RU	278ms	3.8s/epoch	2.62G
	Mamba4Rec	351ms	4.5s/epoch	2.32G
	SASRec	444ms	8.1s/epoch	7.67GB
	BERT4Rec	1372ms	13s/epoch	11.69GB
	LinRec	340ms	5.6s/epoch	4.14GB
	SMLP4Rec	361ms	5.3s/epoch	2.95GB
	Improv.	18.24%	15.56%	-
Video Games	GLINT-RU	247ms	4.5s/epoch	2.49G
	Mamba4Rec	309ms	5.6s/epoch	2.28G
	SASRec	406ms	9.6s/epoch	7.65GB
	BERT4Rec	1290ms	15s/epoch	10.98GB
	LinRec	327ms	6.6s/epoch	4.13GB
	SMLP4Rec	389ms	9.1s/epoch	3.38GB
	Improv.	20.06%	19.64%	-

• Inference time (ms) of each mini-batch, training time (s/epoch) and GPU memory (GB) of GLINT-RU and other baseline models. The best results are bolded, and the second best results are underlined.

4.5. Efficiency Comparison (RQ2)

In this subsection, we analyze the efficiency of GLINT-RU and state-of-the-art sequential recommendation models. We evaluate the model efficiency according to their inference time of each mini-batch, training time and GPU memory occupation.

The results, shown in Table 3, provide several valuable insights: Firstly, by utilizing the efficient Dense GRU module and linear attention module, GLINT-RU dramatically reduces the training time and inference time, improving the training inference time by $15\%\sim 20\%$ compared with most efficient recommendation baseline models. In addition, due to the low computational cost of GRU and linear attention mechanism, GLINT-RU exhibits minimal GPU memory consumption, which is comparable to the state-of-the-art SSM-based efficient model Mamba4Rec. In Section 3.6, we demonstrate that the GLINT-RU exhibits low theoretical computational complexity, as we employs the parallel networks and efficient GRU as core component for our recommender system. The theoretical analysis has been verified by the results in Table 3.

Traditional transformer-based recommendation models like SASRec and BERT4Rec suffer from extended inference time and high GPU memory occupation. When processing long sequential data, the conventional attention mechanism falls behind novel efficient models due to its high computational cost. Among all the baseline models, the SSM-based Mamba4Rec framework exhibits impressive efficiency, but Mamba requires complex mathematical computation, which slows down its inference and training speed. Additionally, LinRec suffers from the inherent shortage of its backbone SASRec that requires stacked transformer layers to enhance the model performance. Such large transformer structures extend both the inference and training time. Although SMLP4Rec achieves high performance in the model accuracy, it struggles to train and inference efficiently, especially when processing long-term sequential data.

4.6. Ablation Study (RQ3)

In this subsection, we will analyze the efficacy of essential components in GLINT-RU architecture. We conduct the ablation study on the ML-1M dataset, and the results are outlined in Table 4, providing insightful observations.

The results verify the essential role of the dense selective GRU module, as the performance of the model will dramatically decrease without the GRU module. This reveals the insights that the gated GRU module effectively capture the long-term dependencies of user-item interactions. Linear attention mechanism supplements the information of relevant items in the sequence. As is shown in Table 4, it improves the performance of GLINT-RU to some extent. Adding a temporal convolution layer incorporates context information from adjacent items, resulting in an enhancement in model performance. In addition, the gated MLP block plays a similar role as the feed-forward network in our framework, which filters complex information from the expert mixing block. It is noteworthy that even without the gated MLP block our framework still outperforms all the state-of-the art efficient models, demonstrating its inherent remarkable superiority for sequential recommendation tasks. After we remove the gated MLP, the GPU memory occupation of GLINT-RU becomes 7.63GB, less than Mamba4Rec shown in Table 3, and the inference time will be reduced to 241ms. We name this framework as ”Light GLINT-RU”, which is more applicable to resource constrained scenarios.

Table 4. Ablation study for Components of GLINT-RU.

Model Components	Recall@10	MRR@10	NDCG@10
Default	0.7379∗	0.4517∗	0.5202∗
w/o Gated MLP (Light GLINT-RU)	0.7260	0.4369	0.5060
w/o Attention	0.7195	0.4312	0.5001
w/o GRU	0.6762	0.3913	0.4593
w/o Temporal Conv1d	0.7232	0.4322	0.5019

• $``\ast"$ indicates the improvements are statistically significant (i.e., two-sided t-test with $p$ ¡ 0.05) over baselines)

4.7. Parameter Analysis

In this subsection, we analyze the impact of the crucial hyperparameter kernel size $k$ in GLINT-RU, which determines the amount of information of each hidden state. We discuss the performance and efficiency of GLINT-RU with varied $k$, providing valuable insights of the stability and robustness of GLINT-RU.

We conduct the parameter analysis on the dataset Amazon Beauty, and the result is shown in Figure 3.

Figure 2. Impacts of kernel size $k$ of the temporal convolution on the performance of GLINT-RU and Mamba4Rec and the GPU Memory occupation of GLINT-RU.

Figure 3. Impacts of kernel size $k$ of the temporal convolution on the training and inference time.

The first three subfigures display the model performance of GLINT-RU with different kernel sizes. Kernel sizes that are larger than 1 enable the aggregation of the information from adjacent items in each hidden state, enhancing the model performance. Additionally, our model exhibits stable and high performance, providing a wide range of choices of kernel sizes. This finding indicates the robustness of GLINT-RU framework. The accuracy of the recommendation system is observed to be positively correlated with the kernel size $k$. This enhancement in performance can be attributed to the fact that a larger kernel size aggregates information from more items, thereby learning a more extensive context into the hidden state. However, as the kernel size continues to expand, dense selective GRU might incorporates irrelevant data into the output state, which might lead to a marginal decline in the accuracy. Mamba4Rec as a novel efficient SSM-based model, also employs a temporal convolution layer in the model structure. However, as is shown in Figure 3, as the kernel size changes, the performance of Mamba4Rec becomes quite unstable compared with our GLINT-RU. Furthermore, our GLINT-RU consistently outperforms Mamba4Rec across all kernel sizes, further demonstrating the stability and superiority of this novel GLINT-RU framework.

The hist plot in Figure 3 shows the GPU memory occupation with different kernel sizes. Setting different kernel sizes for the model does not significantly increase GPU utilization, indicating that the appropriate kernel size can be freely chosen based on accuracy requirements. Moreover, increasing the kernel size has slight impacts on the training and inference time of each mini-batch, which further verifies the efficiency and stability of our model.

5. Related Works

Traditional Sequential Recommendation Models Transformers, RNNs are popular effective frameworks for sequential recommendation. Various studies have been conducted to leverage transformers for recommendation tasks. Transformer-based SASRec  (Kang and McAuley, 2018) makes predictions based on multi-head attention within long-/short-term interactions. SSE-PT (Wu et al., 2020) improves the performance by adding user embeddings to the transformer. MB-STR (Yuan et al., 2022) is also a variant of transformer-based model, which captures diverse user behavior dynamics, and alleviates the impacts from data sparsity problems. RNN-based GRU4Rec  (Hidasi et al., 2015) learns dependencies among the items using gated recurrent units. HRNN (Quadrana et al., 2017) as an advanced RNN-based model adds an additional GRU layer to learn information from user sessions and instantly track the user preferences.

Efficient Recommendation Models To tackle the high computational complexity of effective transformer-based SRSs, and accelerate the inference speed for real-world applications, researchers endeavor to invent increasingly effcient models. AutoSeqRec  (Liu et al., 2023b) is constructed based on Autoencoder, which is a innovative work that captures long-term preferences through collaborative filtering. To further improve the model efficiency, DMAN (Tan et al., 2021) combines the long-term attention net and recurrent attention net to memorize users’ interests dynamically and support efficient inference. LinRec  (Liu et al., 2023a) cuts down the computational cost of transformer-based backbones by changing the dot product in the attention mechanism. LightSAN  (Fan et al., 2021) projects the initial interactions into representations with shorter length, which is also an efficient approach for transformer-based models. SSM-based models like Mamba4Rec (Liu et al., 2024) and RecMamba (Yang et al., 2024) utilizes selective state space models to achieve high performances and high efficiency, becoming emerging powerful tools for sequential recommendation tasks. FMLP-Rec (Zhou et al., 2022) with learnable filters and SMLP4Rec (Gao et al., 2024) with diverse mixers are representative pure MLP-based efficient SRSs. LRURec (Yue et al., 2024) is constructed based on linear recurrent units, and achieves fast inference bu employing recursive parallelization.

6. Conclusion

In this paper, we have presented an innovative dense selective GRU framework GLINT-RU for sequential recommendation tasks. Due to the parallel netork design and the implementation of efficient dense selective GRU, the computational cost can be substantially reduced, thereby resulting in faster inference speed compared with state-of-the-art efficient models. Additionally, our model GLINT-RU tackles the issue of low accuracy of traditional GRU frameworks, and represents a significant advancement in the efficient recommendation models. GLINT-RU applies gate mechanisms to every module, deeply filtering and selecting the information according to the data environment. Dense GRU allows each hidden state to contain the information from the adjacent items, which captures local temporal features of user behavior to enhance the predictive accuracy. Our extensive experiments demonstrate that GLINT-RU achieves outstanding performance, not only improving the model accuracy, but also accelerating the training and inference speed dramatically. These results underscore our GLINT-RU’s potential to become a novel, stable, and efficient framework on datasets with various sparsity and sequence length. As an efficient and accurate recommender system that defeats the state-of-the-art efficient RNN-, transformer-, MLP- and SSM-based recommendation models, we believe that our novel framework will become a valuable foundation for research in more domains in recommender systems.