Empowering 3D Visual Grounding with Reasoning Capabilities

measures models’ understanding of 3D spatial relationships among objects in a 3D scene. It encompasses the ability to comprehend the layout and structure of the 3D scene, the 3D location of objects within it, which could serve as the foundation of navigating or planning movements in the 3D environment.

involves understanding and inferring the purpose, function, or affordance of objects within the 3D scene. For example, function reasoning allows an embodied agent to recognize that a chair is for sitting, a lamp is for lighting, and a refrigerator is for storing food at low temperatures. Such understanding enables the embodied agent to assist users and to perform complex tasks more effectively (e.g., turning on a lamp when the room gets dark, or navigating to a refrigerator to fetch a drink).

allows an embodied agent to not only understand its environment but also to interact with it in a goal-directed manner. For example, given a question shown in Fig. 2: “If I’m cooking dinner in the kitchen, where is the nearest place for me to throw the rubbish?”, an agent need to use such reasoning ability to infer the location of objects (in this question, a rubbish bin) based on their function and spatial relationships under the specific setting (the kitchen).

plays a critical role in human-robot interaction, where the target objects are determined by understanding human emotions, preferences, and behavioral patterns. This ability makes the embodied agents more attuned to the emotional and psychological states of humans, allowing them to provide more personalized, empathetic, and contextually appropriate responses and solutions, such as: “I’m very sad, is there something can make me feel happy?” shown in 2.

focuses on preventing harm and ensuring the well-being of humans and other creatures in the 3D environment. It requires the embodied agent to identify and assess the risk and make safety-aware decisions, such as: “Where should first aid kits be placed for easy access but out of reach of children?”.

Unlike the 2D image encoder used for 3D-LLM, the 3D visual encoder directly extracts features from 3D point clouds to capture more geometric and spatial information about the 3D structures and fine-grained details. The powerful 3D visual encoder which captures comprehensive geometry, structure, layout, and fine-grained information is critical to accurate 3D grounding. Subsequently, the 3D features $f_{scene}$ produced by the 3D visual encoder and the grounding feature $h_{loc}$ from 3D-LLM are sent to the query selection module.

We adopt a cross-attention mechanism, where we treat $f_{scene}$ as $Q$ (Query), and $h_{loc}$ as both $K$ (Key) and $V$ (Value), to implicitly obtain a feature-level reasoning activation heatmap. During this scene look-back process, this module roughly locates scene features that have a high response to the <LOC> token. We then select the $k$ most relevant features as the object query $f_{query}$.

is a classical transformer decoder, which consists of M transformer decoder layers, In each decoder layer, the object queries $f_{query}$ go through the text feature $h_{loc}$ cross-attention layer and scene feature $f_{scene}$ cross-attention layer. Finally, the prediction head takes the updated object queries as input and predicts the final 3D locations and matching score.

In comparison to previous works [10, 14, 20, 12] that directly predicts the bounding boxes by the LLM, the extra grounding module has the following advantages: 1) Based on the MLLM reasoning results, it allows the grounding module to perceive the scene again and focus on the region under the implicit guidance of MLLM, adapted to the user queries and the reasoning results. 2) The scene representation perceived in the grounding module can be more precise and fine-grained, which is complementary to the visual-centric reasoning module. 3) The two-step reasoning-grounding pipeline is flexible and can generalized to other types of predicting formats such as segmentation masks (by simply replacing the 3D box decoder with a 3D mask decoder).

We use the pretrained weights of 3D-LLM as the initialization for the visual-centric reasoning module. Except for freezing the 3D visual encoder pretrained on [36], the rest of parameters in the visual-centric reasoning module and 3D grounding module in our framework are trained in an end-to-end manner. The training supervision is a weighted sum of the next token prediction loss from 3D-LLM and the 3D detection loss from the 3D grounding module.

The 3D detection loss is defined following:

We load the pretrained weights of 3D-LLM and the 3D visual encoder, and finetunes the LoRA of 3D-LLM, the query selection module, and the 3D box decoder with an instruction tuning dataset. To construct the instruction tuning dataset, we reformulate the data annotations from existing 3D datasets into question-answer or question-answer-bounding-box pairs. Specifically, the 3D visual grounding data from ScanRefer [6], SR3D, NR3D [1] and the 3D object detection data from EmbodiedScan [36] are formulated into question-answer-bbox pairs, and the 3D/spatial question answering data from SR3D [1], CLEVR3D [40], SQA3D [28] are formulated into question-answer pairs without bounding boxes. The information shown in Tab. 1 illustrates how we unify the instruction and output with task-specific templates. More details can be found in the Appendix. The reformulated data combined with our proposed ScanReason dataset, serve as the instruction tuning dataset.

For the 3D grounding module, we adopt the pre-trained multi-modal 3D encoder in EmbodiedScan as the 3D visual encoder. During the trainning stage, We use LoRA to effeciently finetune the 3D-LLM to preserve the original 3D scene understanding capability and reduce the computation costs. The number of object queries $k$ in the query selection module is set to be 256.

The training is done on 8 NVIDIA A100 GPUs. We adopt AdamW optimizer with a learning rate of 1e-4 and use a learning rate scheduler WarmupDecayLR with the warmup steps of 100. The total batch size is set to be 16. The loss weight parameters $\lambda_{text}$ and $\lambda_{det}$ in total loss $\mathcal{L}$ are set to 1.0 and 1.0, respectively, and the weight $\lambda_{L1}$, $\lambda_{IOU}$ and $\lambda_{contrast}$ in $\mathcal{L}_{det}$ are set to 1.0, 2.0 and 1.0.

In order to verify the superiority of our model in grounding ability and facilitate comparison between current models, we report the explicit grounding performance on the existing 3D visual grounding task. Since the evaluation settings of Nr3D and Sr3D [1] are based on ground-truth object proposals, while ScanRefer [6] requires models to output 3D bounding boxes, we choose ScanRefer as our benchmark for comparison. We divide the existing methods into two categories, one is the grounding model designed specifically for the 3D visual grounding task, and the other is generalist MLLMs which can understand a variety of 3D vision-language tasks. The original 3D-LLM embed 3D locations in the vocabularies and represent the grounded 3D bounding boxes by a sequence of discrete location tokens. However, since the fine-tuned model of 3D-LLM on ScanRefer and related location tokens implementation are not accessible, we adapt 3D-LLM to directly output 3D numerical coordinates representing 3D bounding boxes by fine-tuning the pre-trained model on our reformulated 3D visual grounding data, denotes as 3D-LLM (vg). As shown in Tab. 2, current generalist MLLM models still lag behind the specialist models in terms of the grounding ability. By incorporating the 3D grounding module into MLLM, ReGround3D shows the SOTA performance on the traditional 3D visual grounding task.

The various visual grounding models rely on explicit text-object alignment in the input object expression to achieve localization, which fall to be applied to 3D reasoning grounding task. We performed a comparision between our proposed method ReGround3D and existing MLLM methods, including 3D-LLM(vg) and Chat3D-v2 [21]. Besides, inspired by Chat-3D v2, which first segments objects, then equips them with unique object identifiers to conduct effective object grounding, we set up a LLM-based 3D reasoning grounding baseline: We first segment the objects from the scene using a 3D instance segmentor [31], then convert the segmented object information including their categories and 3D bounding boxes into text as input of LLM (InternLM2-7B [32]). Besides, to better validate the performance of our model and ensure the fair comparison, we remove the ScanReason dataset from the training data, denoted as ReGround3D*. As show in Tab. 3, the LLM-based reasoning method (Mask3D [31] + InternLM2-7B [32]) possesses a very strong function reasoning ability, but struggles understanding of 3D spatial relationship. Additionally, irrespective of whether ScanReason is used in training, our model significantly outperforms the existing MLLMs. By synergize the reasoning and grounding process utilizing the Chain-of-Grounding (CoG) mechanism, the 3D reasoning grounding performance of ReGround3D can be further improved (from 28.98 to 30.62), especially on the spatial reasoning and logistic reasoning questions. The qualitative results comparison shown in Fig. 5 demonstrates superiority of our method in reasoning human complex instruction based on the 3D scene.

In order to more comprehensively verify the effectiveness of the 3D grounding module we proposed, we step by step verify the performance changes from 3D-LLM to ReGround3D. Apart from the 3D-LLM(vg), we fine-tuned the 3D-LLM model respectively on full reformulated existing data and all instruction tuning data, including ScanReason, denoted as 3D-LLM (full) and 3D-LLM (full+sr). All the <LOC> token in answers of the data used for fine-tuning 3D-LLM have been converted to the numerical coordinates of the corresponding boxes. The results in Tab. 4(a) show that with the same tuning dataset, our ReGround3D achieves far superior performance to 3D-LLM (28.98 vs. 19.21) by introducing the 3D grounding module.

Tab. 4(b) showcases the impact of different training data types on 3D visual grounding performance. 3D Object Detection (3D OD) provides the explicit 3D semantic category and visual alignment whereas 3D Question Answering (3D QA) data injects the basic 3D scene understanding ability into the model, which have certain benefits for the visual grounding ability of the model. In addition, we find that training with 3D reasoning grounding dataset can further improve the performance on 3D visual grounding.

While CoG Mechanism boosts the performance with interleaved reasoning and grounding steps during inference, it uses the relevant objects information explicitly presented in the question to help find the target objects. A natural question arises: if we input the information of all the objects instead of relavant objects into ReGround3D during CoG, will this make the model more accurately find the target objects? We first use the existing 3D Segmentor [31] to segment all objects in the scene, then update the original question using all the object 3D bounding boxes information according to the template in Sec. 4.3 and send into ReGround3D. However, experiments show that using all the object information will instead reduce the 3D reasoning grounding performance from 28.98 to 27.67. One possible reason could be the model’s attention is dispersed by too many irrelavent objects.

Generally speaking, 3D object detection datasets contain information about 3D bounding boxes of all objects in the pre-defined list of categories. In order to cover as many objects as possible, we chose to construct question and answer pairs based on the EmbodiedScan [36] dataset, which includes 160k 3D-oriented boxes spanning over 760 categories. During the model training process, we convert the annotations of 3D boxes into a specific question answering pair template: “User: <scene> Where is the <category> in this 3D Scene? Assistant: Sure, <LOC>.” Here, <category> is randomly selected from the ground-truth categories contained in the current 3D scene, <scene> is the placeholder of 3D scene tokens.

3D visual grounding data aims at localizing the unique object in 3D scenes given the descriptive object expression. The descriptions of objects in these data typically explicitly include the object attributes and their spatial relationships with other objects. To ensure diversity in training data, we selected ScanRefer [6], SR3D [1], NR3D [1] as training data. Considering the variety of object descriptions, it is difficult to simply reformulate the object expression using a simple template like: “ User: <scene> Where is <expr> in the 3D Scene?. Assistant: It is <LOC>.” Therefore, we choose to retain the original object description as much as possible and use a template: “Here is a description about an object: “ <expr> ”, where is the object in the 3D Scene? Assistant: It is <LOC>.”, where <expr> represents the object description in the data. Besides, we have created a range of similar question templates that are randomly selected during the training process.

We hope the model can understand 3D position in a more natural way using numerical values in Natural Language. We use $[x,y,z,dx,dy,dz]$ to represent a 3D box, where $[x,y,z]$ represents the center of a 3D area and $[dx,dy,dz]$ represents the 3D box size, these coordinates can appear anywhere in the input text. Since there are no explicit coordinate question answering pairs in the 3D vision-language datasets, we turned our attention to the SR3D dataset. SR3D is a template-based generated dataset that not only provides expressions of target objects but also provides object ids for target objects and anchor objects. Based on SR3D, we constructed a 3D QA dataset focusing on 3D positional relationships between objects. For example, the query “select the trash can that is beneath the desk” from SR3D dataset can be transformed into “User: Is the trash can $[x_{1},y_{1},z_{1},dx_{1},dy_{1},dz_{1}]$ beneath the desk $[x_{2},y_{2},z_{2},dx_{2},dy_{2},dz_{2}]$?” Assistant: Yes.” with the assistance of GPT-3.5.

Considering that we expect the model can also output reasonable answers in the conversation, we also introduce 3D QA data during the training process to further enhance the model’s 3D visual question answering and scene understanding capabilities. We reformulate CLEVR3D [40] data into a simple question-answer template: “User: <scene> <question>. Assistant: <answer>.” The SQA3D [28] dataset not only provides questions but also provides the situation in which the questions are asked. We reformulate it into a template: “User: <scene> <situation> <question>. Assistant: <answer>.” wherein the <question> the placeholder for the question and <situation> is the placeholder of situation while raising the corresponding question.

In addition to the types of data mentioned above, We also employed our own proposed 3D reasoning grounding data to train the model, further enhancing its capability to handle complex reasoning questions. The output format is more akin to a combination of 3D Question Answering (QA) and localization, where the model’s response not only includes the target object but also provides an explanation for the selection of the target object. We adopted a template: “User: <scene> <question>. Assistant: Sure, <LOC>, <reason>.”