HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Models

Abstract

We introduce HouseCrafter, a novel approach that can lift a floorplan into a complete large 3D indoor scene (e.g., a house). Our key insight is to adapt a 2D diffusion model, which is trained on web-scale images, to generate consistent multi-view color (RGB) and depth (D) images across different locations of the scene. Specifically, the RGB-D images are generated autoregressively in a batch-wise manner along sampled locations based on the floorplan, where previously generated images are used as condition to the diffusion model to produce images at nearby locations. The global floorplan and attention design in the diffusion model ensures the consistency of the generated images, from which a 3D scene can be reconstructed. Through extensive evaluation of the 3D-Front dataset, we demonstrate that HouseCraft can generate high-quality house-scale 3D scenes. Ablation studies also validate the effectiveness of different design choices. We will release our code and model weights. Project page: https://neu-vi.github.io/houseCrafter/

Figure 1: HouseCrafter can lift floorplans to 3D scenes. Top: It adapts a 2D diffusion model to generate multi-view RGB-D images across different locations of the scene in an autoregressive manner. The RGB-D images are then fused into a 3D mesh. Bottom: HouseCrafter can generate high-quality 3D meshes of the scene that are faithful to the input floorplan.

1 Introduction

High-fidelity 3D environments are crucial for delivering truly immersive user experiences in AR, VR, gaming, and beyond. Traditionally, this process has been labor-intensive, demanding meticulous effort from skilled artists and designers, especially for intricate indoor settings with numerous furniture pieces and decorative objects. The development of automated tools for generating realistic 3D scenes can significantly improve this process, streamlining the creation of complex virtual environments, which enables faster iteration cycles and empowers novice users to bring their creative visions to life. Such tools hold immense potential across industries like architecture, interior design, and real estate, facilitating rapid visualization, iteration, and collaborative design.

Recent advances in denoising diffusion models(Ren et al., 2023; Ju et al., 2023) show great promise toward develo** 3D generative models using 3D data. However, in contrast to the abundant availability of 2D imagery(Schuhmann et al., 2022), 3D data requires intensive labor to create or cquire(Dai et al., 2017; Chang et al., 2017; Fu et al., 2021; Ge et al., 2024; Behley et al., 2019; Yeshwanth et al., 2023). Thus, using 2D generative models(Rombach et al., 2022; Saharia et al., 2022) is a promising direction for 3D generation. InSong et al. (2023); Tang et al. (2023), 2D diffusion models are used to texturize a given raw 3D scene. But obtaining the untextured 3D scene, either in the format of mesh or point cloud, is not trivial. Alternatively, a 3D scene can be estimated based on generated multi-view observations(Liu et al., 2023b; Ye et al., 2023; Weng et al., 2023; Liu et al., 2023c; Shi et al., 2023b; a; Long et al., 2023; Liu et al., 2024; 2023a; Kant et al., 2023; Szymanowicz et al., 2023; Kant et al., 2024; Wang et al., 2024; Zheng & Vedaldi, 2023; Hu et al., 2024; Huang et al., 2023; Voleti et al., 2024). However, these works only investigate object-centric generation with relatively simple camera positions.

For 3D scene generation, text-to-image diffusion models are employed to create room panoramas(Song et al., 2023; Tang et al., 2023), offering visually appealing results. But converting these panoramas into 3D representations without additional input is challenging. Other works(Höllein et al., 2023; Chung et al., 2023; Shriram et al., 2024) obtain a 3D representation of the scene by continuously generating 2D images of the scene and projecting them to 3D space using depth provided by monocular depth estimation models(Piccinelli et al., 2024; Ke et al., 2024). While achieving good results on a small scale, these methods struggle to scale up to bigger scenes, which tend to produce repeated content and distorted geometry. Instead of using textual descriptions, layout maps can better convey the global guidance for scene generation. Several studies have explored this approach at the room-scale level, demonstrating the benefits of incorporating layout information (Schult et al., 2023; Fang et al., 2023; Bahmani et al., 2023). However, extending this method to house-scale generation poses challenges, as the current strategy of generating all scene content in one batch becomes impractical for larger, more complex scenes.

In this paper, we present HouseCrafter, an autoregressive pipeline for house-scale 3D scene generation guided by 2D floorplans, as shown in Fig. 1. Our key insight is to adapt a powerful pre-trained 2D diffusion model (Rombach et al., 2021) to generate multi-view consistent RGB and depth (RGB-D) images, decoupling color and geometry, across different places of the scene to reconstruct the 3D house. Specifically, we sample a set of camera poses within the scene based on the given floorplan. A novel view synthesis model is developed to generate RGB-D images at these poses in a batch-wise manner. In each batch, the model takes the already generated RGB-D images at neighboring poses (initially empty) as reference and simultaneously generates RGB-D images at nearby target poses, guided by the local view of the floorplan. With all the generated RGB-D images inside the house, we use the TSDF fusion(Zeng et al., 2017) to reconstruct the scene, providing explicit meshes for downstream applications (e.g., in an AR/VR application).

Our novel-view synthesis model is inspired by the object-centric generation model EscherNet(Kong et al., 2024). Although it shows promising results of ensuring the view consistency with camera position encoding, it is not designed to handle the complexity of geometry and appearances on a scene level, resulting in failure when camera move away from the initial object. We make two important modifications to adapt it for 3D house generation. We first extend it to take 2D floorplan as an additional input, leading to globally consistent scene generation. Second, we incorporate depth information into both the input and output of novel view synthesis, decoupling geometry and appearance of the scene and yielding better generation results. The idea of RGB-D novel view synthesis is also investigated by Hu et al. (2024). However, MVDFusion is not designed for scene generation either and produces low-resolution depth images due to concatenation with the downsampled features of the RGB images in the denoising process. Instead, our model produces high-resolution depth images, leading to high-quality 3D scene reconstruction.

We evaluate our model on the 3D-Front dataset (Fu et al., 2021). Through our experiments, we demonstrate the effectiveness of our RGB-D novel view synthesis model in generating images at the novel views that are consistent not only with the input reference views and floorplan but also among the generated images themselves. Moreover, we demonstrate the model’s efficacy in generating more compelling 3D scenes that are globally coherent than existing methods.

In summary, our key contributions are summarized as follows.

• We introduce a novel method HouseCrafter, which can lift a 2D floorplan into a 3D house. Compared with existing room-scale methods(Höllein et al., 2023; Bahmani et al., 2023), our approach can generate globally consistent house-scale scenes.

• We present a RGB-D novel synthesis method, which takes nearby RGB-D images as reference to generate a set of RGB-D images at novel views, guided by the floorplan. Compared to existing methods(Kong et al., 2024; Hu et al., 2024), our approach generates semantically and geometrically consistent multi-view RGB-D images, enabling high-quality and efficient 3D scene reconstruction.

• Through both quantitative and qualitative evaluation, we demonstrate the effectiveness of our model in producing images that are faithful to both reference images and floorplan. We also show that our approach can generate globally coherent house-scale indoor scenes.

2 Related Work

3D Object Generation. Recent advancements in 2D image generation (Rombach et al., 2021; Blattmann et al., 2023) have inspired attempts to use diffusion models for 3D generation. Some works (Poole et al., 2022; Lin et al., 2023; Yi et al., 2024) optimize 3D representations (Mildenhall et al., 2021; Kerbl et al., 2023) by leveraging the denoising capabilities of diffusion models. However, these models struggle to maintain a single object instance across denoising updates and are unaware of camera poses, limiting the quality of the optimized 3D representations.

Alternatively, some works convert generated images into 3D models (Liu et al., 2023b; Ye et al., 2023; Weng et al., 2023; Liu et al., 2023c; Shi et al., 2023b; a; Long et al., 2023; Liu et al., 2024; 2023a; Kant et al., 2023; Szymanowicz et al., 2023; Kant et al., 2024; Wang et al., 2024; Tochilkin et al., 2024; Zheng & Vedaldi, 2023; Hu et al., 2024; Huang et al., 2023).  Liu et al. (2023b) demonstrated that diffusion models (Rombach et al., 2021) fine-tuned on large-scale object datasets (Deitke et al., 2023; 2024) can generate consistent multi-view RGB images, enabling 3D model reconstruction. Building on this, subsequent research has focused on enhancing multi-view image quality by integrating 3D representations  (Yang et al., 2023; Liu et al., 2023c; Kant et al., 2023; Weng et al., 2023; Shi et al., 2023b; Liu et al., 2024; 2023a; Hu et al., 2024) or using cross-view attention  (Zheng & Vedaldi, 2023; Blattmann et al., 2023; Kong et al., 2024; Shi et al., 2023b; Voleti et al., 2024).

Inspired by these approaches, we aim to generate multi-view images at the scene level. Our model uses multi-view RGB-D images and 2D layout as conditions to generate new multi-view RGB-D images. Integrating depth enhances multi-view consistency and provides explicit scene geometry for 3D reconstruction. Unlike Kong et al. (2024), which only outputs multi-view RGB images, and  Hu et al. (2024), which denoises depth images with RGB latents, our model denoises both RGB and depth images in the latent space. This maintains geometry awareness and produces high-resolution depth images and high-quality 3D reconstructions, ensuring geometric and semantic consistency across views.

Text-guided 3D Scene Generation Text-to-image models can be also utilized for 3D scene generation. Some works  (Rockwell et al., 2021; Zhang et al., 2023; Yu et al., 2023; Chung et al., 2023; Ouyang et al., 2023; Höllein et al., 2023; Shriram et al., 2024) continuously aggregates frames with existing scenes, using monocular depth estimators to project 2D images into 3D space, but faces challenges like scale ambiguity and depth inconsistencies. Recent work improves geometry by training depth-completion models (Engstler et al., 2024). However, most of these methods focus on forward-facing scenes, struggling for larger or more complex scenes like rooms or houses since global plausibility is not guaranteed (Höllein et al., 2023).

To enhance global plausibility, MVDiffusion  (Tang et al., 2023) and Roomdreamer  (Song et al., 2023) generate multiple images in a single batch to form a panorama, though without geometry generation. Gaudi (Bautista et al., 2022), directly generates global 3D scene representation, producing 3D scenes with globally plausible content, but the quality is limited by the scarcity of 3D data with text.

Our pipeline generates views of the scene autoregressively but in batches. Compared to image-by-image generation pipelines (Höllein et al., 2023; Chung et al., 2023; Shriram et al., 2024), batch generation scales better and benefits from the built-in cross-view consistency of multi-view models. Additionally, by including depth images, HouseCrafter addresses scale ambiguity and leverages geometry from previous steps to generate novel views.

Layout-guided 3D Scene Generation. Complimenting to text, the layout provides the detailed position of objects in the scene. Early work (Vidanapathirana et al., 2021) is able to uplift a 2D floorplan to a 3D house model but only focuses on the architectural structure, i.e. floor, wall, ceiling. Also conditioned on 2D layout, BlockFusionWu et al. (2024) achieves commendable results in geometry generation but does not generate texture.

For both geometry and texture generation, Ctrl-Room (Fang et al., 2023) and ControlRoom3D (Schult et al., 2023) show that 3D layout guidance improves geometry and object arrangement compared to text-only methods (Höllein et al., 2023). However, these methods generate a single panorama, limited to room-scale scenes. CC3D (Bahmani et al., 2023), closest to our work, uses 2D layout guidance to produce a 3D neural radiance field, enabling textured mesh but still limited to single-room scenes. Our method effectively uses 2D layout guidance to scale to larger scenes, such as entire houses.

Other works Other approaches treat indoor scene generation as an object layout problem (Wen et al., 2023; Feng et al., 2024; Yang et al., 2024). These works focus on predicting floor layouts and furniture placement using with language model, and retrieving suitable objects from a database. Alternatively, Ge et al. (2024) create augmented layouts from templates, while others use procedure generation (Deitke et al., 2022; Raistrick et al., 2024) These approaches complement our pipeline, as we can use predicted floorplans to generate the scene’s texture and geometry accordingly.

3 Proposed Method: HouseCrafter

3.1 Overview

Our goal is to lift a 2D floorplan to a 3D scene that we can interact with, where explicit scene representation is desired, e.g., in terms of meshes. If we had enough 3D data, training a generative model that outputs the desired 3D asset would be the most straightforward solution. In practice, however, 3D data is harder to acquire and thus far more scarce than 2D imagery. Therefore, in this paper, we resort to generating multi-view 2D observations of the scene first and then reconstructing it in 3D. It allows us to harness the powerful generative prior of recent advances in diffusion-based models that are trained using a large set of 2D images.

Figure 2: Layout-guided novel view RGB-D generation model. Adopted from Eschernet, our model has three important design changes for 3D scene generation. First, our model simultaneously denoises the latent of RGB images $\{{\mathbf{x}}_{i}^{r}\}_{i=1}^{N_{r}}$ and depth images$\{D_{i}^{r}\}_{i=1}^{N_{r}}$, enabling geometry and texture consistency. Second, the introduced layout-attention block allows the target images to condition on the given floorplan $L$. Lastly, DeCaPE is proposed to leverage the depth images of the reference views, allowing the attention between the point cloud features of reference views and target image features with only camera poses.

As shown in Fig. 1, we sample a lot of locations inside the house based on the 2D floorplan and then visit these locations autoregressively in a batch-wise manner. In each batch, with our developed novel view synthesis model, we take already generated RGB-D images from nearby locations as references, conditioned on the floorplan, we generate a set of both semantically and geometrically consistent RGB-D images in novel neighboring locations simultaneously. After exhausting these locations, we use an off-the-shelf TSDF fusion model Zeng et al. (2017) to reconstruct a detailed 3D vertex-colored mesh from the generated RGB-D images.

3.2 Layout-guided Novel View RGB-D Image Generation

We fine-tune the UNet of the StableDiffusion v1.5 Rombach et al. (2021) to leverage its powerful generation capacity obtained from training on web-scale data. Specifically, given the 2D floorplan $L$, the already generated depth images $\{D_{i}^{r}\}_{i=1}^{N_{r}}$ and the latent features of already generated RGB images $\{{\mathbf{x}}_{i}^{r}\}_{i=1}^{N_{r}}$ at certain poses $\{{\mathbf{P}}_{i}^{r}\}_{i=1}^{N_{r}}$ as references, the goal of our novel view synthesis model is to denoise the latents of RGB-D images $\{({\mathbf{x}}_{j}^{n},{\mathbf{d}}_{j}^{n})\}_{j=1}^{N_{n}}$ at the novel poses $\{{\mathbf{P}}_{j}^{n}\}_{j=1}^{N_{n}}$. $N_{r}$ and $N_{n}$ denote the number of reference and novel images, respectively. Both the reference pose ${\mathbf{P}}_{i}^{r}$ and novel pose ${\mathbf{P}}_{j}^{n}$ are in the $SE(3)$ space.

For the reference RGB image $I_{i}^{r}$, we use a lightweight image encoder (Woo et al., 2023) to get its latent ${\mathbf{x}}_{i}^{r}$. The latent ${\mathbf{d}}_{j}^{n}$ is obtained by replicating the single-channel target depth image into 3 channels and normalizing the depth value to $[-1,1]$ then passing through the VAE encoder. In this way, we can use the pre-trained VAE to encode it just like an RGB image. At the novel pose ${\mathbf{P}}_{j}^{n}$, the RGB and depth latents ${\mathbf{x}}_{j}^{n}$ and ${\mathbf{d}}_{j}^{n}$ are gradually denoised in $T$ steps starting from pure Gaussian noises. Finally, the latents are fed into the VAE decoder to get the RGB and depth images. In this section, we omit the denoising step for brevity.

An illustration of the model is shown in Fig. 2. Our model architecture is inspired by designs of SOTA object-centric novel view synthesis models (Zheng & Vedaldi, 2023; Kong et al., 2024), but re-designed for the geometric and semantic complexity of scene-level contents. First, we extend both the reference conditioning and image generation to the RGB-D setting instead of RGB only as RGB-D images provide strong cues for 3D scene reconstruction. Secomd, we insert a layout attention layer at the beginning of each unet block to encourage the generated images to be faithful to the floorplan, ensuring global consistency in generating a house-scale scene. Moreover, the cross-attention layer, which is introduced in prior works for reference-novel view attention, is updated to leverage geometry from the reference depth, leading to higher-quality image generation.

Multi-novel-view RGB-D Image Generation. Given RGB and depth latents ${\mathbf{x}}_{j}^{n}$ and ${\mathbf{d}}_{j}^{n}$, instead of denoising them separately, we concatenate them along the channel dimension as ${\mathbf{z}}_{j}^{n}=[{\mathbf{x}}_{j}^{n},{\mathbf{d}}_{j}^{n}]$. In this way, the model can effectively fuse the information of RGB and depth images into a single representation to ensure the semantic consistency between them at a single view. We double the input and output channels of the UNet to accommodate ${\mathbf{z}}_{j}^{n}$. When we denoise a set of latents $\{{\mathbf{z}}_{j}^{n}\}_{j=1}^{N_{n}}$ simultaneously, it ensures consistency across RGB and depth images both semantically and geometrically across different views and thus leads to higher-quality generation as shown in the experiments.

Floorplan Conditioning. We use a vectorized representation $L$ for the floorplan  Zheng et al. (2023), which describes the structure and furniture arrangement of the house from a bird-eye view. $L=\{o_{i}\}_{i=1}^{N}$ consists of $N$ items, where each component $o_{i}=\{c_{i},p_{i}\}$ is specified by its category $c_{i}$ and geometry information $p_{i}$. If the component $o_{i}$ represents furniture (e.g., a chair), $p_{i}$ defines the 2D bounding box enclosing the object. For other components, including walls, doors, and windows, it specifies the start and end points of a line segment corresponding to them.

To use it as condition to the diffusion model, we encode the floorplan with respect to each target view. Fig. 3 illustrates the encoding process for a novel view. For every pixel of its latents ${\mathbf{z}}_{j}^{n}\in\mathbb{R}^{C\times H\times W}$, we shoot a ray ${\mathbf{r}}$ originating at the camera center of ${\mathbf{z}}_{j}^{n}$ going through the pixel center, which is orthogonally projected down as ${\mathbf{r}}^{\prime}$ to the floor plane. Along the projected ray ${\mathbf{r}}^{\prime}$, we take at most $M$ points that intersect with the 2D object bounding boxes or other floorplan components (e.g., walls). With each intersection point, we obtain its position and the object category. Gathering across all the pixels of ${\mathbf{z}}_{j}^{n}$, we get ${\mathbf{c}}_{j}\in{\mathbb{N}}^{M\times H\times W}$ for the semantic category and ${\mathbf{p}}_{j}\in\mathbb{R}^{M\times 2\times H\times W}$ for the point position where the dimension of 2 consists the depth along the ray and the height from the floor.

Figure 3: Floorplan Encoding.

Note that we exclude the intersections after the ray first hits the wall to take the occlusion into effect, and use zero-padding to ensure the same number of intersection points per ray for batching.

To inject the floorplan information ${\mathbf{c}}_{j},{\mathbf{p}}_{j}$ into the latent ${\mathbf{z}}_{j}^{n}$, we first embed them into a latent space,

$${\mathbf{l}}_{j} =\texttt{Embed}({\mathbf{c}}_{j})+\texttt{PosEnc}({\mathbf{p}}_{j}),$$ (1)

where Embed() map each semantic class to a latent vector and PosEnc() is sinusoidal position embedding, to obtain ${\mathbf{l}}_{j}\in\mathbb{R}^{M\times C\times H\times W}$ which encodes both geometry and semantic of the floorplan.

Subsequently, the layout-attention block modulates RGB-D latents using cross-attention between the input latents and ${\mathbf{l}}_{j}$ on pixel level, each latent feature in ${\mathbf{z}}_{j}^{n}$ is the query and the floorplan features along the corresponding ray are the keys and values, meaning the attention for each pixel is performed independently. We provide more technical details in the appendix (Section A).

Multi-view Reference RGB-D Image Conditioning. In addition to being faithful to the input floorplan, the generated RGB-D images should be consistent with the reference images as well. Our multi-view RGB-D conditioning design is inspired by EscherNet (Kong et al., 2024), an object-centric novel-view synthesis method. By cross-attending from the target RGB latents (as query) to the reference RGB images (as key and value), where the image features are simply treated as tokens in sequences, it can take multiple RGB reference images as conditions to ensure its output’s quality. To encode the camera poses and thus capture the relative transformation of the target and reference view, Camera Positional Encoding (CaPE) was introduced to augment the visual tokens. In contrast, not only have RGB reference, we also have depth which can provide geometry reference. Hence, we introduce Depth-enhanced Camera Positional Encoding (DeCaPE) to better enhance the visual tokens to capture their similarity in 3D and thus improve the generation quality.

We first revisit CaPE and then describe DeCaPE. To avoid notation clutter, let’s denote ${\mathbf{P}}_{Q}={\mathbf{P}}_{j}^{n}$ and ${\mathbf{P}}_{K}={\mathbf{P}}_{i}^{r}$. Further, we have ${\mathbf{v}}_{Q}$ and ${\mathbf{v}}_{K}$, which are two tokens from ${\mathbf{z}}_{j}^{n}$ and ${\mathbf{x}}_{i}^{r}$, respectively. In CaPE, $\phi({\mathbf{P}})$ is defined in analogy to camera extrinsic ${\mathbf{P}}$ so that the visual tokens in high-dimensional space can be transformed via $\phi({\mathbf{P}})$ in the similar way that point cloud in 3D space is transformed via ${\mathbf{P}}$. The similarity between ${\mathbf{v}}_{Q}$ and ${\mathbf{v}}_{K}$ is then computed as

$$s_{QK}=\langle\phi({\mathbf{P}}_{Q}^{-\intercal}){\mathbf{v}}_{Q},\phi({\mathbf{P}}_{K}){\mathbf{v}}_{K}\rangle={\mathbf{v}}_{Q}^{\intercal}\phi({\mathbf{P}}_{Q}^{-1})\phi({\mathbf{P}}_{K}){\mathbf{v}}_{K} ={\mathbf{v}}_{Q}^{\intercal}\phi({\mathbf{P}}_{Q}^{-1}{\mathbf{P}}_{K}){\mathbf{v}}_{K}.$$ (2)

The key property of CaPE is that $\mathbf{P_{Q}^{-1}}\mathbf{P_{K}}$ encodes the relative transformation of the camera poses while being invariant to the choice of the world coordinate system. Eq.(2) can be interpreted as the feature of the reference view (key) in its camera coordinate system is transformed to the coordinate system of the novel view (query) before taking the dot product with the query feature. Since we have the explicit 3D position of the reference tokens from the reference depth image, DeCaPE uses the 3D position to augment ${\mathbf{v}}_{K}$ in its camera coordinate before applying the camera transformation,

$$s_{QK}={\mathbf{v}}_{Q}^{\intercal}\phi(\underbrace{{\mathbf{P}}_{Q}^{-1}{\mathbf{P}}_{K}}_{\text{camera poses}})({\mathbf{v}}_{K}+\underbrace{\texttt{PosEnc}({\mathbf{p}}_{K})}_{\text{3D position from depth}}),$$ (3)

where ${\mathbf{p}}_{K}$ is the 3D position of ${\mathbf{v}}_{K}$ in the camera coordinate of the key (reference view), which is obtained from depth image, and PosEnc() is a learnable positional encoding. While preserving the invariance to the choice of world coordinate, Eq.(3) enhances the similarity (attention score) computation of CaPE for the cross attention and therefore leads to better generation as we will show in the experiments.

Figure 4: Refinement example for a bed: To improve the quality of the noisy bed mesh (left), we sample cameras surrounding(highlighted in blue) the bed and generate RGB-D images in one batch, allowing more complete, smooth mesh (right)

3.3 3D Scene Reconstruction and Post Refinement

Autoregressive RGB-D Image Generation. The images are generated in a batch-wise manner, the order of which is decided by a connected pose graph. The camera poses are uniformly placed across the scene space, with randomly chosen rotation. We construct the pose graph by linking every two camera poses whose relative distance and rotation angle fall within a threshold. To generate the first batch, thanks to the classifer-free guidance, we take only the layout as condition to generate RGB-D images. Next, we generate RGB-D images at the neighboring locations conditioned on the already generated images in the first batch. We then traverse the entire graph in a batch-wise manner following this procedure. When traversing the graph and encountering a pose $v$ whose images have not been generated, we choose generated views within $\delta_{r}$ hops from $v$ as reference views and poses within $\delta_{n}$ hops from $v$ that have not been generated as novel views (We provides detailed procedure and ablation of $\delta_{r}$ and $\delta_{n}$ in Appendices B). With all the generated RGB-D images covering the entire house, we fuse them into a 3D mesh using TSDF fusion Zeng et al. (2017).

Post Refinement for Scene Reconstruction. While the pose graph provides good coverage of the scene, holes still exist in the reconstructed mesh, which happens in the region with clustered objects. In addition, for some objects (e.g., chairs, sofas, and beds), denser RGB-D images are needed to obtain detailed geometry and texture. Examples are shown in Fig. 4 (a). To address both issues, we densely sample more camera poses looking at each object in the scene and then generate all RGB-D images around the same object in a single batch. In this way, the dense, object-centric poses allow complete and detailed observations of the object and the single-step generation ensures the cross-view consistency, leading to higher reconstruction quality, as shown in Fig. 4 (b). We provide more details in Appendices B.

Figure 5: Qualitative comparison We show two random viewpoints for each scene as well as a top-down views. We compare our model with CC3D Bahmani et al. (2023) and Text2Room Höllein et al. (2023). HouseCrafter generates results with better geometry and textures. More examples are provided in Fig. 9.

4 Experiment

4.1 Experimental setup

Dataset. We conduct experiments on 3D-FRONT Fu et al. (2021), a synthetic indoor scene dataset that contains rich house-scale layouts and is populated by detailed 3D furniture models. Compared with other indoor scene datasets Dai et al. (2017); Chang et al. (2017), it allows us to render high-quality images of the scene at any selected pose, which is essential to training our novel view RGB-D image diffusion model. For each house in the dataset, we obtain the floorplan based on furniture bounding boxes and wall mesh and generate the training images by rendering from sampled poses. Nearly 2000 houses with 2 million rendered images are used for training while 300 houses are for evaluation

Figure 6: User Study  Participants significantly favor our method over baselines, for both overall quality and coherence to the floorplan.

Evaluation. We evaluate the multi-view RGB-D image generation and the quality of the reconstructed 3D scene meshes. Regarding the multi-view RGB-D generation, we evaluate the consistency among the multi-view images and their visual quality. For consistency, we consider two aspects: reference-novel (R-N) and novel-novel (N-N) view consistency. While the open-ended nature of the generation task makes the evaluation challenging due to the absence of ground truth information, we can measure the consistency of two views within their overlapped region, which can be estimated via the depth and poses. Given the estimated overlap region, we evaluate RGB consistency using PSNR and depth consistency using Absolute Mean Relative Error (AbsRel) and percentage of pixel inliers $\delta_{i}$ with threshold $1.25^{i}$. We also report Fréchet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) for the visual quality.

To evaluate the faithfulness to the input floorplan, we rely on the state-of-the-art 3D instance segmentation method, ODIN (Jain et al., 2024). We extract top-down 2D boxes of the 3D segmentation results to compare with the floorplan’s boxes using mAP@25 (Lin et al., 2014). While the absolute value of mAP does not directly reflect the layout compliance of the generated results due to segmentation errors, we assume that mAP has positive correlation with layout compliance, meaning better generation results leading to higher mAP. We also report mAP of ground-truth images as a reference.

Table 1: Quantitative comparison in terms of visual quality (IS) and compliance with layout guidance (mAP@25)

Method	Visual	Layout
	IS $\uparrow$	mAP@25$\uparrow$
Text2Room	5.35	-
CC3D	4.02	25.60
HouseCrafter	4.24	46.48
GT-3DFront	4.50	54.51

Regarding 3D scenes, we conduct an user study, involving 12 participants, to compare our results with baseline methods in terms of perceptual quality and coherence to the given floorplan. For each baseline, 8 pairs of meshes (our vs. baseline) are shown to the participant. We also add 3 pairs with grounthtruth meshes, resulting in a total of 228 data points. In addition, we report IS calculated from RGB images rendered at random poses for each scene. For methods that have layout guidance, mAP of instance segmentation is also reported. We provide more details about evaluation in the Appendix  C.3.

4.2 Comparison with state of the art

Baselines. There are no direct methods that generate 3D houses from floorplans. Closest to our work is CC3D  (Bahmani et al., 2023), which produces a room-scale indoor scene from 2D layout. CC3D represents the scene as a feature volume that can be rendered with a neural renderer. We also compare against Text2Room (Höllein et al., 2023), which generates an indoor scene from a series of text prompts. Since Text2Room (Höllein et al., 2023) does not receive any layout guidance, we only compare to it in terms of visual quality.

Results. We provide a detailed quantitative analysis in Fig. 6 and Table 1 and quanlitative comparisons in Fig. 5. Both human (Fig. 6) and automated Table 1 evaluations show that our method performs better in generating faithful results to the layout guidance. However, IS greatly favors Text2Room over our method and CC3D, while the users significantly prefer our results regarding the visual quality of the generated mesh. The higher IS of Text2Room is due to the more diverse scenes generated by the text-to-image model Rombach et al. (2022) trained on the web-scale dataset. Although our results are less diverse due to fine-tuneing on a smaller dataset, it can produce more realistic rooms with information from the floorplan, as recognized by users.

Table 2: Ablation studies of different design choices for novel view RGB-D image generation. The best results are highlighted with bold and the second best with underline.

Variant	Output Depth	Input Depth	Layout Cond.	RGB Metrics				Depth Metrics
				FID $\downarrow$	IS $\uparrow$	PSNR $\uparrow$		AbsRel $\downarrow$		$\delta_{0.5}$$\uparrow$
						R-N	N-N	R-N	N-N	R-N	N-N
①	✗	✗	✗	49.35	5.00	-	-	-	-	-	-
②	✓	✗	✗	33.39	5.23	20.99	22.60	23.56	11.48	79.14	88.79
③	✓	✓	✗	35.77	5.16	20.91	21.98	22.28	12.05	81.78	88.23
④	✓	✗	✓	15.64	4.70	25.36	24.79	7.65	7.85	90.44	91.77
⑤	✓	✓	✓	16.70	4.74	25.31	24.69	6.79	7.37	92.20	92.65

4.3 Ablation studies

We perform ablation studies for various design choices of the generation model on a set of 300 houses from 3D-FRONT datasets. We sample camera poses in groups of 6, 3 reference and 3 novel views. In each group, reference-novel consistency is measured using the correspondence of each novel view with all reference views, while novel-novel consistency is measured based on 3 pairs in each up of 3 novel views. Regarding layout evaluation, we use images generated in the autoregressive pipeline.

Table 3: Layout compliant evaluation.

Variant	Input	mAP@25$\uparrow$
	Depth
④	✗	48.46
⑤	✓	52.26
GT		52.56

Generating depth improves visual appearance. Variant pair (①, ②) in Table 2 demonstrates that by forcing the model to learn to generate depth, the FID and IS of RGB output are both improved, indicating better performance of the RGB generation.

Depth conditioning enhances geometry consistency. As shown in variants pairs (②,③) and (④,⑤) in Table 2, reference depth images improves the depth consistency with a stronger effect in R-N than N-N, while having mixed influences on the RGB metrics. The geometry improvement also benefits layout compliance (Table 3), demonstrating the effectiveness of the depth condition.

Layout guidance is critical for both appearance and geometry quality. Variant pairs (②, ④) and (③, ⑤) show strong improvement in all metrics especially the depth by having the layout conditioning. The results reinforce the finding from previous works  Schult et al. (2023); Fang et al. (2023) that coarse depth and high-level semantic information from the layout have a significant impact on the generation results.

5 Conclusion

In this work, we present HouseCrafter, a pipeline that transforms 2D floorplans into detailed 3D spaces. We generate dense RGB-D images autoregressively and fuse them into a 3D mesh. Our key innovation is an image-based diffusion model that produces multiview-consistent RGB-D images guided by floorplan and reference RGB-D images. This capability enables the generating of house-scale 3D scenes with high-quality geometry and texture, surpassing previous approaches which could only generate scenes at the room scale.

Appendix A Details of Floorplan Conditioning

For a novel view with the latent feature ${\mathbf{z}}_{j}^{n}\in\mathbb{R}^{C\times H\times W}$ (where $C$ is the feature dimension and $H\times W$ the spatial dimensions), we obtain the layout information ${\mathbf{l}}_{j}\in\mathbb{R}^{M\times C\times H\times W}$ at the (latent) pixel-level by casting rays through the pixels and encoding semantic and geometric information at every intersection point between the projected ray and floorplan components.

Subsequently, we use cross-attention at the ray-level where each pixel feature the in ${\mathbf{z}}_{j}^{n}$ is the query and the layout features along the ray are the keys and values, meaning the attention for each ray is performed independently. To illustrate the operation we added the batch dimension $B$ and use einops Rogozhnikov (2022) notation:

$${\mathbf{z}}_{j}^{n} \leftarrow\textrm{rearrange}({\mathbf{z}}_{j}^{n},\textrm{ B C H W }\rightarrow\textrm{(B H W) 1 C})$$

$${\mathbf{l}}_{j} \leftarrow\textrm{rearrange}({\mathbf{l}}_{j},\textrm{B N C H W}\rightarrow\textrm{(B H W) N C})$$

$${\mathbf{z}}_{j}^{n} \leftarrow\textrm{MHA}(q={\mathbf{z}}_{j}^{n},k={\mathbf{l}}_{j},v={\mathbf{l}}_{j})$$

$${\mathbf{z}}_{j}^{n} \leftarrow\textrm{rearrange}({\mathbf{z}}_{j}^{n},\textrm{(B H W) 1 C}\rightarrow\textrm{ B C H W }),$$

where MHA() is multihead attention layer. The layout information injection is applied in the first block of each feature level in the Unet blocks of the base diffusion model. Note that each level operates at a different resolution, so this amounts to injecting the encoded floorplan at different scales.

In the design described above, we choose to inject into each pixel the information from a single ray while alternatively, a receptive field with kernel size $K>1$ provides more spatial information. We argue that the quadratic growth $O(K^{2})$ of the sequence length of keys and values is expensive for the attention operation while the local information exchange between pixels can be handled in the convolution layers of the network. Furthermore, attention to intersection points from a single ray omits the requirement of using 3D positions for these points, which depend on an arbitrary world coordinate, since the depth along the ray is enough to differentiate these points. We also use the height with respect to the floor for the position because the up direction is a well-defined canonical direction for the indoor scene and the height may help the model decide the visible object along the ray

Appendix B Details of 3D Scene Reconstruction and Post Refinement

B.1 Autoregressive RGB-D Image Generation.

To generate RGB-D images for scene reconstruction, we first create a connected pose graph $G(V,E)$, where the vertices $V$ are camera poses uniformly placed across the free space obtained from the layout, with randomly chosen rotation. Two poses are linked if their relative distance and rotation angle fall within a threshold.

The reference and novel poses are selected while traversing the graph. The procedure is described in Algo. 1. To control the number of poses in each generation step, we use two parameters $\delta_{r},\delta_{n}$ which are the hop distance with respect to the current pose for the reference and novel poses. When visiting a pose $v$ whose images have not been generated, we choose generated views within $\delta_{r}$ hops from $v$ as reference views and the novel poses are those that have not been generated and within $\delta_{n}$ hops from $v$.

Figure 7: Influence of $\delta_{r}$ and $\delta_{n}$. In the left chart we vary $\delta_{n}$ from $1$ to $4$ while kee** $\delta_{r}=4$ and vary $\delta_{r}$ in the right chart. Increasing $\delta_{r}$ (right) and $\delta_{n}$ (left) improves both visual quality (FID) and layout compliance (mAP) of the generated image sequences. $\delta_{n}$ has more significant effect than $\delta_{r}$.

We exam influence of $\delta_{r}$ and $\delta_{n}$ on the generated image sequence using FID and layout evaluation (Fig. 7). Specifically, we vary $\delta_{n}$ in the range $[1,4]$ while kee** $\delta_{r}=4$ and vice versa. With the hop distance of $4$ the number of views can be as many as $80$, we limit the number of novel/reference views at $60$ due to the memory constraint. As shown in Fig. 7, higher hop distance leads to higher FID and mAP.

Algorithm 1 Autoregressive generation via graph traversal

$G(V,E)$: Pose graph

$\delta_{n}$: Hop distance for novel views

$\delta_{r}$: Hop distance for reference views

$X\leftarrow\emptyset$ $\triangleright$ Initialize set of generated poses.

for $v$ in $DFS(G)$ do $\triangleright$ traverse graph via depth-first search.

     if $v\notin X$ then

         $X_{r}\leftarrow X\cap N(v,G,\delta_{r})$ $\triangleright$ Get reference poses. $N(v,G,d)$: nodes within $d$ hop from $v$

         $X_{n}\leftarrow N(v,G,\delta_{n})\backslash X$ $\triangleright$ Get novel poses.

         if $X_{n}\neq\emptyset$ then

              $Generate(X_{r},X_{n})$ $\triangleright$ Generate novel views.

              $X\leftarrow X\cup X_{n}$

         end if

     end if

end for

B.2 Post Refinement for Scene Reconstruction.

After generating images for all poses in the graph, we further generate object-centric views for furniture in the scene to reduce the missing observation. To sample the camera location, we use a heuristic based on the 2D floorplan and the statistics of the object’s height in the dataset to avoid positions that may be inside the object. In particular, for each object we derive a 3D bounding box from its 2D box in the floorplan and the maximum height of the objects in the dataset with the same category. Using derived bounding boxes as occupied regions, for each object we sample 20 poses within $2$ meter looking at the object center, these views are generated in a single batch using nearby, previously generated views as the reference.

Appendix C Details of Evaluation

C.1 Consistency Evaluation

In this section, we describe the correspondence estimation for a pair of posed RGB-D images. Then we provide the details of the evaluation metrics.

Correspondence estimation Given a pair of views, each with RGB and depth, $(I_{1},D_{1})$ and $(I_{2},D_{2})$, we warp images $I_{1},D_{1}$ of the first view to the second view, obtaining $I_{1\rightarrow 2},D_{1\rightarrow 2}$. If the pair of images views are perfectly consistent, the correspondence region $\mathcal{M}$ is the region that the warped depth $D_{1\rightarrow 2}$ match perfectly with $D_{2}$,

$$\mathcal{M}\coloneq\mathbbm{1}(D_{1\rightarrow 2}=D_{2}),$$ (4)

where $\mathbbm{1}()$ is indicator function. To account for the potential inconsistency of the generated images, we introduce a tolerance threshold $\tau$ to estimate the correspondence,

$$\mathcal{\hat{M}}\coloneq\mathbbm{1}(|D_{1\rightarrow 2}-D_{2}|<\tau).$$ (5)

Given the estimated correspondence $\mathcal{\hat{M}}$, the level of consistency is computed for depth image pair $(D_{1\rightarrow 2},D_{2})$ and the RGB image pair $(I_{1\rightarrow 2},I_{2})$.

RGB Metrics. Given the image pair $(I_{1\rightarrow 2},I_{2})$ and the correspondence $\mathcal{\hat{M}}$, we compute the peak signal-to-noise ratio PSNR for color consistency,

$$PSNR\coloneq 20\cdot\log_{10}(255)-10\cdot\log_{10}(MSE),$$ (6)

where

$$MSE\coloneq\frac{1}{\sum_{k}\mathcal{\hat{M}}(k)}\sum_{k}\mathcal{\hat{M}}(k)\cdot[I_{1\rightarrow 2}(k)-I_{2}(k)]^{2},$$ (7)

$k$ is pixel index. Note that we omit averaging over the color channels to simplify the equation.

Depth Metrics. Given the image pair $(D_{1\rightarrow 2},D_{2})$ and the correspondence $\mathcal{\hat{M}}$, we compute Absolute Mean Relative Error (AbsRel) and percentage of pixel inliers $\delta_{i}$ for depth consistency. AbsRel is calculated as:

$$AbsRel\coloneq\frac{1}{\sum_{k}\mathcal{\hat{M}}(k)}\sum_{k}\mathcal{\hat{M}}(k)\cdot\frac{|D_{1\rightarrow 2}(k)-D_{2}(k)|}{D_{2}(k)}.$$ (8)

The percentage of pixel inliers $\delta_{i}$ is calculated as:

$$\delta_{i}\coloneq\frac{1}{\sum_{k}\mathcal{\hat{M}}(k)}\sum_{k}\mathcal{\hat{M}}(k)\cdot\mathbbm{1}\left(\max\left(\frac{D_{1\rightarrow 2}(k)}{D_{2}(k)},\frac{D_{2}(k)}{D_{1\rightarrow 2}(k)}\right)<1.25^{i}\right).$$ (9)

We choose $i=0.5$ to have a tight threshold.

C.2 Layout Evaluation

The layout evaluation protocol is the ”inverse” of HouseCrater where we predict the top-down 2D bounding boxes of objects in the generated scene. The predicted 2D bounding boxes are then compared with 2D boxes from the given floorplan using mean Average Precision at the intersection-over-union threshold of 0.25, mAP@25. Specifically, we use ODIN Jain et al. (2024), a 3D instance segmentation method that takes multi-view posed RGB-D images as input and predicts instance segmentation of the point cloud accumulated from input images. Then, top-down 2D boxes are extracted from the segmented instances. As a scene may have up to 2000 images based on its size, we cannot pass all the images to ODIN at once. Instead, these images are partitioned by room, we do segmentation per room. This strategy does not affect the evaluation results since an object in the scene do not span in more than one room. We finetune ODIN on 3D-Front dataset to make the segmentation results more reliable since both HouseCrafter and CC3D are trained on this dataset.

C.3 User Study

Figure 8: User Study Interface. We show users 2 meshes at a time, one is produced by our model and the other is produced by a baseline method. We then ask users to choose one mesh that appears “better looking in general”, and one mesh that appears “align better” with the given floorplan.

We conduct a user study to evaluate the results produced by Text2Room, CC3D, and our method. In the study, we ask 12 participants to rate the results in a pair-wise manner. Specifically, we present the participants with two meshes at a time and ask them to choose: i) the one that appears more visually appealing; and ii) the one that is more coherent with the provided floorplan. The interface is shown in Fig. 8. Since Text2Room does not take layout as a form of guidance, we do not report participants’ answers to the second question if one of the meshes is produced by it. However, we still ask the question to prevent unconscious bias. Given that CC3D generates results at the room level rather than for entire houses, we clip our results and floorplan to the specific room CC3D produces when making comparisons.

Appendix D Implementation details

D.1 Training

We initialize our model from StableDiffusion v1.5 Rombach et al. (2021). For the first layer of the Unet, we duplicate the pre-trained weights and divide the weights by two to accommodate the depth’s latent and to reduce the change of the output scale. For the last layer of the Unet, we only duplicate the pre-trained weights. The model is trained for $15,000$ iterations in 2 days with an effective batch size of 256 ($4$ samples per GPU $\times 8$ GPUs $\times 8$ gradient accumulation steps). Each data sample contains 3 reference views and 3 novel views with the resolution of 256. We use Adam optimizer with a learning rate of $10^{-4}$. All training is conducted on a machine with $8$ A6000 48GB GPUs.

Appendix E Limitations and Future Directions

Our work is the first that can generate textured meshes of 3D scenes at the house-scale, and yet without limitations, allowing intriguing future directions.

First, the employed TSDF fusion method produces reasonable results in fusing generated RGB-D images and robust their inconsistency. However, it cannot model the view-dependent color, baking the lighting effect into the mesh texture, and thus giving unsatisfactory results. To address this issue, a reconstruction method that is robust to the inconsistency of generated multi-view images and able to model view-dependent color is required.

Second, while using image generation models gives the advantages of using large-scale image data as prior for 3D generation, the current pipeline has a lot of redundancy from the high overlap of multiview images. Thus an effective poses sampling strategy that can balance the view overlap for consistency and efficiency is a promising direction.

Lastly, in our proposed method of injecting the layout guidance to the generation process, only the geometry and semantics of the object are leveraged, while the information about the object instance is omitted. We believe that instance-awareness can give better scene understanding thus generating scene more faithful to the floorplan.

Appendix F Additional Results

Figure 9: Additional comparisons with CC3D and Text2Room