StarVector: Generating Scalable Vector Graphics Code from Images

The efficacy of our model relies heavily on the image encoder, which needs to preserve intricate details and semantic content in the original image because, unlike captioning, where the output is typically short, SVG generation requires generating much longer sequences of code to obtain results of higher complexity. The image encoder projects the input image into a set of 2D embeddings rich in fine-grained details. To choose the best encoder, we draw inspiration from the success of pre-trained encoder backbones in downstream computer vision tasks such as classification [57], retrieval, and generation [25], including both convolutional and transformer-based models. Formally, we experiment with CLIP ViT-L/14 [57], ConvNext [44] (both pre-trained on LAION-2B [66]), and VQGAN [25], which we pre-train on an image reconstruction task using raster images from SVG-Stack. As the output of the image encoder, we utilize all the available hidden representations in the last layers to bring the most rich features. We define the output of the encoder $z_{v}$ as a flattened 2D grid of $L_{v}$ embedding sequences. For CLIP we have $L_{v}=257$ embeddings, including the CLS token. For VQGAN, we use the pre-quantization layers and flatten them to obtain $L_{v}=196$ embeddings. For ConvNext, we flatten the last activation map to obtain $L_{v}=49$ embeddings.

Adapter. The Adapter module performs a non-linear projection of the image embeddings into the LLM embedding space, producing a set of visual token embeddings (or visual tokens). This transformation matches the embedding dimensionality and aligns the image representations with the language model’s embedding space, effectively bridging the visual and SVG code modalities for the generation task. The Adapter is composed of a sequence of fully connected (FC) layers with Swish [58] activation function and Batch Normaliazation [33].

The CodeLLM generates the complete SVG code conditioned on the visual tokens representing the image. We employ the StarCoder architecture by Li et al. [40] with pre-trained weights, which provide a general model for code completion tasks. StarCoder is a decoder-only architecture that uses Multi-Query Attention [68] for efficient sampling. To address the long sequence lengths and high memory demands, we use flash-attention [18], enabling fine-tuning StarCoder with a context length of 8,192 tokens, the only restriction of our models. This approach mitigates the quadratic complexity typically associated with neural attention in long sequences. The fine-tuning process updates all the model weights to overcome the distribution shift from the original pre-training task of general code generation to our specific task of image-to-SVG conversion. We emphasize that the pre-trained StarCoder is not trained to generate SVG code and thus needs to be fine-tuned end-to-end.

To encompass varying degrees of visual complexity across different colors, shapes, and text, we select datasets comprising examples of fonts, emojis, icons, and real-world examples e.g., the ones seen on websites. The datasets included in SVG-Bench are visually diverse and frequently used by digital artists in real-life scenarios. We use SVG-Fonts introduced as Glypazzn [45] and SVG-Icons in DeepSVG [13]. In the absence of a standard benchmark, we create different splits for all the datasets, which we release to the community for further research. Following is a description of two datasets we propose for SVG generation.

In evaluating SVG models, it is crucial to employ metrics that accurately measure the fidelity of the generated SVG with the original image, considering both vector and raster-pixel representations. Traditional pixel-based metrics may not be entirely adequate for SVG assessment, as the predominance of background colors can skew them. For instance, a simple SVG with a predominantly white background might misleadingly score well in these metrics. To address this, our evaluation framework also includes deep perceptual-based metrics and vector-space metrics, offering a more comprehensive and nuanced assessment of SVG conversion quality. We compute the following metrics:

• •

  Pixel-based metrics. We employ Mean Squared Error (MSE) and Structural Similarity Index (SSIM) [81, 80]. MSE quantifies the average squared difference between the generated and the original images’ pixels, offering a straightforward measure of pixel-level accuracy. SSIM evaluates image quality based on the understanding of visual perception, measuring changes in structural information, luminance, and contrast.

• •

  Vector-based metrics. We utilize Chamfer distance (CD), a metric adapted from point cloud literature [84]. This involves discretizing each SVG into a set of points by sampling paths at regular intervals. CD measures the average nearest-point distance between corresponding points in two SVGs, providing a quantitative measure of similarity. A smaller CD indicates that the two SVGs are more similar, while a larger distance suggests they are more distinct. Having two SVGs $s_{1}$ and $s_{2}$, defined with a set of points $p_{1}\in s_{1}$ and $p_{2}\in s_{2}$ in 2D, the CD is defined as,

  $\displaystyle c(s_{1},s_{2})=\frac{1}{|s_{1}|}\sum_{p_{1}\in s_{1}}\min_{p_{2}\in s_{2}}\|p_{1}-p_{2}\|_{2}^{2}+\frac{1}{|s_{2}|}\sum_{p_{2}\in s_{2}}\min_{p_{2}\in s_{1}}\|p_{2}-p_{1}\|_{2}^{2},$

  (1)

  where $|s_{i}|$ is the cardinality of set $s_{i}$, and $\|.\|_{2}^{2}$ is the squared Euclidean norm.

• •

  Perceptual-based Metrics. We incorporate the Learned Perceptual Image Patch Similarity (LPIPS) [87] metric, which uses deep learning models trained on human perceptual judgments. This metric is particularly effective in capturing the nuances of human visual perception, providing a more subjective assessment of image quality beyond mere pixel-level comparisons.

Here we describe the baselines used to compare StarVector’s performance in the task of image-to-SVG conversion. We consider previous deep learning-based methods and rule-based traditional methods. We evaluate the baselines with publicly available code in our proposed setup.

Im2Vec [61] uses an end-to-end VAE, trained using only image supervision to produce vector graphics. Input rasterized image is encoded to a ‘global’ latent vector, which is passed to an RNN to produce latent code for each path. The path decoder decodes these path codes into Bezier paths to generate the output SVG. We used the publicly available code⁶⁶6https://github.com/preddy5/Im2Vec to report the results.

DeepSVG [13] was introduced as a hierarchical path-based VAE encoder-decoder transformer architecture. Here input paths are encoded separately using a path encoder which are aggregated using a second encoder to produce a latent vector. The decoder uses this latent vector to output the path representations which provide actual draw commands and arguments. We used the open-source code⁷⁷7https://github.com/alexandre01/deepsvg to reproduce the results on different datasets. However, since the DeepSVG framework only allows simplified SVGs, we report results on the ‘simplified’ test set in Table 5.

VTracer⁸⁸8https://github.com/visioncortex/vtracer [78] is a rule-based algorithm to convert images to SVGs. This 3-step pipeline algorithm relies on the hierarchical clustering of images which are traced into vectors. First, pixels are converted into paths, which are simplified into polygons. In the last step, polygons are smoothened and approximated with a Bezier curve fitter.

GPT-4-Vision (Preview) [52] was recently released by OpenAI as a vision-based multimodal model, with a limit of 100 API calls per day in the preview mode. We use GPT-4V by inserting an image and zero-shot prompting to generate SVG code. See Appendix 11 for more details.