Galaxy Build-up in the first 1.5 Gyr of Cosmic History: Insights from the Stellar Mass Function at $\mathbf{z\sim 4-9}$ from JWST NIRCam Observations

In order to measure consistent colors across the different available filters, we have to take into account the wavelength dependence of the point spread function (PSF). We wish to produce a PSF-matched photometric catalog, matching all filters to the PSF resolution in the reddest JWST/NIRCam wide filter, F444W.

We extract PSFs in each field and for all filters directly from the mosaics. First, we identify stars in each filter from a preliminary catalog as

The PSF may vary as a function of the position across the footprint of a given field. After conducting some tests, we however conclude that for our adopted aperture radius of 0.16″(see below), those differences only affect the encircled energies on the 1% level or below. We therefore neglect them, and derive one PSF per field.

We use the python tool psf.EPSFBuilder ³³3https://photutils.readthedocs.io/en/stable/api/photutils.psf.EPSFBuilder.html (Anderson & King, 2000; Anderson, 2016) which is part of the photutils package (Bradley et al., 2022), to derive an effective PSF from the background-subtracted cutouts around the selected stars. We use a normalization radius of 10 pixels (0.4″), $\sigma$-clip** with a 5$\sigma$ cut, and a maximum of 10 clip**-iterations, a quartic smoothing kernel and a maximum of 50 PSF fitting/modelling iterations. For the NIRCam short-wavelength channel (NIRCam/SW) filters, for which the FWHM is close to the pixel scale, we apply an oversampling of 3 and use a quadratic smoothing kernel which yield better results. In the JADES/GOODS-S footprint, we use images sampled to a smaller 0.02″pixel scale in the NIRCam/SW filters to extract PSFs in which case we adjust the normalization radius to 20 pixels and need no oversampling to obtain accurate PSFs. We then resample those PSFs to the 0.04″pixel scale which is used for all photometric measurements.

We compute matching kernels from all the ACS and NIRCam PSFs to the NIRCam/F444W PSF using the software package pypher (Boucaud et al., 2016) with the regularization parameter set to 1e-4 and convolve each flux and rms image with the corresponding kernel to match the PSF resolution in F444W. Fluxes and flux uncertainties for each source (see below) are measured from those PSF-matched images.

For the WFC3 filters whose PSFs are broader than the NIRCam/F444W PSF, we follow a different procedure. First, we compute kernels matching all of them to the broadest PSF among them, WFC3/F160W, using pypher as described above. Then, we also produce a matching kernel from F444W to F160W and generate flux and rms images, PSF-matched to F160W, for all the WFC3 filters and F444W. In order to match the WFC3 fluxes to the PSF-resolution in F444W, we make use of the following equality

Figure 1 illustrates the PSF extraction and matching process. On the top, we show PSFs for the NIRCam/SW filter F200W, extracted from the four different fields used in this work (CEERS-EGS, PRIMER-UDS, PRIMER-COSMOS and JADES-GS) as well as a generic PSF generated through webbpsf. For the latter, we set the jitter_sigma parameter to 0.022 which has been shown to well reproduce radial profiles of NIRCam-observed stars in Morishita et al. (2023), and also does so in our case. The bottom left panel shows logarithmic radial profiles of all the PSFs shown in the top panels which agree very well, in particular at small radii $\lesssim$0.35″, including our adopted aperture radius of 0.16″(see below). In the bottom right of Figure 1 we show the encircled energies of 4 different PSFs extracted from the CEERS field, divided by the encircled energies of the corresponding F444W PSF. We chose one PSF from HST/ACS (F606W), HST/WFC3 (F125W), NIRCam/SW (F150W) and NIRCam/LW (F277W) respectively for illustrative purposes. Then, we match each of the four PSFs to its respective reference PSF (F160W for F125W; F444W for all others) by convolving it with the corresponding matching kernel and recompute the encircled energies. The resulting plot, showing them relative to the respective reference PSF (F444W or F160W), serves as an internal check of the PSF-matching procedure and demonstrates its self-consistency. For a perfect matching, the displayed ratio would be $=1$ at all radii. In practice, the residuals are $<$1% for the F125W (and all filters that are PSF-matched to F160W) and $\ll$1% for all filters matched to F444W. The displayed filters are representative for the measured residuals in all filters.

The software SourceExtractor (SE, Bertin & Arnouts 1996) is used to detect sources and to perform photometric measurements. SE is run in dual image mode using an inverse-variance weighted stack of the original (unconvolved) F277W+F356W+F444W images as the detection image, a detection threshold of 1.3 relative to the rms image, a minimum area of 7 pixels and a deblending parameter of 3e-4. Before detecting sources, the image is smoothed with a 3$\times$3 pixels Gaussian filter with a FWHM of 1.5 pixels (following the same procedure as in Weaver et al., 2023a). We measure fluxes on the PSF-matched images in each filter respectively in circular apertures with four different radii, 2.5, 4, 5 and 8.75 pixels (0.1″, 0.16″, 0.2″, and 0.35″).

We scale all fluxes to the flux measured in Kron-like apertures by SE in a PSF-matched, inverse-variance weighted stack of the NIRCam long wavelength channel wide filters (F277W+F356W+F444W), where the Kron ellipse itself is inferred from the detection image using the default Kron parameters 2.5 and 3.5. Whenever the area encircled by the Kron ellipse is smaller than the area of the circular aperture, we use the circular aperture fluxes and do not apply any scaling. Next, we scale the fluxes to “total” fluxes by measuring the fraction of the encircled energy of the Kron ellipse (or the circular aperture) on the F444W PSF and dividing the Kron-corrected fluxes by that fraction. Whenever the Kron ellipse is larger than the 4.04″$\times$ 4.04″PSF, we instead approximate it as a circle with radius $\sqrt{ab}\times kron\_radius$, where $a$, $b$ and $kron\_radius$ are the SE output characterising the Kron ellipse, and infer the fraction of the encircled energy from a simple radius vs. encircled energy table generated through webbpsf. Finally, we correct all fluxes for Milky Way foreground extinction using the $E(B-V)$ map from Schlafly & Finkbeiner (2011) and the extinction model from Fitzpatrick & Massa (2007) through the python package extinction. The inferred correction factors for each filter and field are listed in Table 1.

We estimate the true rms by summing the pixels within apertures put on the flux image divided by the rms image. First, we choose 5000 random positions without any nearby objects or flagged pixels according to the flag images and the segmentation map from a preliminary SE run with low relative detection threshold (0.9) and minimum area (5 pixels). We perform measurements in circular apertures of radii 1,2,…,10 pixels, measure the scatter among the apertures of a given size respectively, and divide it by the aperture radius times $\sqrt{\pi}$ to get the measured scatter per pixel. In theory, if the rms image perfectly describes the (random) noise and there is no correlated noise, this scatter measured on what is effectively a signal-to-noise image, should be equal to one (see e.g., Whitaker et al., 2011, for further reading). In practice, it depends on the aperture size and the photometric filter, and is typically in the range $0.8-1.5$. To obtain the appropriate scaling factor for a given aperture size, we linearly interpolate between the measured values for 1,2, … 10 pixel apertures. We multiply the uncertainties on all fluxes in a given filter, measured from the rms map respectively, by this factor. Further, we apply an error floor of 5% to the total errors to account for remaining systematic uncertainties and to allow for more flexibility in the SED-fitting described below.

We use a similar procedure to estimate the 5$\sigma$ depths of each image respectively in different circular apertures, by measuring the scatter among the fluxes measured in the 5000 randomly placed apertures described above on the flux image respectively. The resulting 5$\sigma$ depths in each field and filter are listed in Table 1. Those have to be interpreted as average depths over the entire field. In particular, the GOODS-S field has JWST imaging data from various programs, leading to varying depths across the field, while the other three fields have more homogeneous coverage in most of the available filters.

We run the SED fitting code eazy (Brammer et al., 2008) on the final total fluxes and errors, using the blue_sfhz template set⁴⁴4https://github.com/gbrammer/eazy-photoz/tree/master/templates/sfhz. This consists of 13 templates generated through the Flexible Stellar Population Synthesis (FSPS) code (Conroy et al., 2009; Conroy & Gunn, 2010), with redshift-dependent SFHs and physical properties. To account for extreme emission lines observed at high redshifts, the best fit to a NIRSpec spectrum of a galaxy with strong emission lines at $z\sim 8.5$ from Carnall et al. (2023a) is added as a 14th template. We make use of the zeropoint optimization in eazy, to correct for possible remaining calibration offsets based on the template fitting residuals. Performing 3 iterations we find small corrections of typically $1-4$% for all the JWST and HST filters, with the exception of correction factors up to $8$% for some JWST filters in GOODS-S, and of up to $13$% for the HST B-band (F435W). Finally, we allow the best-fitting redshift to be in the range $z_{\rm phot,\,eazy}\in(0.01,20)$.

We mask bright saturated stars, together with their diffraction spikes as well as noisy regions - mostly along the edges of the images - by hand, to avoid having to deal with objects that are contaminated by stellar light and or with substantially enhanced noise and include the masked regions as 1-valued pixels in our flag images. In order to flag corresponding sources generously, we apply a binary dilation⁵⁵5https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.binary_dilation.html 10 times to the flag image in each filter. We then define binary flags flag_FIL for each filter FIL. For each object in the catalog, if its isophotal footprint according to the segmentation map from SE overlaps with one or more flagged pixels in the dilated flag image, we set the corresponding flag_FIL = 1 and the total flux and error to -99 - so that the corresponding measurements are ignored in the subsequent SED-fitting runs.

In order to also flag less bright and unsaturated stars, we define a stellar_flag, for which we use two different criteria:
First, we identify bright point sources (mag(F444W) $<$ 24.5) through their flux ratio 1.2 < F(0.35″) / F(0.16″) < 1.4, in analogy to equation 1. Second, we match our catalogs to sources in the GAIA DR3 catalog (Gaia Collaboration et al., 2016, 2023) with non-zero proper motion, using a large search radius of 2.5″ to include diffraction spikes and nearby contaminated sources around non-saturated stars. In our full catalogs over all four fields combined, this flags 0.5% of the sources.

We further define a huge_flag with which we flag sources with very large spatial extent (R${}_{50}$(F444W) > 50 pix) where R${}_{50}$(F444W) is the half-light radius measured by SE in F444w or unreasonably large Kron radii ($\sqrt{(}ab)\,{\rm kron\_radius}>150$ pix) where a, b and kron_radius are adopted from the SE output. The former criterion flags some extended foreground galaxies with unproblematic photometry but those are irrelevant for this work as we only consider galaxies at $z\gtrsim 3$ (see below). In total, only 0.1% of all sources are flagged as “huge”.

Flagging sources with 0 $<$ R${}_{50}$(F444W) $<$ 1.2 pixels and mag(F444W) $<$ 28.5 (approximately corresponding to the 5$\sigma$ depths of the images, cf. Table 1) reliably identifies some remaining spurious detections and hot pixels which we specify as a junk_flag. This flags another 0.5% of all objects in the catalogs.

Finally, we flag sources with a S/N ratio $<3$ in all of F115W, F150W, F200W, F277W, F356W and F444W (sn_flag, 2.3% of all sources), which is largely but not entirely superseded by the S/N cut applied in Section 2.3, as well as sources for which eazy returns a best-fitting $\chi^{2}$ value of $-1$ (nofit_flag). The latter affects as many as 10.7% of all sources, the vast majority of which have no best-fitting redshift ($z_{\rm phot,\,eazy}=-1$). In most cases, this is due to missing filters, since we set up eazy to require at least 5 filters to provide a fit. This is entirely superseded by requiring all six NIRCam wide filters from F115W to F444W in Section 2.3. The nofit_flag however also includes sources with $z_{\rm phot,\,eazy}\approx 20$ where the best-fitting solution converged to the edge of the prior (0.6%).

To assess the detection completeness of our catalogs, we use the GaLAxy survey Completeness AlgoRithm 2 (GLACiAR2) software (Leethochawalit et al., 2022), which builds on its predecessor GLACiAR (Carrasco et al., 2018). This software injects modelled galaxies into real images and then runs SE to obtain statistics on the fraction of recovered sources as a function of their input and output magnitude. We start by choosing a representative 1.5′$\times$1.5′cutout in each field (CEERS, PRIMER-UDS, PRIMER-COSMOS and JADES-GS) that does not contain any masked stars (Section 2.2.4). On this cutout, we run GLACiAR2 with the schechter_flat shape of the input luminosity function, performing 10 iterations with 500 galaxies per iteration, but only injecting 100 galaxies at a time to avoid overcrowding. Sources are injected in 34 magnitude bins ranging from 22.5 to 30.5 AB.

In this work, we only consider the detection completeness as a function of magnitude and therefore simulate all galaxies at a fixed redshift of 6 and with a flat SED ($\beta=-2$). We run GLACiAR2 on the three filters F277W, F356W and F444W which together form the stacked detection image (see Section 2.2). The injected galaxies follow a Gaussian distribution in the logarithm of their sizes, centred at R${}_{\rm eff}=0.8\,{\rm kpc}$ which is inferred from the measured half-light radii of the galaxies in our sample. They have Sérsic disc profiles with 25% of the galaxies having Sérsic indices of 1 and 2 respectively and 50% having a Sérsic index of 1.5. We further use 5 equally sized bins in inclination and eccentricity, respectively.

Given this setup, GLACiAR2 returns a matrix where the fraction of recovered sources is specified as a function of both input and measured output magnitude. This allows us to account for the fact that galaxies scatter between input and output magnitude bins. The completeness is then given by the fraction of recovered sources that also have ${\rm S/N}>10$ in the detection image. Typical completeness factors at mag(det-img.)$<28.5$, i.e., above the 5$\sigma$ depths of all fields (see Table 1), range from 0.75 in JADES-GS to 0.91 in PRIMER-UDS.

To avoid large corrections, we wish to only show SMFs in a regime where our sample is mostly complete. To this end we chose a rather conservative S/N threshold of 10 in the detection image in Section 2.3. In addition, we now derive 80% mass completeness limits per field in each redshift bin following a similar approach as, e.g., Pozzetti et al. (2010). The mass limit depends on the mass-to-light ratio distribution of the galaxy population. For each source, we compute the hypothetical minimum mass $M_{*,hyp}$ at which it would still be included in our catalog. Thus, $M_{*,hyp}=M_{*}/r_{\rm S/N}$, where $r_{\rm S/N}={\rm(S/N)_{\rm det}/(S/N)_{\rm thresh}}$, (S/N)${}_{\rm det}$ is the signal-to-noise ratio in the detection image and (S/N)${}_{\rm thresh}=10$.

The 80th percentile of the distribution of all $M_{*,hyp}$ in a given redshift bin then represents the mass above which 80% of all sources are observed above our signal-to-noise threshold, i.e., with (S/N)${}_{\rm det}>10$. We round the inferred mass up to the next mass bin-edge to obtain our completeness limit. This is done for each field separately as different fields have different depths. In the following, we only show SMFs in mass-bins where the completeness is expected to be $>80$%. The adopted completeness limits range from log$(M_{*,{\rm lim}}/{\rm M_{\odot}})=7.5$ in GOODS-S at $z\sim 4$ to log$(M_{*,{\rm lim}}/{\rm M_{\odot}})=9$ in PRIMER-UDS, -COSMOS and CEERS at $z\sim 9$.

To account for mass incompleteness within a given redshift bin j, we apply a $V/V_{\rm max}$ correction (Schmidt, 1968). For each source q in the sample, we derive the maximum redshift $z_{\rm max}(q)$ at which it would still be observed above the signal-to-noise threshold (S/N)${}_{\rm thresh}=10$, using the following equation

There are three contributions to the uncertainty of the SMF measurement: the Poisson uncertainty on the number count in a given stellar mass and redshift bin, the uncertainty in stellar mass and redshift – which are a result of uncertainties in the photometry and the SED-fitting procedure – and cosmic variance.

We estimate the Poisson uncertainty using the frequentist central confidence interval (see Maxwell 2011).

In order to take uncertainties in both the photometry and the SED-fitting and the associated uncertainties in $M_{*}$ and $z$ into account, we make use of the Bayesian nature of bagpipes and sample our SMFs from its posterior distributions. Specifically, we sample values of $M_{*}$, $z$, and $\beta$ for each galaxy 1,000 times from the respective posterior distributions and compute 1,000 corresponding SMFs (for all, as well as for “UV-blue” and “UV-red” sources separately, see Section 3.2.3). In each stellar mass and redshift bin, we use the median number count among the 1000 sampled values as our final measurement of the number density and the scatter as a contribution to its uncertainty.

Finally, we estimate cosmic variance following the methodology presented in Moster et al. (2011) through the publicly available python package cosmic-variance⁷⁷7https://pypi.org/project/cosmic-variance/, taking field geometry into account, and using equation (7) in Moster et al. (2011) to combine the cosmic variances inferred for each field to a total cosmic variance. The inferred values strongly increase with mass and redshift. At $z\sim 4$ they range from an uncertainty of 10% for the $8.5<{\rm log}(M_{*}/{\rm M_{\odot}})<9$ bin to $\sim$37% at the high-mass end, while at $z\sim 9$, the equivalent values are 70% and 191%. At the lowest masses shown in each redshift bin, the contribution of cosmic variance slightly increases as the corresponding SMF values are not inferred from all four fields, but only from CEERS and/or JADES-GS which are sufficiently deep to probe those masses (see Section 2.7.2). In Appendix A, we present SMFs for each field separately. The scatter among them provides a constraint on the cosmic variance, and it is consistent with or smaller than the values inferred based on Moster et al. (2011).

The three uncertainties are added in quadrature to obtain a total uncertainty.

Having derived the binned SMFs, we fit a single Schechter function (Schechter, 1976), characterised by a power-law slope $\alpha$, a characteristic mass $M^{*}$, above which the function drops exponentially, and a normalisation $\Phi^{*}$. Specifically, we perform the fitting to each of the 1000 SMF realisations drawn from the bagpipes posterior distributions (for UV-red, UV-blue and all galaxies respectively, Section 3.2.3). We use the inferred median values for $\alpha$, $M^{*}$ and $\Phi^{*}$ as our final Schechter fitting parameters and the scatter among the values derived in the 1000 respective fits as the uncertainty of the fitting parameters.

As has been known for a long time, the number count of galaxies in a given mass and redshift bin is an overestimate of the true number count due to the so-called Eddington bias (Eddington, 1913). Because of the steepness of the SMF, the number of lower mass galaxies scattering into bins of higher mass is always larger than vice versa, therefore biasing the inferred SMF towards higher masses. Various approaches to account for this bias have been discussed in the literature (e.g. Ilbert et al., 2013; Grazian et al., 2015; Adams et al., 2021; McLeod et al., 2021).

We follow the approach outlined in Ilbert et al. (2013) and convolve our Schechter function by the uncertainty in $M_{*}$ before fitting $\alpha$, $M^{*}$ and $\Phi^{*}$. Instead of using an analytic approximation to characterise the uncertainty in $M_{*}$, we construct a numerical uncertainty distribution in log($M_{*}$) in each redshift bin based on the median bagpipes posterior distribution. Those median distributions turn out to be quite well characterised by normal distributions with standard deviations $\sigma_{M_{*}}=0.13,0.17,0.19,0.20,0.22$ and 0.25 in our redshift bins centred at $z\sim$4, 5, 6, 7, 8 and 9 with the inferred distributions being slightly skewed towards lower masses. If we refer to the inferred median uncertainty distribution as $\mathrm{U}$ and the Schechter function as $\Phi$, the convolved Schechter function $\Phi_{\rm conv}$ reads

One of the key questions in early galaxy science was whether the HST-based source selections at $z>3$ might be missing a large fraction of red, obscured sources. Thanks to JWST’s NIR sensitivity, this question can now be answered. In Figure 4, we show the distributions of stellar mass $M_{*}$ and UV-slope $\beta$ as estimated through bagpipes (Section 2.4) in five different redshift bins centred at $z$ = 4, 5 and 6, and a combined bin $z\sim 7-8$ ($6.5<z<8.5$), for all the sources in our sample and for the subset of sources that are matched with the B15 sample.

As explained in Section 2.4, $\beta$ is inferred from the best fit SED. We specifically highlight the objects that are not photometrically detected in the rest-frame UV (diamonds, ${\rm S/N}<2$ in the filter closest to 1500Å rest-frame), meaning that their $\beta$ value is based on an extrapolation of the fit in the rest-optical and not itself well constrained by the photometry. This mostly affects the reddest and most massive sources. Also, the LRDs and red compact sources (filled and empty big markers) identified in Section 2.5 are primarily located in the upper right part of the diagrams, at high masses and UV-slopes. As expected, most of them are not detected in the rest-UV and many of them were missing from the B15 sample.

The number of sources detected in this work is a factor of 10 - 30 $\times$ larger compared to the B15 sub-sample in each redshift bin. The highest increase is seen at $z\sim 4$ because only the GOODS-S field has complete HST/ACS B-band (F435W) coverage, allowing for the $z\sim 4$ B-band dropout selection in B15.

In the other three redshift bins shown, the two samples are based on roughly the same survey area over the four fields studied here. Specifically, $>80$% of our survey area is also covered by the HST H-band, as can be seen in Table 1. The smoothed, normalised histograms on top of each panel show that we probe $\sim 0.25-0.5$dex lower in mass compared to B15. Assuming a low-mass end slope of the SMF of $\alpha=-2$ (cf. Section 3.2.4), this results in a factor of 3 to 10 $\times$ increase in the number of galaxies. To quantify this more accurately, one would however need to take into account the relative survey depth between the underlying JWST and HST imaging for the two samples respectively, which varies from field to field (and even within fields).

A remaining excess of galaxies detected here can be attributed to the selection functions in B15 (see their Figure 4). By design, the color-color LBG selections identify galaxies around a certain redshift, but the completeness of the selection drops as one moves further away from the central redshift of the bin. Conversely, our photo-z selection, in principle, selects galaxies uniformly across the whole redshift range in each bin.

The $\beta$-distributions, shown on the right of each panel are roughly consistent between the full sample and the B15-matched sub-sample. This has to be interpreted cautiously since it has been well established that $\beta$ and $M_{*}$ are correlated at the redshifts studied here (e.g., Finkelstein et al., 2012). As we are probing lower masses compared to B15, one might expect the $\beta$-distributions of the full sample to be slightly shifted towards lower values. However, towards the lowest masses probed, the $\beta$-values do not show a strong trend with mass anymore, but stabilise around slopes of $\beta\sim-2.2$ with a “floor”at $\beta\sim-2.5$ induced by the models used in the SED-fitting (Section 2.4). It can therefore be inferred from Figure 4 that there is no strong bias seen in the B15-matched sample in terms of the overall distribution of UV-slopes. Importantly though, there is a small population of UV-red galaxies seen at all redshifts which is not present in the B15-matched sample, especially beyond the contours characterising the bulk of the sample. Those sources preferentially lie at high stellar masses, log($M_{*}/{\rm M_{\odot}})\gtrsim 9$, but reach down to as low as log($M_{*}/{\rm M_{\odot}})\sim 8$.

In order to investigate the abundance of UV-red galaxies at fixed stellar mass, i.e., to eliminate the mass-dependence of the UV-slope $\beta$, we split our sample into “UV-blue” and “UV-red” galaxies adopting a simple cut of $\beta>-1.2$ for the “UV-red” sample. This roughly corresponds to a dust extinction of $A_{V}\sim 1\,$mag. As discussed in, e.g., Wilkins et al. (2011), the value of $\beta$ depends on various physical properties of a galaxy like the metallicity, the star formation history and the IMF. It is however mainly driven by dust extinction and can be taken as a good proxy for E(B - V). Here, we simply want to make an observational distinction between UV-red and UV-blue galaxies which is defined somewhat arbitrarily, and is only used to illustrate differences and trends in the galaxy population. Subsequently, “UV-red” refers to $\beta>-1.2$ and “UV-blue” to $\beta<-1.2$ unless otherwise stated.

In Figure 5, we show the fraction of UV-red galaxies measured in logarithmic bins of mass with bin edges defined as log($M_{*}/{\rm M_{\odot}})=[8,8.5,9,9.5,10.5,11.5]$ and in four different redshift bins with bin edges $z=[3.5,4.5,5.5,6.5,8.5]$, i.e., $z\sim 4$, 5, 6 and $z\sim 7-8$. The wider bins at higher masses and redshifts are chosen to increase the number statistics.

We do not show the fractions in some mass and redshift bins due to the very small number of objects. In particular, at $z\sim 4$, there are 53 sources in the highest mass bin with a UV-red fraction of $\sim 95$%. None of those 53 sources shows up in the B15 sample because, as noted in Section 2.6, it only covers the GOODS-S field at $z\sim 4$ while 45/53 of the ultra-massive sources at $z\sim 4$ are scattered over the three other fields studied in this work (UDS, COSMOS and EGS).

Figure 5 shows that the UV-red fraction – as measured from our full sample – is a strong function of mass, with close to no UV-red galaxies at log($M_{*}/{\rm M_{\odot}})\sim 8$ and $\gtrsim 40$% UV-red galaxies at log($M_{*}/{\rm M_{\odot}})\sim 10$, but does not show a clear dependence on redshift. The conspicuously high UV-red fraction at $z\sim 7-8$ and log($M_{*}/{\rm M_{\odot}})\sim 10$ decreases to $\sim 48$%, if LRDs and red compact sources are removed, making it more consistent with the fractions measured at that mass in lower redshift bins, where including or excluding the red compact objects does not make a significant difference.

To provide another comparison, we also show the fraction of sources analogous to HST-dark galaxies. We use the selection cuts from Williams et al. (2023) to identify such sources requiring a red color ${\rm F150W}-{\rm F444W}>2.2$, and a faint magnitude at $\sim 1.5$µm, ${\rm mag(F150W)}>27\,{\rm AB}$. The latter cut implies that a corresponding source would typically remain undetected in CANDELS, where the H-band at $\sim 1.6$µm is the reddest available filter, reaching depths $\lesssim 27\,{\rm AB}$ (see B15). Since this selection is not actually based on HST-data, we subsequently refer to the corresponding sources as “HST-dark” in quotation marks. We show the fraction of “HST-dark” sources in dark red, with uncertainties in analogy to the other fractions shown.

In the highest mass-bin at $z\sim 4$, the “HST-dark” fraction reaches 34%, indicating that at least about a third of those galaxies would be expected to be missing from HST-based LBG-samples. Gottumukkala et al. (2023) applied a similar selection, requiring ${\rm F150W}-{\rm F444W}>2.1$, combined with a weaker cut in F150W, ${\rm mag(F150W)}>25\,{\rm AB}$. They show that $\sim$60% of their selected objects are massive, dusty galaxies at $z>3$. If we keep our red cut at 2.2 but apply their weaker magnitude cut in F150W, the fraction of such sources in the highest mass bin at $z\sim 4$ is 91%, indicating that dusty, optically-dark or faint galaxies completely dominate in that regime.

At higher redshifts, the number of galaxies with log($M_{*}/{\rm M_{\odot}})>10.5$ drops significantly. We find 8 in total at $z\sim 5$ with a UV-red fraction of 7/8 (87.5%) and 1/8 sources being recovered in B15 (albeit one of the UV-red sources). None of those 8 objects is selected as “HST-dark”, but 5/8 pass the selection applied in Gottumukkala et al. (2023). In the highest two redshift bins, we do not show the fractions in the highest mass bin. At $z\sim 6$ there are only two such sources. One of them is “HST-dark”, and they are both not present in B15. We discuss the 13 sources discovered at $z\sim 7-8$ with log($M_{*}/{\rm M_{\odot}})>10.5$ (none of which is matched with the B15 sample) in Section 4.4 and argue that most of them likely have wrongly inferred masses and/or redshifts. Note also, that towards higher redshifts, an increasing fraction of the “HST-dark” sources consists of LRDs and red compact sources. In particular, the total fraction of “HST-dark” sources at $M_{*}\sim 10^{10}\,{\rm M_{\odot}}$ drops to 4% (7%) at $z\sim 6$ ($z\sim 7-8$) if LRDs and red compact sources are removed.

Figure 5 further shows that the UV-red fraction is consistent between B15 and the full sample up to log($M_{*}/{\rm M_{\odot}})\sim 10$ and $z\sim 6$. At higher masses and/or redshifts, the fraction of red, but also of “HST-dark” galaxies indicate that typical HST-based LBG-samples may be missing a significant fraction of them. At $z\sim 7-8$, the UV-red fraction is close to zero in the B15 sub-sample at all masses while the full sample still shows an increasing UV-red fraction with mass, consistent with the trend seen at lower redshift. Interestingly, the B15 subsample also appears to be missing some UV-red galaxies at $z\sim 6$ and with $M_{*}\sim 10^{9}\,{\rm M_{\odot}}$.

Figure 6 shows the inferred SMFs in our six different redshift bins. At each redshift, the number density of galaxies is computed in mass-bins with a width of 0.5 dex, starting with the lowest mass bin in which the completeness of our sample is estimated to be $>80$% (see Section 2.7.2), and going up to log($M_{*}/{\rm M_{\odot}})=12$. The error bars represent the 16th and 84th percentile of the 1000 SMF realisations sampled from the bagpipes posterior distributions, with Poisson uncertainty and cosmic variance added in quadrature (see Section 2.7.4). If the lower uncertainty reaches to a value $\leq 0$, we plot a downward pointing arrow instead of an error bar below the SMF point, but keep the upper error bar. In bins where we count zero galaxies, we plot an upper limit given by the Poisson single-sided 1$\sigma$ upper limit from Table 1 in Gehrels (1986). We also show the SMFs inferred when including LRDs (Section 2.5). Our best-fitting Schechter functions, convolved with the uncertainty in log($M_{*}$) to account for the Eddington bias, are overplotted as the green dashed lines.

Schechter fits from the pre-JWST literature, convolved with our inferred uncertainty distributions for consistency, are overplotted in various colors (Duncan et al., 2014; Grazian et al., 2015; Song et al., 2016; Stefanon et al., 2017; Davidzon et al., 2017; Bhatawdekar et al., 2019; Kikuchihara et al., 2020; Furtak et al., 2021; Stefanon et al., 2021; Weaver et al., 2023b); the SMF from Navarro-Carrera et al. (2023) is based on HST data from the CANDELS catalogs, combined with some of the early public JWST imaging over the UDS and GOODS-S fields. The results from Wang et al. (2024) are derived from PRIMER data, including MIRI, and those from Harvey et al. (2024) are based on a compilation of various deep fields observed by JWST (PEARLS, CEERS, GLASS, JADES GOODS-S, NGDEEP and SMACS0723). All the Schechter functions in Figure 6 are only shown above their respective mass completeness limit, or in the regime where they are specified in the respective papers. Our SMFs are broadly consistent with the literature, with a few noteworthy exceptions which we list here and discuss in more detail below.

First, at $z\sim 4$, we measure a relatively high number density of galaxies at the high-mass end compared to Schechter fits for most SMFs from the literature, but are still consistent with measurements from a number of works (e.g., Caputi et al., 2015; Weaver et al., 2023b; Gottumukkala et al., 2023; Wang et al., 2024) (see Section 4.2). This relatively high abundance of massive galaxies is not affected by LRDs or red compact objects as defined in Section 2.5, and it is not seen at $z\sim 5-6$, implying a strong evolution of the SMF high-mass end in that redshift range. The main driver of this evolution are the 53 predominantly very red galaxies at $z\sim 4$ with masses log($M_{*}/{\rm M_{\odot}})>10.5$ which we have discussed separately in Section 3.1.2.