Surveys of star-forming regions reveal that the dust mass of protoplanetary discs decreases by several orders of magnitude on timescales of a few million years. This decrease in the mass budget of solids is likely due to the radial drift of millimetre (mm) sized solids, called pebbles, induced by gas drag. However, quantifying the evolution of this dust component in young stellar clusters is difficult due to the inherent large spread in stellar masses and formation times. Therefore, we aim to model the collective evolution of a cluster to investigate the effectiveness of radial drift in clearing the discs of mm-sized particles. We use a protoplanetary disc model that provides a numerical solution for the disc formation, as well as the viscous evolution and photoevaporative clearing of the gas component, while also including the drift of particles limited in size by fragmentation. We find that discs are born with dust masses between 50 M· and 1000 M·, for stars with masses, respectively, between 0.1 M· and 1 M·. The majority of this initial dust reservoir is typically lost through drift before photoevaporation opens a gap in the gas disc for models both with and without strong X-ray-driven mass-loss rates. We conclude that the decrease in time of the mass locked in fragmentation-limited pebbles is consistent with the evolution of dust masses and ages inferred from nearby star-forming regions, when assuming viscous evolution rates corresponding to mean gas disc lifetimes between 3 Myr and 8 Myr.