This study investigates the effects of displacement damage induced by heavy ion irradiation on the structural and electrical properties of ferroelectric hafnium zirconium oxide (HZO) capacitors and thin films, establishing displacement-damage limits and decoupling the electrical degradation from irradiation-induced phase transformation by comparing films with different starting phase constitution. Using 6 and 8 MeV Au2+ ions, the radiation tolerance of HZO capacitors was assessed through polarization-voltage and capacitance-voltage measurements. The findings reveal that significant losses in remanent polarization do not occur until high displacements per atoms (DPA), around 10-2 DPA. While lower DPA values allow for partial recovery of polarization through voltage cycling, the highest levels result in irreversible degradation. Structural characterization via grazing-incidence X-ray diffraction and Fourier transform infrared spectroscopy indicates that the HZO films containing only the metastable phases maintain their phase stability, with no significant amorphization or phase transformations, even at the highest DPA levels. However, films containing the equilibrium monoclinic phase experience phase transformations, likely due to increased oxygen vacancies which destabilize this phase. Photoluminescence spectroscopy provides evidence of a significant change of the defect populations in the irradiated samples. The introduction of oxygen vacancies during irradiation is identified as a primary factor contributing to the degradation of ferroelectric properties due to their ability to pin ferroelectric domains. This work underscores the critical role of phase composition in determining the radiation tolerance and performance of HZO capacitors and demonstrates that these devices are promising candidates for radiation environments due to their high intrinsic radiation hardness.