Higher levels of physical activity (PA) and cardiorespiratory fitness (CRF) are associated with lower risk of incident cardiovascular disease (CVD). However, the relationship of aerobic PA and CRF with risk of atherosclerotic CVD outcomes and heart failure (HF) seem to be distinct. Furthermore, recent studies have raised concerns of potential toxicity associated with extreme levels of aerobic exercise, with higher levels of coronary artery calcium and incident atrial fibrillation noted among individuals with very high PA levels. In contrast, the relationship between PA levels and measures of left ventricular structure and function and risk of HF is more linear. Thus, personalizing exercise levels to optimal doses may be key to achieving beneficial outcomes and preventing adverse CVD events among high risk individuals. In this report, we provide a comprehensive review of the literature on the associations of aerobic PA and CRF levels with risk of adverse CVD outcomes and the preceding subclinical cardiac phenotypes to better characterize the optimal exercise dose needed to favorably modify CVD risk.