The present study explores the validity of inferences that may be drawn from the Trends in International Mathematics and Science Study (TIMSS) learner achievement data by measuring the South African curriculum–TIMSS alignment for the Grade 8 science assessments from 2007 to 2019. The Singapore curriculum–TIMSS alignment indices are a fair benchmark for comparison, as Singapore is the best-performing country in the TIMSS. The research questions that guide the present study included the calculation and comparison of the South African curriculum–TIMSS and Singapore curriculum–TIMSS alignment indices. The study analysed 585 South African curriculum items, 659 Singapore curriculum items and 1025 TIMSS assessment items from 2007 to 2019. The present study employed four raters and achieved an inter-rater reliability coefficient of 0.85. The South African curriculum–TIMSS analysis achieved an average content alignment index of 0.34, and an average cognitive alignment index of 0.54. As a benchmark, the Singapore curriculum–TIMSS analysis achieved an average content alignment index of 0.36 and an average cognitive alignment index of 0.74. The literature reviewed supports acceptable content and cognitive alignment indices of 0.67 and 0.65, respectively. The present study concludes that valid inferences from learner performance in the TIMSS may not be drawn for the case of South Africa, and even for Singapore valid inferences are based on cognitive alignment only. However, South Africa’s continued participation in the TIMSS is encouraged as the literature supports many benefits associated with the TIMSS non-achievement data.