Objective To explore the different long-term effects of infancy and childhood trauma on adult memory function. Methods A total of 1 542 workers of Kailuan Group born before and after the Tangshan earthquake in 1976 were investigated from January to December in 2014. All subjects were divided into four groups according to the birth date and the average score of childhood trauma questionnaire（ CTQ）： infancy exposure + CTQ high group， infancy exposure + CTQ low group， no exposure + CTQ high group， no exposure + CTQ low group. The Hopkins Verbal Learning Test（ HVLT） and Brief Visuospatial Memory Test（ BVMT） were used to evaluate the verbal memory and spatial memory respectively for three times， and the total scores of three assessment results were recorded. Results There were statistically significant differences in age， gender and education level among the four groups（ Fage=470.094， Fgender=11.911， Fdegree of education=43.484， P ＜ 0.01）. The age of infant exposed group was greater than that of no exposure group. Male were more than female in the four groups， and the education level was mainly high school. There was no significant difference in marital status， smoking history， drinking history and family monthly income（ P ＞ 0.05）. In male participants， there were statistically significant differences in the scores of HVLT1-3， HVLT total score， BVMT1-3 and BVMT total score in male participants（ P＜0.05）， but no difference in above scores in female（ P＞0.05）. In addition to HVLT1， the other scores of infant expouse + CTQ high group were significantly lower than those of the no exposure + CTQ high group（ P＜0.05）. All scores of the no exposure + CTQ high group were lower than those ofthe no exposure + CTQ low group（ P＜0.05）. Only the score of HVLT3 in the infants exposure + CTQ high group was lower than that in the no exposure + CTQ high group（ P＜0.05） and BVMT3 score of infant exposure + CTQ low group was lower than that of no exposure + CTQ low group（ P＜ 0.05）. In female， there was no statistical significance in the differences of all scores（ P＞0.05）. In male participants with education level of junior high school and below， only the difference in score of HCLT2 was statistically significant（ P＜0.05）， and there was no statistical significance in other score of memory function among the 4 groups（ P＞0.05）. The differences in all scores of the participants with senior high school education background were statistically significant（ P＜0.05）， while there was no statistically significant difference in all scores mentioned above of participants with college degree and above among the four groups（ P ＞ 0.05）. Conclusions In male， infancy and childhood trauma can lead to impairment of verbal and spatial memory in adulthood， which caused by childhood trauma is more obvious. Accepting higher education may counteract the long-term effects of early trauma on memory.