Cortisol is known to affect memory processes. On the one hand, stress-induced or pharmacologically induced elevations of cortisol levels enhance memory consolidation. On the other hand, such experimentally induced elevations of cortisol levels have been shown to impair memory retrieval. However, the effects of individual differences in basal cortisol levels on memory processes remain largely unknown. Here we tested whether individual differences in cortisol levels predict picture learning and recall in a large sample. A total of 1225 healthy young women and men viewed two different sets of emotional and neutral pictures on two consecutive days. Both sets were recalled after a short delay (10 min). On Day 2, the pictures seen on Day 1 were additionally recalled, resulting in a long-delay (20 hr) recall condition. Cortisol levels were measured three times on Days 1 and 2 via saliva samples before encoding, between encoding and recall as well as after recall testing. We show that stronger decreases in cortisol levels during retrieval testing were associated with better recall performance of pictures, regardless of emotional valence of the pictures or length of the retention interval (i.e., 10 min vs. 20 hr). In contrast, average cortisol levels during retrieval were not related to picture recall. Remarkably during encoding, individual differences in average cortisol levels as well as changes in cortisol did not predict memory recall. Our results support previous findings indicating that higher cortisol levels during retrieval testing hinders recall of episodic memories and extend this view onto interindividual changes in basal cortisol levels.