Emotional information is typically better remembered than neutral content, and previous studies suggest that this effect is subserved particularly by the amygdala together with its interactions with the hippocampus. However, it is not known whether amygdala damage affects emotional memory performance at immediate and delayed recall, and whether its involvement is modulated by stimulus valence. Moreover, it is unclear to what extent more distributed neocortical regions involved in e.g., autobiographical memory, also contribute to emotional processing. We investigated these questions in a group of patients with Alzheimer's disease (AD), which affects the amygdala, hippocampus and neocortical regions. Healthy controls (n = 14), patients with AD (n = 15) and its putative prodrome amnestic mild cognitive impairment (n = 11) completed a memory task consisting of immediate and delayed free recall of a list of positive, negative and neutral words. Memory performance was related to brain integrity in region of interest and whole-brain voxel-based morphometry analyses. In the brain-behavioral analyses, the left amygdala volume predicted the immediate recall of both positive and negative material, whereas at delay, left and right amygdala volumes were associated with performance with positive and negative words, respectively. Whole-brain analyses revealed additional associations between left angular gyrus integrity and the immediate recall of positive words as well as between the orbitofrontal cortex and the delayed recall of negative words. These results indicate that emotional memory impairments in AD may be underpinned by damage to regions implicated in emotional processing as well as frontoparietal regions, which may exert their influence via autobiographical memories and organizational strategies.