With robots being introduced into caregiving, particularly for older persons, various ethical concerns are raised. Among them is the fear of replacing human caregiving. While ethical concepts like well-being, autonomy, and capabilities are often used to discuss these concerns, this paper brings forth the concept of social dignity to further develop guidelines concerning the use of robots in caregiving. By social dignity, we mean that a person's perceived dignity changes in response to certain interactions and experiences with other persons. In this paper, we will first present the concept of social dignity, and then identify a niche where robots can be used in caregiving in an ethical manner. Specifically, we will argue that, because some activities of daily living are performed in solitude to maintain dignity, a care recipient will usually prefer robotic assistance instead of human assistance for these activities. Secondly, we will describe how other philosophical concepts, which have been commonly used to judge robotic assistance in caregiving for the elderly so far, such as well-being, autonomy, and capabilities, are less useful in determining whether robotic assistance in caregiving is ethically problematic or not. To conclude, we will argue that social dignity offers an advantage to the other concepts, as it allows to ask the most pressing questions in caregiving.