We analyze a model of a nonlinear bath consisting of a single two-level system coupled to a linear bath (a classical noise force in the limit considered here). This allows us to study the effects of a nonlinear, non-Markoffian bath in a particularly simple situation. We analyze the effects of this bath onto the dynamics of a spin by calculating the decay of the equilibrium correlator of the z-component of the spin. The exact results are compared with those obtained using three commonly used approximations: a Markoffian master equation for the spin dynamics, a weak-coupling approximation, and the substitution of a linear bath for the original nonlinear bath.