A preliminary definition of improvisation encompasses such a broad range of human actions that it is helpful to consider both improvisation and rhythm in terms of embodied cognition and a notion of bodily empathy. This suggests a possible (though unstable and inconclusive) connection to action understanding, empathy, and mirror neurons, while acknowledging the latter’s disputed status. The concept of action understanding offers a reconsideration of improvisation and music cognition with or without bodies (i.e. live or recorded). The relationship of improvisation, rhythm and embodiment to contemporary theories of expectation, speech, and the evolution of music are considered. Action understanding is posited as the foundation of both music cognition and the perception of improvisation, marking both processes as inherently intersubjective, even whether the other’s body is absent or fantasized (as is the case with recorded music).