In recent years, an increasing number of proper motions have been measured for Galactic X-ray binaries. When supplemented with accurate determinations of the component masses, orbital period, and donor luminosity and effective temperature, these kinematical constraints harbor a wealth of information on the systems’ past evolution. We developed an analysis that allows us to consider all this available information and reconstruct the full evolutionary history of X-ray binaries back to the time of core collapse and compact object formation. This analysis accounts for four evolutionary phases: mass transfer through the ongoing X-ray phase, tidal circularization before the onset of Roche-lobe overflow, motion through the Galactic potential after the formation of the compact object, and binary orbital dynamics at the time of core collapse. The constraints on compact object progenitors and kicks derived from this are of immense value for understanding compact object formation and exposing common threads and fundamental differences between black hole and neutron star formation. Here, we present the results of such an analysis for the black hole X-ray binary XTE J1118+480. Assuming that the system originated in the Galactic disk and the donor had solar metallicity. We find that a high magnitude asymmetric natal kick is not only plausible but required for the formation of the system. We also investigate a globular cluster origin of XTE J1118+480 which would requires a low metallicity donor star. It turns out that such a scenario involves a lot of fine tuning and seems rather improbable.