We demonstrate a striking but previously unnoticed relationship between city size and the black-white wage gap, with the gap increasing by 2.5% for every million-person increase in urban population. We then look within cities and document that wages of blacks rise less with agglomeration in the workplace location, measured as employment density per square kilometer, than do white wages. This pattern holds even though our method allows for non-parametric controls for the effects of age, education, and other demographics on wages, for unobserved worker skill as proxied by residential location, and for the return to agglomeration to vary across those demographics, industry, occupation and metropolitan areas. We find that an individual’s wage return to employment density rises with the share of workers in their work location who are of their own race. We observe similar patterns for human capital externalities as measured by share workers with a college education. We also find parallel results for firm productivity by employment density and share college-educated using firm racial composition in a sample of manufacturing firms. These findings are consistent with the possibility that blacks, and black-majority firms, receive lower returns to agglomeration because such returns operate within race, and blacks have fewer same-race peers and fewer highly-educated same-race peers at work from whom to enjoy spillovers than do whites. Data on self-reported social networks in the General Social Survey provide further evidence consistent with this mechanism, showing that blacks feel less close to whites than do whites, even when they work exclusively with whites. We conclude that social distance between blacks and whites preventing shared benefits from agglomeration is a significant contributor to overall black-white wage disparities.