The study of poverty in historical archaeology has not yet developed a productive engagement with the complex political economy of impoverishment. A primary concern with culture and representation has instead supported the production of often essentialized subjects who ultimately mirror the problematic foundations of the "culture of poverty" thesis. This paper critiques the processes of constructing impoverished subjects and considers the notion of a "poverty of culture" as a relational position for analysis. Working in close collaboration with members of an impoverished African American community in Setauket, New York, alternative readings of poverty, culture, heritage, and archaeology are discussed. These alternatives serve as the foundations of a community-driven project informed by indigenous meanings and interests in the archaeological past in order to challenge the marginalization of this part of the broader local community.