He was speaking following work led by the Department of Justice in collaboration with the local community and Armagh City, Banbridge and Craigavon Borough Council saw the 3m-tall peace wall removed.
A local developer is planning to regenerate and redevelop the derelict site.
Mr Tennyson has said: “We all want to be part of a community that’s respectful and inclusive, and where people can live and work free of barriers, whether they be mental or physical.
“The removal of the Water Street interface, which has stood for 27 years, is a really welcome and encouraging step forward towards achieving the united community we should all be striving for.
“I’d like to pay tribute to both my colleague Justice Minister Naomi Long in paving the way for such a significant development and her ongoing commitment to tackling division in our society, and local residents and community organisations who have been pivotal in bringing us to this point.
“The project is particularly exciting in the context of the proposed redevelopment of Water Street, and I hope this will be a precursor to wider regeneration in the area.”