Plans to change a young offenders institute into an adult prison housing sex offenders are likely to go ahead in early July, a local councillor said.
The proposal is to create a category C facility in the former Ashfield Young Offenders Institution in Pucklechurch.
A meeting between councillors and the National Offender Management Service on Tuesday was told the village could not present a legal challenge to the plans.
People in the village, near Bristol, fear it will be stigmatised.
'Great disappointment'
The move to change Ashfield to an adult prison was part of a government announcement on the future of the prison service in January.
Pucklechurch ward councillor Ben Stokes, of South Gloucestershire Council, said: "Legally the prison service is within its rights to change the classification of the prison without consultation so we cannot mount a legal challenge to the plans.
"We expressed great disappointment that the whole situation was not handled better."
At the meeting, councillors sought a commitment from the government to work with community leaders to allay fears about the impact of the change and the way ex-prisoners will be integrated into the surrounding area.
"We want a liaison group to be formed with the prison and for there to be regular meetings with us," said Mr Stokes.
"We've been told the change is likely to happen early in July and people in Pucklechurch don't want it to become known as the sex offenders village with all the stigma attached to that," he said.
The Ministry of Justice has been invited to comment.