But the government rejected the suggestion, saying that it aimed to resettle most by the end of this year.
About 250,000 people fled the final bloody phase of the civil war between the
government and Tamil Tiger rebels.
Meanwhile, a human rights group accused the government of failing to probe rights abuses during the conflict.
It was in the final weeks of the war that hundreds of thousands of civilians streamed out of rapidly-diminishing rebel-held territory.
They were ultimately housed in government-run camps in the district of Vavuniya.
The UN expressed concern about the permanent nature of the shelters being put up in these camps.
The official, Mark Cutts, said that nothing less than a new city had been created at Manik Farm, the massive complex of camps where he worked for the past month as a senior co-ordinator.
He said bulldozers were working constantly to clear jungle and that phone lines, schools, banks and even a cash machine had been built. He said this was "phenomenal" but described government plans to replace tents with more permanent structures as a "big worry".
"Senior military officials have also told us that they don`t expect to see any significant returns in the next six months, On the contrary, some senior officials told us just yesterday that they expect probably not more than 20% of these people will have returned in the next year," Mr Cutts said.
But Sri Lanka`s human rights minister Mahinda Samarasinghe said it was "absolutely false" to suggest that it would take so long. He said it was not the military but the government who took such decisions and that it aimed to resettle most people by the end of this year.
In the past week, the government says about 2,000 displaced people have been resettled in their villages in the north-west. These people fled their homes about two-and-a-half years ago.
It says that the refugees living in the camps are being strictly vetted to ensure they have no links with the rebels. Only after that process can their return home be considered.