The US-backed motion called on Colombo to address alleged abuses of international humanitarian law.
It passed with 24 votes in favour, 15 against, eight abstentions. Sri Lanka denounced the process as "arbitrary".
Correspondents say that the US has become increasingly frustrated by Sri Lanka's approach to the rights issue.
In 2010 the European Union withheld trade preferences to Sri Lanka over its perceived failure to address human rights concerns.
In a statement, Sri Lanka's mission to the UN said the vote was a "selective and arbitrary process".
"The obvious reality is that voting at the Human Rights Council is now determined not by the merits of a particular issue but by strategic alliances and domestic political issues in other countries which have nothing to do with the subject matter of a Resolution," the statement said.
Sri Lanka's army defeated the separatist Tamil Tigers in May 2009, putting an end to 26 years of brutal civil war - but the final phase of that war has been a source of considerable controversy, with both sides accused of war crimes.
The resolution tabled by the US:
asks the government to explain how it will address alleged violations of international humanitarian law
asks how Sri Lanka will implement the recommendations of an internal inquiry into the war
encourages the UN human rights office to offer Sri Lanka advice and assistance and the government to accept such advice
But there have been unconfirmed reports the text was revised during the proceedings. Among the countries voting in favour of the resolution were Belgium, the US and India. China and Russia were among nations which supported Sri Lanka and opposed the resolution.
India's support for the motion is likely to cause diplomatic tensions, analysts say.
Thousands of people in Sri Lanka, including some religious clerics and former military officers, have taken part in marches to protest against the resolution in recent weeks.
Campaign against 'traitors'
The vote comes amid a government campaign against what it calls "traitors", which has targeted journalists and human rights workers.
State television is using long slots in its Sinhala-language bulletins to denounce Sri Lankan journalists, some now in exile but some still in the country, who it says are helping the defeated Tamil Tiger rebels or "betraying the motherland".
Those based in Sri Lanka are not named but the TV repeatedly zooms in on thinly disguised photographs of them, promising to give their names soon and "expose more traitors".
State media have been similarly deprecating human rights workers who are in Geneva for the Human Rights Council session, the BBC's Charles Haviland in Colombo says.
A local organisation, the Free Media Movement, has condemned the broadcasts as "highly unethical". Such state broadcasts have in the past resulted in violent attacks on some accused people.
The Sri Lankan government commissioned its own investigation into the war last year.
Its Lessons Learnt and Reconciliation Commission (LLRC) cleared the military of allegations that it deliberately attacked civilians. It said that there had been some violations by troops, although only at an individual level.
But another report commissioned by the UN secretary general reached a different conclusion, saying that allegations of serious rights violations were "credible" on both sides.
Rights groups Amnesty International described it as "a vital step forward for the country and for international justice".
Human rights groups estimate that up to 40,000 civilians were killed in the final months of the war. The government recently released its own estimate, concluding that about 9,000 people perished during that period.