The sentence follows months of tension between Cambodia's two main political parties, the Cambodian People's Party of Prime Minister Hun Sen and the opposition Cambodia National Rescue Party (CNRP).
Hun Sen has been Cambodia's leader for three decades but his grip on power was shaken in a 2013 general election when the CNRP won 55 seats in the National Assembly, leaving the Cambodian People's Party with 68 seats in the 123-seat assembly.
Members of the opposition have complained of a crackdown by the government and its allies in a bid to intimidate critics before a general election in 2018.
The Phnom Penh Municipal Court found Sam Rainsy and two members of his social media team, Ung Chung Leang and Sathya Sambath, guilty of citing the false 1979 border treaty.
The fake treaty, which they posted on Facebook, purported to show Vietnam and Cambodia agreeing to get rid of their mutual boundary.
Presiding Judge Leang Samnath sentenced Sam Rainsy to five years in prison and Ung Chung Leang and Sathya Sambath to three years, all in absentia.
"The court orders the arrests of Ung Chung Leang, Sathya Sambath and Sam Rainsy to serve these sentences," Judge Leang Samnath told the court.
Cambodia has fretted for centuries about its much bigger neighbors, Vietnam to the east and Thailand to the northwest, encroaching on its territory. The issue remains emotive and many Cambodians are suspicious of both countries.
Sam Rainsy has been living in France since 2015 to avoid arrest in a separate defamation case. He did not respond to a request for comment on Tuesday but said on Twitter the case against him and the two others had been "fabricated" by a "kangaroo court".
The CNRP said it had no knowledge of the whereabouts of Ung Chung Leang and Sathya Sambath.