Brown received a three-year, $10.5 million contract, with $4 million guaranteed, according to multiple media reports.
Despite making just five starts last season, Brown's total of 537 rushing yards and six rushing touchdowns led the team. He averaged 5.3 yards on 102 carries, tied with Carolina Panthers quarterback Cam Newton for the second-best mark among NFL players with at least 100 carries. Only Arizona Cardinals running back Andre Ellington had a better per-carry average, 5.5 yards.
Brown also made 27 receptions for 214 yards and two touchdowns in 2013, all the top figures of his five-year pro career.
The move to San Diego reunites Brown with Chargers general manager Tom Telesco, who was part of the Colts' front office when Indianapolis chose Brown with the 27th overall pick in the 2009 draft out of Connecticut.
Brown's top rushing season with the Colts was 2011, when he ran 134 times for 645 yards and five touchdowns.
In San Diego, Brown joins a backfield that includes running backs Ryan Mathews and Danny Woodhead.