The Pickens County Courthouse in the center of Carrollton was erected in 1877. The first courthouse in Carrollton was burned on April 5, 1865 by troops of Union General John T. Croxton. A second courthouse was also destroyed by fire on November 16, 1876. Though arson was suspected, no arrest was made until January 1878 when Henry Wells, an African Americanwith a criminal history, was arrested and locked in the garret of the new building. According to legend, Wells was peering down from the north garret window with his face pressed against the glass as a mob gathered to hang him. A bolt of lightning struck the window pane and "etched" Wells' anguished face into the window glass. A vague image resembling a face can still be seen in that lower right-hand pane of that window.
Carrollton is located at(33.260858, -88.094452).
According to the U.S. Census Bureau, the town has a total area of 2.1 square miles (5.4 km2), of which 2.1 square miles (5.4 km2) is land and 0.04 square miles (0.10 km2) (0.96%) is water.
The Pickens County Board of Education is located in Carrollton. Students in the area can attend Carrollton Elementary School for Grades K-6. Carrollton is also the home of the Pickens County Educational Center, a branch of Bevill State Community College.