In the XIV century , Chavaniac, a village with Gallo-Roman origins, became a seigneury belonging to the wealthy Suat family from Langeac. During this period the manor, which was without a doubt originally only a fortified house, was built. The Chavaniac manor became part of the Lafayette family patrimony through the marriage, in 1708, of Edouard Motier de La Fayette and Mary Catherine Suat Chavaniac.
At the end of the XVII century, the fortified house was destroyed in a fire. Rebuilt in 1701, the new building was enlarged to become the castle where the Marquis de Lafayette was born on September 6, 1757. Starting in 1791, during the heart of the French Revolution, the building was modernized by the architect Ambroise Laurent Vaudoyer who was assisted by the painter Albert Ancica. Above the castle’s main entrance, instead of the Lafayette family’s coat of arms, a Phrygian cap can be found. In this regard, the Château de Chavaniac belongs to a very small club of "revolutionary castles".
A detailed description of the castle in 1883 presented the building as, "a vast parallelogram with each side having an average length of 23 meters and a height of about 12 meters. The main facade, facing north, is flanked by two 15 meter bell towers that have a circumference of 20 meters each."
The most significant changes took place later, beginning in 1917, then from 1920 to 1925 (construction of a square tower) following the initiative of the French Heroes Lafayette Memorial Fund later known as Memorial Lafayette of which John Moffat was the chairman. Achille Proy, an architect having worked on historic monuments in the Haute-Loire, was in charge of some of these relatively recent transformations.
Starting in 1917, the Memorial engaged in a long series of restorations and new construction. The roof was redone, foundations were stabilized, a tower was rebuilt and the south wing was built in the rear with a square tower. This recreated, for the new residents, the medieval silhouette that Lafayette intentionally removed in 1791. The interior was oriented around the large living room, new rooms were finished and electricity was supplied from Brioude (1921). A pumping station supplying the castle with water, central heating and a telephone were installed. In addition to these new amenities, the south wing included numerous bedrooms equipped with bathrooms; a real demonstration of the American Way of Life in Chavaniac! The castle welcomed the New World within its old structure while maintaining or even recreating a traditional appearance.
The Château de Chavaniac, modified three times since the fire that destroyed it during the late XVII century, reflects the renovations envisioned by the Suat family in 1701, the Lafayette family in 1790, and the Memorial managed by John Moffat starting in 1920.