One Man’s Opinion

Dick Nicholas

Volume 13, Number 12
December 2005

We all know about Robert Edward Lee the General. Arguably, as many words have been written about the military exploits and leadership of this Confederate icon during the four years of the Civil War as about other famous Virginians such as George Washington and Thomas Jefferson. As is frequently the case, however, once you get past the focal point of the subject’s career and accomplishments, the information overload diminishes rapidly. Try, for example, to find out something about Jefferson’s family. While whole libraries and endless collections are rightfully devoted to Albemarle’s first citizen and our country’s third president, information about his younger brother Randolph, or some of the rest of the family is almost nonexistent. Scholars and researchers just don’t seem interested in the subject matter.

And so it is with Robert E. Lee and his family. For example, what did the rest of Lee’s family do during the Civil War? We know that his three sons Custis, William Henry Fitzhugh (called “Rooney” to distinguish from his first cousin Fitzhugh Lee), and Robert, Jr., all served honorably in various roles in the Army of Northern Virginia. Custis and Rooney both became generals in the Confederate army, and the youngest Robert left the University of Virginia to become a Captain. Meanwhile, the four daughters maintained their presence on the home front, except for Annie who died early in the war from typhoid fever.

But what about their father’s siblings? A few genealogy and history books tell us that General Lee was the fifth of six children of Henry (“Light-Horse Harry”) Lee and Ann Hill Carter. At the time of the Civil War four were living: Charles Carter (1798–1871), Anne Kinloch (1800–1864), Sydney Smith (1802–1869), and Robert Edward (1807–1870). Robert also had a half sister Lucy Grymes (1786–1860) and a half brother Henry (1787–1837) from his father’s first marriage to Matilda Lee. The younger Henry Lee following a scandal was distinguished from his famous father by the uncomplimentary sobriquet of “:Black-Horse Harry.” He died in disgrace and exile in Paris in 1837.

Charles Carter Lee, the oldest brother, graduated from Harvard and became a lawyer in Washington about 1820. However, Carter Lee had a hard time settling into any sort of a career until he married Lucy Penn Taylor in 1847 and settled down on her property called “Windsor Forest” in Powhatan County near Fine Creek Mills. He was too old for military service in the Civil War and spent the rest of his life tilling the soil. General Lee stopped there to spend the night April 14, 1865, on his sorrowful horseback ride from the Appomattox surrender to his rented home in Richmond. The two brothers spent most of the evening talking, but when bedtime came, the old general insisted on staying in his tent set up in the yard. It was to be the last time that he would spend the night under those circumstances. He also visited his brother several times later that summer when he and Mrs. Lee were living at nearby “Derwent.”

Charles had several children and a young daughter Mildred (also known as “Powhattie,” probably after her home county of Powhatan) spent some time with her uncle Robert, then President of Washington College, while attending school in Lexington in the winter of 1867–68. The Lexington home must have been full during those years because two of Robert’s nephews, George a son of Charles, and Smith’s son Robert also stayed there while they attended Washington College

While Charles Carter Lee had no active involvement in the Civil War, it was a different story for Sidney (sometimes spelled Sydney) Smith Lee. The second son and brother of Robert E. Lee was known in his youth as “Rose,” and in his prime as “Smith.” He and his younger brother were very close, and the two communicated frequently throughout the years. He was the best man for Robert when the latter married Mary Anna Randolph Custis in 1831.

Photographs of Smith Lee show him to look much like his younger brother. A Confederate soldier described him as “a splendid-looking man with a heavy black mustache,” and he was reputed to be the most handsome of the Lee brothers. Another report said that he was “almost as handsome as Robert and of fine, cordial manners.” Though much alike physically, the two brothers were, nonetheless quite different in other ways. Whereas Robert E. Lee was often described as stately and dignified, almost classical in looks and demeanor, Robert E. Lee, Jr., painted a different picture of Smith Lee: “No one could be near my Uncle Smith without feeling his joyful influences.”

Thanks to an appointment by President Monroe, Smith Lee left home in 1820 and entered the Navy as a midshipman. He eventually rose to the rank of captain, and commanded a ship in Commodore Matthew C. Perry’s historic expedition to Japan in 1853. In an amazing coincidence, Smith was commandant of the U.S. Naval Academy at the same time that his brother was superintendent of the U.S. Military Academy.

In another notable happenstance, he and his younger brother Captain Robert E. Lee participated in the Battle of Vera Cruz, Mexico, in 1847. Although primarily a land operation, General Winfield Scott borrowed three 64-pounder shell guns and three 32-pounders from the accompanying fleet to assist in the bombardment of the Mexican positions. Lieutenant Smith Lee was assigned to one of the guns in the naval battery, and the two brothers fought their first and only action together. Robert wrote home about the warlike scene: “I stood by his gun when I was not wanted elsewhere. . . . I . . . am at a loss what I should have done had he been cut down before me.”

After informing Francis P. Blair and General Winfield Scott in Washington on April 18, 1861, that he could not take up arms against his native state, a sorrowful Robert E. Lee agonized about what course of action he should take. As his brother Smith was on duty in Washington, the younger Lee stopped to discuss the question with him. They could come to no immediate conclusion on it and parted in the expectation of meeting again before either of them took any action. It was not to be. As Douglas Southall Freeman put it: “:Dearly Lee loved the Union, anxious as he was to see it preserved, he could not bear arms against the South. All Lees had been Americans, but they had been Virginians first.”

There was one notable defection among the Lee siblings concerning the family’s allegiance to Virginia and the Confederacy. Robert’s older sister Ann Kinloch had married William Louis Marshall, a clergyman, lawyer, and jurist in Baltimore. Marshall was Unionist in his sympathies and her son Louis was a captain in the United States Army at the beginning of the war, and later promoted to colonel. Although Ann sided with her husband and son, Lee took the time to write her an explanatory letter at the time he made his decision to go with the Confederacy. Even though the siblings were on opposite sides in the conflict, Ann obviously retained affection for her brother, sending a message saying that she doubted the Federals “can whip Robert.”

A year later during the Second Manassas campaign Rooney Lee’s cavalry command narrowly missed capturing his first cousin Louis Marshall at Pope’s headquarters. it had been a bitter pill to Lee that his nephew served on Pope’s staff, and Lee later commented “I’m sorry to find him in such bad company.” In a letter to his wife he reinforced his feelings about his nephew by saying: “I could forgive [him] for fighting against us, but not his joining Pope.”

After a 40-year career in the United States Navy Smith Lee, like his brother, resigned his commission to defend his home state in the newly formed Confederate States of America. Initially Captain Sidney Smith Lee was named to a joint commission of army and navy officers. The commission’s assignment was “to name all efficient and worthy Virginians and residents of Virginia in the Army and Navy of the United States; for the purpose of inviting them into the service of Virginia.” Later he commanded the James River Squadron and the inter-service command at Drewry’s Bluff, then left active command to be chief of the Office of Orders and Detail in Richmond. His performance was apparently satisfactory in all these positions, although a young ambitious captain in charge of the artillery at Drewry’s Bluff complained about being “commanded by an octogenarian imbecile Capt. Sidney Smith Lee, brother of the General but utterly unlike him.”

Smith Lee lived at “Richland” near Aquia Creek in Stafford County on the Potomac River and died there in 1869. Robert heard the news of his death while he and Mrs. Lee were at Rockbridge Baths, and much to his regret arrived in Alexandria a day late for the funeral. The two were devoted brothers and close friends and had kept in touch throughout their lives. The loss of his brother was a great sorrow to Robert.

Like his younger brother, Smith Lee sent three sons into the army including Fitzhugh (better known as “Fitz”), who graduated from West Point in 1856. After the war, General Fitz Lee tried farming at his father’s old home at “Richlands,” but soon turned to politics. He was elected governor of Virginia in 1885, and in 1898, in spite of his corpulent dimensions (he weighed over 300 pounds!), came out of retirement to be commissioned as a major general of volunteers during the Spanish American War. He and Joe Wheeler were the only two Confederate generals to again serve as a general in the U.S. Army. He died in 1905 in Washington at age 70, and was buried in Hollywood Cemetery in Richmond.