Robert E. Lee and His Children
Virginia Louise Lee


The brightest flowers blooming there (Arlington) are my daughters.

R. E. Lee

If the time came when Robert E. Lee felt that he must take a more active part in the rearing of his sons, he was no less concerned about the education of his daughters. With both his chief aim was the development of their characters, particularly in the areas of self-control and devotion to duty. By precept and example he taught his children those virtues he thought best calculated to ensure their happiness here and in the life to come.

With his daughters, however, he was in fact much less strict than with his sons, and even admired a certain amount of sauciness and wilfulness in them, if it did not transcend the bounds of courtesy and obedience. And all the Lee daughters were very much personalities in their own right.

Mary, the oldest of the four, early displayed signs of strong will and individuality. Annie, the second daughter, was never very strong and due to an eye injury suffered as a child, she especially appealed to her father’s tender heart. Eleanor Agnes, called Agnes, was the beauty of the family, and of the girls most often her father’s companion, having many traits of character in common, and congenial in temperament with him. Mildred, the fourth daughter, and youngest child, was rather petted by her father. She was high-spirited and lively and he enjoyed teasing her most of them all because he knew he could always get a swift reaction from her.

He had been ecstatically happy at the birth of his son Custis, “Mr. Boo,” and the advent of his first daughter was even more of a delight to him. It was after the birth of his fourth child and second daughter, Annie Custis Lee, in 1839, that he, then the father of two boys and two girls, and on his first prolonged absence from them, began to realize how much his career deprived him of a normal home life and how much he missed the pleasure of his children’s company.

It was not until 1841 and with five children, for Agnes had been born while Annie was still under two years of age, that Lee, the father, had his own home and family all together for any length of time. The assignment in New York, interspersed by visits home to Arlington for Christmas and new babies,1 lasted for five years. The Mexican War brought orders for active duty and once more the home was broken up, mother and children returning to the old “homeplace,” Arlington, while the father prepared to leave for Texas and the opportunity that was to give him the merited recognition his abilities and character deserved.

From Mexico he wrote long, loving, playful letters to his wife and family that speak so eloquently of his affection for them and of his compassion for the innocent hurt by war, especially the children. At Cerro Gordo he had come upon a Mexican drummer boy whose arm had been shattered, pinned under the body of a dead soldier. His little sister stood by, helpless and crying. Lee had the body removed and the boy taken to the surgeons, then he was off to rejoin the battle.2

The war over, and his conduct and bravery having earned him three promotions, the new Colonel Lee, with a few lines in his face and few grey strands mingling with the black of his hair, came home to another period of peace and normalcy.3 In 1849 he was sent to build up the port of Baltimore and the family settled down for three years.

Mary and Annie had been taught at home by their mother, but in 1847 a governess had been engaged for them. When they moved to Baltimore they went to school, but the winter of 1850-51 Annie and Agnes remained at Arlington after Christmas as they liked it there much better and continued studying under a governess.

In 1852 Colonel Lee was appointed Superintended of the United States Military Academy at West Point, a position which he did not want to accept as he did not feel qualified to undertake it.4 It was an honor to be selected, however, and an added inducement was the presence of Custis, already a cadet there.

Annie and Agnes at Arlington, the busy superintendent, with some of his children around him, did not forget his absent daughters. He wrote,

My Precious Annie, my limited time does not diminish my affection for you, Annie, nor prevent my thinking of you and wishing for you. I long to see you through the dilatory nights. At dawn when I rise, and all day, my thoughts revert to you, in expressions that you cannot hear or I repeat.5

He gives her some advice about always paying attention to all her duties, and tells her to be sure and stand straight now that she is growing so tall, and to tell Agnes the same.

Mary greatly enjoyed her position as assistant hostess. The Commandant held open house on Saturday afternoons as one means of getting to see Custis, who kept to his room in the barracks more than his mother liked, and also to get to know personally the caliber of the young cadets. Rob, Jr., thirteen at the time, was usually there and said it was pitiful to see how stiff and miserable the cadets were on these occasions, in awe of the superintendent, but that his father, with his wonderful genial air and pleasant manners, always succeeded in soon putting them at ease.6

In the summer of 1853 the superintendent went home to Virginia on vacation. Mrs. Custis had died the previous April, and it grieved Robert almost as much as it did her daughter.7 She had loved and respected her steady son-in-law, and he had reciprocated her affection. Freeman suggests that this grief, plus the experience of the Mexican War and the heavy sense of responsibility he felt for the many young men at West Point, deepened his religious impulses. At any rate during this vacation, Lee, with his two older daughters, Mary and Annie, was confirmed in July at Christ Church, Alexandria.8

Three years at the academy were ended when Colonel Lee was transferred to a line position by his assignment as Lieutenant Colonel of the new Second Cavalry with orders to report to Louisville, Kentucky.9 Mrs. Lee and her daughters returned to Arlington, with Rob, who soon left there for boarding school. Rooney was at Harvard and Custis was on active duty in Florida.

The following autumn Mrs. Lee made arrangements for Annie and Agnes, her two middle daughters, to enroll in the Virginia Female Institute at Staunton.10 Mary was usually away on extended visits to friends and relatives. This left the mother bereft of all her children except Mildred, and Rob on weekends. Her health had worsened and by this time she was practically an invalid although she kept the news of her condition from her husband, who had enough hardships to sustain in Texas, where his regiment had been sent on frontier duty.

Conditions there were anything but pleasant and the chief occupation of the troops was chasing roving bands of Indians who disappeared into thin air. The officers and men were unhappy in this duty, undergoing as they did needless trials, for every expedition resulted in frustration as they wily Indians evaded their pursuers. The temperature was over one hundred degrees and there was much sickness.

Colonel Lee wrote often and cheerfully to his family, trying to make the best of a bad situation, as became his character. But playful descriptions of scenes and people alternated with deeper feelings as his sense of futility at the life he was leading at times overcame his sanguine disposition. More and more his correspondence conveys that spirit of total, unquestioning faith in God, and man’s utter dependence on Him, that was to develop in him through the years.11 It was because of this faith, apparent to those about him, that he was asked by a sergeant to read the funeral service over his little boy, the second time Colonel Lee had so officiated.

It was during this unhappy period of service in Texas that we can again see something of the inheritance and training Robert E. Lee received from his parents. When “Light-Horse Harry” Lee had gone through a similar period of depression and frustration with his army career he had refused to listen to reason, resigned, and so ruined his life. Robert, faced with the same situation, carefully weighed the pros and cons, and fortified by faith, endured. Optimism had been another of Harry Lee’s outstanding characteristics and it was repeated in the sanguine disposition of his own son. But where its abuse by one had led to disaster, in the other the steadying power of the religion implanted by his mother helped him to keep the forces of despair and hope in proper balance. Thus he avoided extremes of any kind and by adopting a policy of moderation in all things he disciplined until he had refined almost to the point of non-existence those innate elements so disruptive of peace of mind.

It is in this process of subjugating, by faith and reason, the natural instincts of man as an animal and developing the higher instincts of man as an obedient son of God that Robert E. Lee assumes heroic proportions. For we must admit, without in any way sacrificing our own mental balance, that few men have succeeded as well as he did in that respect. To see him thus and admire him for it is not blind hero worship. Rather it is, as Bradford says, an incentive, a support to us weaker mortals, that, being human like us he overcame much, so, strengthened by his example, we can overcome much.12

As he drew closer to his heavenly Father, he became a less heavily paternal, moralizing parent and more and more a kindly, loving father, thinking only of his children and what he could do for them. His changing attitude is reflected in his letters to his sons at this time in which he defers to their opinion and seeks advice instead of giving it.13 His letters to his daughters, as always, were full of love and bantering, with now and then a brief reference revealing the depth of his depression. It is in this time of discouragement and monotony that his family assumes, next to God, the central place in his life.

Disappointed in the army, lonely, bored, his only consolation his faith, he seems then to have fully realized how much of love and security and joy in his children his career had deprived him of. He had always wished to be with them and especially to oversee their training, but had accepted separation as inevitable because of his profession. Now he wondered if it had been worth it and he wished to be with his children because of his own need, not theirs alone. He came to believe that of all things on earth a home and family life were the most important and the most soul satisfying. As a child he had not experienced them due to circumstances beyond his control; now he had a home and family yet here he was far away and both he and they suffered because of it. From then on, it was Robert E. Lee’s deepest, perhaps main, desire to have his family always around him. Beyond even a Virginian’s almost fanatic devotion to family, Robert Lee loved his.

It was because this was so that after the Seven Days battles the triumphant commander of the victorious Army of Northern Virginia, at a brief reunion with his wife and children, could “act the same loving father to us all . . . as if our comfort and happiness were all he had to care for.” It was because this was so that he found time from all his multiple duties to write to them, not briefly or hastily, but fully and tenderly, as if all other considerations, including war, were secondary. If he gave his family all the stopped-up love of a lonely childhood the reciprocal affection of his children compensated for the loving tenderness an absent father, an ill and harassed mother, and scattered brothers and sisters had not given a sensitive, emotional, warm-hearted young boy.

Texas was hard, boring, irksome duty and if Colonel Lee sometimes felt depressed it was usually only temporary and he consoled himself with the philosophy he expressed to Agnes, “We make a great deal of our own happiness and misery in this world, and we can do more for ourselves than others can for us. You must expect discomforts and annoyances all through life. No place or position is secure from them, and you must make up your mind . . . to bear them.”14

Constantly thinking of others in spite of his own feelings, everywhere he went he kept his eyes open for a yellow cat for Mr. Custis. Much of the subject matter of his letters is devoted to cats and he told the story of a French couple, who, coming out to greet him, were followed by a stately procession of cats, of grave visage and erect tails. French-educated, they immediately set to when the claret was set out on the table or his refreshment. He told Mildred that Jim Nooks, a bottle-fed cat, had died of apoplexy, an end he had foreseen, for Jim had coffee and cream for breakfast, pound cake for lunch, turtle and oysters for dinner, buttered toast for tea, and Mexican rats, taken raw, for supper. “But I saw ‘cats is cats’ in Sarassa,” he wrote, referring to the French cats of Mr. and Mrs. Monod. Mildred was now eleven years old and he continues in the same letter, “You must be a great personage now — sixty pounds! I wish I had you here in all your ponderosity. I want to see you so much. Can you not pack up and come to the Comanche country?”15

So the years passed, in arduous duty, court-martial cases, changes of station. In July, 1858, came a change for the better when Albert Sidney Johnston, Colonel of Lee’s regiment, was transferred to Washington. Lee assumed command and transferred to San Antonio.16 He had an entire house for his quarters and found friends there, particularly a Major Chilton, paymaster. The major’s two little daughters, Laura and Emmie, took the place of his own beloved daughters and he greatly enjoyed their presence.17

Four months later news of Mr. Custis’s death interrupted this pleasant tour of duty. Mrs. Lee was alone and sick and there was no male member of the family to take charge of affairs. So began the series of leaves from his post that stretched over two years and brought him back to Arlington and the task of restoring it, and to his wife and daughters.

They added to his burdens for Mrs. Lee’s health and that of two of his daughters was so bad that he had to take time to accompany them to the springs for treatment. He nursed them when they were at home, in addition to repairing the buildings and grounds, and managed to keep in touch with army affairs.

If he had not felt that he was losing ground professionally by his continued absence from the army, he might have enjoyed his years at Arlington. In spite of the difficulty of accomplishing repairs, which went slowly due to lack of funds, he was with his four beloved daughters again and found much happiness in their companionship.18 He cared for the ailing ones with tender love and devotion and with the competence of a trained nurse. There were always visitors coming and going at Arlington, and when the girls were better they were off on visits to friends, visits sometimes lasting, in the Virginia fashion, for weeks or months at a time. So there was gayety and joy at Arlington which the convivial Lee found stimulating and which relieved him of some of his worries. Whatever his load he was always cheerful with his daughters. Mrs. Lee’s invalidism prevented her from attending the wedding of their son Rooney in 1859 but his daughters went with him. It took place at Shirley, his mother’s old house, recalling pleasant memories, and was a welcome respite from his cares.

In 1860 he felt he could safely leave Arlington and he returned to his regiment in Texas.

Christmas of 1860, the last Christmas the Lees were to spend at Arlington, brought Mildred and Rob home from boarding school. Custis was home and the three other daughters, so for the last time the old mansion was filled with friends and relatives to celebrate the holidays. While down in the southwest the husband and father was appalled at the turn events had taken after the election of Lincoln. “The country is between a state of anarchy and civil war. May God avert both of these evils from us,” he wrote home.19

Soon after being promoted to Colonel of the First Cavalry, March 16, 1861,20 with the secession of Texas, Mississippi, Florida, Georgia and Alabama an accomplished fact, Lee was recalled to Washington. In a matter of weeks the civil war he had intimated would occur was also a fait accompli, and the Virginian Lee went with his state.21

He was anxious for his wife and daughters left behind in the Federal capital and in several letters from Richmond suggested their early removal to a safer place. He warned Mrs. Lee that the war might last ten years and to be prepared for such an eventuality.22

The record of the next four years is one of constant traveling about for all members of the family making it extremely difficult to maintain an orderly and coherent account of their activities in relation to each other. This is particularly true in the case of the daughters who were often in widely separated places and who so frequently changed their location, like birds, that even Douglas Southall Freeman was finally reduced to using the verb “flutter” as best approximating their movements.

Mrs. Lee went to her old haven, Ravensworth in Fairfax County, while Annie and Agnes found refuge in the home of a cousin of their mother, Dr. Stuart, in King George County. Mildred was safe in school at Winchester and Mary stayed with friends in Fauquier.

With the daughters visiting from place it was difficult for their father to keep up with them but under the pressure of preparing a defenseless Confederacy for war, hindered by an inept government, he wrote as often as he could. To Mildred at Mrs. Powell’s School, he wrote informing her he now had a beautiful white beard that was much admired.23 A week later he wrote to his “darling daughters” at Clydale, thanking them for their letters and regretting his lack of time in which to reply. He begged that they would not stop writing to him on that account. His only wish, he said, was to see them, be with them, and never part from them again.24

The first Christmas of the war the family was scattered everywhere. From South Carolina he sent one of his daughters as a Christmas gift some money and “some sweet violets I gathered for you this morning, while covered with white dense frost, whose crystals glittered in the bright sun like diamonds, and formed a brooch of rare beauty and sweetness. . . .”25

Later, more in his old teasing, jesting style he writes Mildred on her sixteenth birthday, “And are you really sweet sixteen? That is charming and I want to see you more than ever. But when that will be, my darling child, I have no idea.” A note of sadness creeps in when he tells her he hopes that after the war the family will all be united and he can spend some years with his children, a wish he often expresses in his letters from this time on. He goes on, “It has been a long time since I have seen you and you must have grown a great deal. I have grown so old, and become so changed that you would not know me. But I love you just as much as ever, and you know how great a love that is.” She must have been on a diet for he warns her not to carry it too far, as he does not admire women who are too thin. He concludes, “I have but little time to write,my (sic) dear daughter. You must excuse my short and dull letters. Write me when you can, and love always your devoted father, R. E. Lee.”26

After five months in South Carolina General lee was back in Richmond and that much nearer his family. But despite their proximity he was not to see his wife or daughters for over a year from the day he had left Arlington. Mrs. Lee, at the White House, was finally forced to move into Richmond where her husband had made arrangements for her convenience. He left the decision to move up to her, warning her not to expect any permanency in the arrangements because of the unsettled state of affairs. In June 1862, Mrs. Lee and Mary came through the federal lines with the gracious assistance of General George McClellan, and at long last saw General Lee again.27

There was a happy reunion in Richmond as Rob came home on furlough and the girls also joined their parents. But it was only temporary. With General Lee off on duty, Mrs. Lee and the daughters went to Hickory Hill for a short visit, then to Warren County Springs in North Carolina to take the waters as well as to relieve General Lee’s worry about their presence in a Richmond filled with constant alarms.

It was here that the first death in the family occurred, for Annie, the delicate one, died there October 20, 1862. She had always been especially dear to her father because of her frail health and injured eye as well as her shy, sweet disposition, but with his customary self-control and devotion to duty, his adjutant, Taylor, tells us that the morning he received the news of her death he first attended to routine army matters before indulging his grief.28

After this sorrow Mrs. Lee wanted to return to Richmond to be near her husband in case he could manage to see her and they could comfort each other in their loss. She left Mildred in school at Raleigh and she and Agnes went to stay in the Richmond home of their friends, the Caskies, as they had no home of their own. From time to time General Lee was able to join them and the happy days spent there did much to relieve his cares.

The second Christmas of the war saw the general away and the family still scattered. After a summer at Hot Springs, Mrs. Lee rented a small, wooden house in Richmond which was too cramped to accommodate the entire family. Mrs. Lee, Agnes and Mildred, home from school, lived there, but Charlotte, Rooney’s wife, like Annie Lee frail and delicate, had to room by herself. This was a source of regret to both Mrs. Lee and General Lee who loved Charlotte as a daughter.29 He came home at intervals and although the house was poorly furnished and inadequate the family held happy reunions there and friends and relatives often came. Social calls and parties occupied some of the girls’ time.30

There was even a prospect that General Lee might be with his family for the third Christmas of the war but he returned to camp on the twenty-first. The joy of the season was marred by the serious illness of Charlotte and on the day after Christmas she died.31 The deaths of his two loved daughters, the wounding and capture of Rooney, and the strain of war all took their toll. Suffering herself, Mrs. Lee was almost desperate about his condition for he had almost constant pain in his left side. His hair and beard were now snow-white, and his once robust health had greatly deteriorated.32

The next few months were comparatively happy. The general came home for frequent consultations at the War Department and mingled as much as his duties and his conscience permitted in the social stream that flowed continuously in and out of his home. Mary, Agnes and Mildred Lee were very much a part of Richmond’s society, and Custis, in the nature of his assignment as President Davis’s aide, also was seen much in society. Mrs. Lee, but now almost totally helpless, was very sociable by nature and imprisoned as she was, more than delighted in her daughters’ activities which drew her somewhat into the current of events, and kept her informed of all that was going on.33

All their energy was not employed in social intercourse, however. Mrs. Lee and her daughters and their female guests knitted industriously, making socks and gloves for the destitute soldiers, “Lee’s Miserables.”34 And some for the commander himself as his grateful letters testify. He wrote Mrs. Lee he could even hear Mildred’s needles rattle as they flew through the meshes, and as he preferred socks with double heels he wanted her to teach Mildred that stitch.35 After visiting Richmond for a few days, he even carried a bag of the socks back to the soldiers himself. During the month of March alone over two hundred pairs of socks were sent to the Stonewall Brigade. The Lee daughters set the example for the other young ladies of the Confederacy who wasted their time on cards and taffy pulls. They faithfully attended church and always accompanied their father there when he was in Richmond.

The letters he wrote to Mildred from Petersburg, in 1864, contained something of the earlier raillery he had employed when writing to his children. He cheerfully hints that Rob is on the high road to matrimony for he had a new uniform from top to toe, a new and handsome horse, and was cultivating a marvelous beard. Once he sent her a song composed by a French soldier to which he expected a reply in French “since she was so versed in that language.”36 He asks her to “give a great deal of love to your dear mother, and kiss your sisters for me. Tell them they must keep well, not talk too much, and go to bed early.”37

In the same tender, loving, gentle way he wrote to his sons during the many separations his life entailed, always reassuring them of his care and love and interest in their welfare, so he wrote, in spite of his own pressures and burdens, to the daughters he adored. Such devotion was later transmitted into action when he was with them, but with them or away from them, his love was never a demanding one, but a sacrificial one, ready to deny self for their good. In the spirit that led him to offer to resign from the army to safeguard his children’s inheritance, was written this paragraph to his wife from Petersburg: “I received . . . the barrel of apples. You had better have kept the latter as it would have been more useful to you than me, and I should have enjoyed its consumption by you and the girls more than by me.”38

Not war, nor privation, nor distance, nor age nor illness could separate Robert E. Lee from the love he had for his children and they for him. A love which such trials only deepened and made more precious, and which was to bring him, in the few years remaining to him after the war, the peace and companionship he had earned at such terrible cost.

Return to Robert E. Lee and His Children