Alone among American former presidents, Buchanan was denied a pleasant and honorable retirement. His well-known southern sympathies gave credence to ridiculous Republican charges that he had somehow been responsible for the fall of Fort Sumter and for the war itself. Stores exhibited banknotes picturing a red-eyed Buchanan with a rope around his neck and the word Judas written on his forehead. Lincoln's war message of 4 July drew heavily from an inaccurate report by General Scott and unfairly damaged Buchanan's reputation still further. A Senate resolution to condemn Buchanan failed but received wide publicity. Newspapers charged that he had failed to prevent secession and war by not strengthening the fort earlier, had negotiated truces with the enemy, had overruled General Scott by sending the Star of the West instead of the Brooklyn , had vetoed Scott's proposals to reinforce Sumter, had scattered the fleet around the world, and had tried to arm the South. Buchanan's most recent former cabinet officers could have come to his rescue with true accounts, but five of them had accepted positions with Lincoln, the others were frightened by adverse public opinion, and none would say a word in his defense. On successive days, newspapers announced that he was in England selling Confederate bonds and that he was in Pennsylvania plotting with spies. His portrait was removed from the Capitol rotunda to keep it from being defaced, and he was even accused of stealing pictures from the White House and keeping the gifts brought by a Japanese delegation.
At first, the attacks made him violently ill, but he soon recovered and defended himself vigorously. He demolished the charges of Scott in an exchange of public letters and finished his memoirs in 1866. The book refuted the charges of malfeasance, demonstrated the hypocrisy of his accusers, and restored his peace of mind. It also blamed the Civil War primarily on northern radicalism and clearly revealed the greatest weakness of his presidency—his thorough emotional identification with the South and his inability to understand and deal with northern public opinion on the issues that had separated the sections. He died on 1 June 1868 with no regrets and still certain that history would vindicate his memory.