During Monroe's last two years as president the struggle over the succession degenerated into what could be called the Era of Bad Feelings. Although Monroe was not a candidate, he was subjected to criticism—often of a petty nature. Crawford, Clay, and Jackson all saw it to their advantage to oppose administration policies. Adams and Calhoun (who withdrew from the campaign early in 1824) remained loyal to Monroe and restrained their supporters. The Crawfordites were especially bitter, since they felt that Monroe owed a particular debt to Crawford for not opposing him in 1816. Monroe remained neutral but the impression prevailed that he preferred Adams.
It was a combination of congressional supporters of Jackson and Crawford who raised questions impugning the president's integrity in the management of the so-called Furniture Fund, money appropriated in 1817 and 1818 for the refurnishing of the White House. The investigation was handled in such a way as to leave a cloud of suspicion, although it was apparent that the only error had been inadequate bookkeeping by the agent Monroe engaged to manage the fund.
The Crawfordites managed to generate considerable embarrassment for the president over the discovery that Ninian Edwards, a Calhoun supporter, had been the author of the "A.B. Letter," which questioned Crawford's management of the Treasury. The subsequent investigation, controlled by Crawford's friends, left the basic issues unanswered but placed the administration in the position of prodding Edwards, just appointed the first minister to Mexico, to resign. A further unpleasantness, stirred up by the Georgia delegation, was aimed at Calhoun but involved an attack on Monroe for refusing to force the Cherokee to agree to land cessions stipulated in earlier treaties.
After the harassments of his last two years in office, it was with a sense of relief that Monroe relinquished the office to Adams in March 1825, happy to retire to Oak Hill and the life of a country gentleman, which he so much loved. He stayed aloof from the political squabbles of the day in spite of all efforts to involve him. He busied himself with the affairs of the University of Virginia, Jefferson's cherished educational project, attending the meetings of the Board of Visitors and serving as rector. Visits to Charlottesville were occasions of joyous reunions with Madison, the two being drawn together in an even closer bond after Jefferson's death in 1826. Monroe's last public service was as a member of the Virginia constitutional convention of 1829, also attended by Madison. Monroe was chosen president but was too feeble to preside, although he did speak on several occasions.
After Monroe's retirement his most pressing concern was to lift the heavy debt, now amounting to $75,000, which had been accumulating since his first mission to France. The depressed state of Virginia land values made it impossible for him to sell Highlands. His efforts to obtain recompense for expenses of his past diplomatic missions (his accounts had never been settled with the State Department) were frustrated by the opposition of Jacksonians and Crawfordites. Finally, in February 1831, as news of the former president's financial plight became generally known, Congress appropriated $30,000 in settlement of his claims. The Bank of the United States took over Highlands in lieu of a $25,000 debt.
The death of Monroe's wife early in 1830 prostrated him with grief; rarely had they ever been separated since their marriage. Monroe's health began to fail so rapidly that he moved to New York to live with his younger daughter, Mrs. Samuel L. Gouverneur. Oak Hill was put up for sale to pay the balance of his debts. Sadly he notified Madison in April 1831 that he would not be able to attend the meeting of the Board of Visitors. When Adams saw his predecessor at that time, he found Monroe extremely weak but nonetheless anxious to discuss the recent revolutions in Europe. On 4 July 1831—the fifth anniversary of the deaths of John Adams and Thomas Jefferson—Monroe died. The funeral, which took place in New York City, was attended by state and civic officials. Vast crowds lined the streets as the cortege made its way to the cemetery. Throughout the country his passing was observed by days of mourning, memorial services, and eulogies, the most moving of which was delivered in Boston by John Quincy Adams. In 1858, Governor Wise of Virginia planned to have Jefferson, Madison, and Monroe reburied in Hollywood Cemetery in Richmond, but only Monroe's remains were reinterred.