The principal facade of the White House, the north front, is of three floors and eleven bays. The ground floor is hidden by a raised carriage ramp and parapet, thus the facade appears to be of two floors. The central three bays are behind a prostyle portico (this was a later addition to the house, built circa 1830) serving, thanks to the carriage ramp, as a porte cochere. The windows of the four bays flanking the portico, at first floor level, have alternating pointed and segmented pediments, while at second floor level the pediments are flat. The principal entrance at the centre of the portico is surmounted by lunette fanlight. Above the entrance is a sculpted swag in relief. The roofline is hidden by a balustraded parapet.

The mansion's southern facade is a combination of the Palladian and neoclassical styles of architecture. It is of three floors, all visible. The ground floor is rusticated in the Palladian fashion. At the centre of the facade is a neoclassical projecting bow of three bays. The bow is flanked by 5 bays, the windows of which, as on the north facade, have alternating segmented and pointed pediments at first floor level. The bow has a ground floor double staircase leading to a Doric colonnaded loggia (with a balcony at second floor level), known as the south portico. The more modern third floor is hidden by a balustraded parapet and plays no part in the composition of the facade.

Following his April 1789 inauguration, President George Washington occupied two executive mansions in New York City: the Samuel Osgood mansion at 3 Cherry Street (April 1789 – February 1790), and the Alexander Macomb mansion at 39–41 Broadway (February – August 1790). The July 1790 Residence Act named Philadelphia, Pennsylvania the temporary national capital for a 10-year period beginning December 1790, while the Federal City was under construction. Philadelphia rented Robert Morris's mansion at 190 High Street (now 524-30 Market Street) for Washington's presidential residence, while Pennsylvania built a presidential palace several blocks away in a futile effort to have Philadelphia named the permanent national capital. Washington occupied the Market Street mansion from November 1790 to March 1797, and altered it in ways that may have influenced the design of the White House. President John Adams also occupied the Market Street mansion, declining the presidential palace (which was bought by the University of Pennsylvania).

The President's house was a major feature of Pierre (Peter) Charles L'Enfant's's plan for the newly established federal city, Washington, D.C. The architect of the White House was chosen in a design competition, which received nine proposals, including one submitted anonymously by Thomas Jefferson. The nation's first president, George Washington, traveled to the site of the federal city on July 16, 1792, to make his judgment. His review is recorded as being brief, and he quickly selected the submission of James Hoban, an Irishman living in Charleston, South Carolina. Washington was not entirely pleased with the original Hoban submission, however; he found it too small, lacking ornament, and not fitting the nation's president. On Washington's recommendation, the house was enlarged by thirty percent; the present East Room, likely inspired by the large reception room at Mount Vernon, was added.

The building Hoban designed is verifiably influenced by the first and second floors of Leinster House, in Dublin, Ireland, which later became the seat of the Oireachtas (the Irish parliament). Several other Georgian era Irish country houses have been suggested as sources of inspiration for the overall floor plan, details like the bow-fronted south front, and interior details like the former niches in the present Blue Room. These influences, though undocumented, are cited in the official White House guide, and in White House Historical Association publications. The first official White House guide, published in 1962, suggested a link between Hoban's design for the South Portico, and Château de Rastignac, a neoclassical country house located in La Bachellerie in the Dordogne region of France and designed by Mathurin Salat. The French house was built 1812–1817, based on an earlier design. The link has been criticized because Hoban did not visit France. Supporters of a connection posit that Thomas Jefferson while visiting the École Spéciale d'Architecture (Bordeaux Architectural College) in 1789 viewed Salat's drawings, and on his return to the U.S. shared the influence with Washington, Hoban, Monroe, and Benjamin Henry Latrobe.

Construction of the White House began with the laying of the cornerstone on October 13, 1792, although there was no formal ceremony. The main residence, as well as foundations of the house, were built largely by enslaved and free African-American laborers, as well as employed Europeans. Much of the other work on the house was performed by immigrants, many not yet with citizenship. The sandstone walls were erected by Scottish immigrants, employed by Hoban, as were the high relief rose and garland decorations above the north entrance and the "fish scale" pattern beneath the pediments of the window hoods. The initial construction took place over a period of eight years, at a reported cost of $232,371.83 ($2.8 million in 2007 dollars). Although not yet completed, the White House was ready for occupancy on or circa November 1, 1800.

Shortages, including material and labor, forced alterations to the earlier plan developed by French engineer Pierre Charles L'Enfant for a "palace" that was five times larger than the house that was eventually built. The finished structure contained only two main floors instead of the planned three, and a less costly brick served as a lining for the stone façades. When construction was finished the porous sandstone walls were coated with a mixture of lime, rice glue, casein, and lead, giving the house its familiar color and name.

As it is a famed structure in America, many replicas of the White House have been constructed.

On Saturday, November 1, 1800, John Adams became the first president to take residence in the building. During Adams' second day in the house, he wrote a letter to his wife Abigail, containing a prayer for the house. Adams wrote:

Theodore Roosevelt had Adams's blessing carved into the mantel in the State Dining Room.

Adams lived in the house only briefly before Thomas Jefferson soon moved in. Despite his complaints that the house was too big ("big enough for two emperors, one pope, and the grand lama in the bargain"), Jefferson considered how the White House might be added to. With Benjamin Henry Latrobe, he helped lay out the design for the East and West Colonnades, small wings that help conceal the domestic operations of laundry, a stable and storage. Today, Jefferson's colonnades link the residence with the East and West Wings.

In 1814, during the War of 1812, the White House was set ablaze by British troops during the Burning of Washington, in retaliation for burning Upper Canada's Parliament Buildings in the Battle of York; much of Washington was affected by these fires as well. Only the exterior walls remained, and they had to be torn down and mostly reconstructed because of weakening from the fire and subsequent exposure to the elements, except for portions of the south wall. Of the numerous objects taken from the White House when it was ransacked by British troops, only two have been recovered. Then-first lady Dolley Madison rescued a painting of George Washington, and in 1939, a Canadian man returned a jewelry box to President Franklin Delano Roosevelt, claiming that his grandfather had taken it from Washington. Most of the spoils were lost when a convoy of British ships led by HMS Fantome sank en route to Halifax off Prospect during a storm on the night of November 24, 1814.

After the fire, President James Madison resided in The Octagon House. Meanwhile, both architect Benjamin Henry Latrobe and Hoban contributed to the design and oversight of the reconstruction, which lasted from 1815 until 1817. The south portico was constructed in 1824 during the James Monroe administration; the north portico was built six years later. Though Latrobe proposed similar porticos before the fire in 1814, both porticos were built as designed by Hoban. The similarity between the South Portico and an elliptical portico, with nearly identical curved stairs at Château de Rastignac in La Bachellerie, France is speculated as the source of inspiration, although this matter is one of great debate. Italian artisans, brought to Washington to help in constructing the U.S. Capitol, carved the decorative stonework on both porticos. Contrary to speculation, the North Portico was not modeled on a similar portico on another Dublin building, the Viceregal Lodge (now Áras an Uachtaráin, residence of the President of Ireland), for its portico postdates the White House porticos' design. For the North Portico, a variation on the Ionic Order was devised incorporating a swag of roses between the volutes. This was done to link the new portico with the earlier carved roses above the entrance.

By the time of the American Civil War, the White House had become overcrowded. The location of the White House was questioned, just north of a canal and swampy lands, which provided conditions ripe for malaria and other unhealthy conditions. Brigadier General Nathaniel Michler was tasked to propose solutions to address these concerns. He proposed abandoning the use of the White House as a residence and designed a new estate for the first family at Meridian Hill in Washington, D.C., but Congress rejected the plan.

When Chester Arthur took office in 1881, he ordered renovations to the White House to take place as soon as the recently widowed Lucretia Garfield moved out. Arthur inspected the work almost nightly and made several suggestions. Louis Comfort Tiffany was asked to send selected designers to assist. Over twenty wagons of furniture and household items were removed from the building and sold at a public auction. All that was saved were bust portraits of John Adams and Martin Van Buren. A proposal was made to build a new residence south of the White House, but it failed to gain support. In the fall of 1882 work was done on the main corridor, including tinting the walls pale olive and adding squares of gold leaf, and decorating the ceiling in gold and silver, and colorful traceries woven to spell "USA". The Red Room was painted a dull Pomeranian red, and its ceiling was decorated with gold, silver, and copper stars and stripes of red, white, and blue. A fifty-foot jeweled Tiffany glass screen, supported by imitation marble columns, replaced the glass doors that separated the main corridor from the north vestibule.

In 1891, First Lady Caroline Harrison proposed extensions to the White House, including a National Wing on the east for an historical art gallery, and a wing on the west for official functions. A plan was devised by Colonel Theodore A. Bingham, which reflected the Harrison proposal. In 1901, Theodore Roosevelt and his family moved in to the White House and hired McKim, Mead, and White to carry out renovations and expansion, including the addition of a West Wing. President William Howard Taft enlisted the help of architect Nathan C. Wyeth to add additional space to the West Wing, which included the addition of the Oval Office.

The West Wing was damaged by fire in 1929, but rebuilt during the remaining years of the Herbert Hoover presidency. In the 1930s, a second story was added, as well as a larger basement for White House staff, and President Franklin Roosevelt had the Oval Office moved to its present location: adjacent to the Rose Garden.

Decades of poor maintenance, the construction of a fourth story attic during the Coolidge administration, and the addition of a second-floor balcony over the south portico for Harry Truman took a great toll on the brick and sandstone structure built around a timber frame. By 1948, the house was declared to be in imminent danger of collapse, forcing President Truman to commission a reconstruction and move across the street to Blair House from 1949 to 1951. The work, done by the firm of Philadelphia contractor John McShain, required the complete dismantling of the interior spaces, construction of a new load-bearing internal steel frame and the reconstruction of the original rooms within the new structure. Some modifications to the floor plan were made, the largest being the repositioning of the grand staircase to open into the Entrance Hall, rather than the Cross Hall. Central air conditioning was added, as well as two additional sub-basements providing space for workrooms, storage, and a bomb shelter. The Trumans moved back into the White House on March 27, 1952. While the house's structure was kept intact by the Truman reconstruction, much of the new interior finishes were generic, and of little historic value. Much of the original plasterwork, some dating back to the 1814–1816 rebuilding, was too damaged to reinstall, as was the original robust Beaux Arts paneling in the East Room. President Truman had the original timber frame sawed into paneling; the walls of the Vermeil Room, Library, China Room, and Map Room on the ground floor of the main residence were paneled in wood from the timbers.

Jacqueline Kennedy, wife of President John F. Kennedy (1961–63), directed a very extensive and historic redecoration of the house. She enlisted the help of Henry Francis du Pont of the Winterthur Museum to assist in collecting artifacts for the mansion, many of which had once been housed there. Other antiques, fine paintings, and improvements of the Kennedy period were donated to the White House by wealthy philanthropists, including the Crowninshield family, Jane Engelhard, Jayne Wrightsman, and the Oppenheimer family. Stéphane Boudin of the House of Jansen, a Paris interior-design firm that had been recognized worldwide, was employed by Mrs. Kennedy to assist with the decoration. Different periods of the early republic and world history were selected as a theme for each room: the Federal style for the Green Room, French Empire for the Blue Room, American Empire for the Red Room, Louis XVI for the Yellow Oval Room, and Victorian for the president's study, renamed the Treaty Room. Antique furniture was acquired, and decorative fabric and trim based on period documents was produced and installed. The Kennedy restoration resulted in a more authentic White House of grander stature, which recalled the French taste of Madison and Monroe. In the Diplomatic Reception Room Jacqueline Kennedy installed an antique “Vue de l'Amérique Nord” wall paper which Zuber et cie designed in 1834. The wallpaper hung before on the walls of a mansion until 1961 when the house was demolished for a grocery store. Just before the demolition, the wallpaper was salvaged and sold to the White House.

The first White House guidebook was produced under the direction of curator Lorraine Waxman Pearce with direct supervision from Jacqueline Kennedy. Sale of the guidebook helped finance the restoration.

As a means of preserving the history of the White House, no substantive architectural changes have been made on the house since the Truman renovation. Since the Kennedy restoration, every presidential family has made some changes to their private quarters of the White House, but the Committee for the Preservation of the White House must approve any modifications to the State Rooms. Aimed at maintaining the historical integrity of the White House, the congressionally authorized committee works with the First Family—usually represented by the First Lady, the White House Curator, and Chief Usher—to implement the family's proposed plans for altering the house.

During the Nixon administration (1969–74), First Lady Pat Nixon refurbished the Green Room, Blue Room, and Red Room, working with Clement Conger, the curator appointed by President Richard Nixon. Mrs. Nixon's efforts brought more than 600 artifacts to the house, the largest acquisition by any administration. Her husband created the modern press briefing room over Franklin Roosevelt's old swimming pool. Nixon added a single-lane bowling alley to the White House basement.

Computers and the first laser printer were added during the Carter administration, and the use of computer technology was expanded upon during the Reagan administration. A Carter-era innovation, a set of solar water heating panels that were mounted on the roof of the White House, was removed during Reagan's presidency. Redecorations were made to the private family quarters and maintenance was made to public areas during the Reagan years. The house was accredited as a museum in 1988.

In the 1990s, Bill and Hillary Clinton refurbished some rooms with the assistance of Arkansas decorator Kaki Hockersmith, including the Oval Office, the East Room, Blue Room, State Dining Room, Lincoln Bedroom, and Lincoln Sitting Room. During the administration of George W. Bush, first lady Laura Bush refurbished the Lincoln Bedroom in a style contemporary to the Lincoln era; the Green Room, Cabinet Room, and theater were also refurbished.

The White House is one of the first government buildings in Washington that was made wheelchair-accessible, with modifications having been made during the presidency of Franklin D. Roosevelt, who needed to use a wheelchair because of his paraplegia. In the 1990s, Hillary Rodham Clinton—at the suggestion of Visitors Office Director Melinda N. Bates—approved the addition of a ramp in the East Wing corridor. It allowed easy wheelchair access for the public tours and special events that enter through the secure entrance building on the east side. The president travels from the White House grounds via motorcade or helicopter. President Dwight D. Eisenhower became the first president to travel by helicopter to and from the White House grounds.

Today the group of buildings housing the presidency is known as the White House Complex. It includes the central Executive Residence flanked by the East Wing and West Wing. The Chief Usher coordinates day to day household operations. The White House includes: six stories and 55,000 ft² (5,100 m²) of floor space, 132 rooms and 35 bathrooms, 412 doors, 147 windows, twenty-eight fireplaces, eight staircases, three elevators, five full-time chefs, a tennis court, a (single-lane) bowling alley, a movie theater, a jogging track, a swimming pool, and a putting green. It receives about 5,000 visitors a day.

The original residence is in the center. Two colonnades—one on the east and one on the west—designed by Jefferson, now serve to connect the East and West Wings, added later. The Executive Residence houses the president's dwelling, as well as rooms for ceremonies and official entertaining. The State Floor of the residence building includes the East Room, Green Room, Blue Room, Red Room, State Dining Room, Family Dining Room, Cross Hall, Entrance Hall, and Grand Staircase. The Ground Floor is made up of the Diplomatic Reception Room, Map Room, China Room, Vermeil Room, Library, the main kitchen, and other offices. The second floor family residence includes the Yellow Oval Room, East and West Sitting Halls, the White House Master Bedroom, President's Dining Room, the Treaty Room, Lincoln Bedroom and Queens' Bedroom, as well as two additional bedrooms, a smaller kitchen, and a private dressing room. The third floor consists of the White House Solarium, Game Room, Linen Room, a Diet Kitchen, and another sitting room (previously used as President George W. Bush's workout room).

The White House and grounds cover just over 18 acres (approximately 7.3 hectares). Before the construction of the North Portico, most public events were entered from the South Lawn, which was graded and planted by Thomas Jefferson. Jefferson also drafted a planting plan for the North Lawn that included large trees that would have mostly obscured the house from Pennsylvania Avenue. During the mid-to-late 19th century a series of ever larger green houses were built on the west side of the house, where the current West Wing is located. During this period, the North Lawn was planted with ornate carpet-style flowerbeds. Although the White House grounds have had many gardeners through their history, the general design, still largely used as master plan today, was designed in 1935 by Frederick Law Olmsted, Jr. of the Olmsted Brothers firm, under commission from President Franklin D. Roosevelt. During the Kennedy administration, the White House Rose Garden was redesigned by Rachel Lambert Mellon. The Rose garden borders the West Colonnade. Bordering the East Colonnade is the Jacqueline Kennedy Garden, which was begun by Jacqueline Kennedy but completed after her husband's assassination. On the weekend of June 23, 2006, a century-old American Elm (Ulmus americana L.) tree on the north side of the building, came down during one of the many storms amid intense flooding. Among the oldest trees on the grounds are several magnolias (Magnolia grandiflora) planted by Andrew Jackson. Michelle Obama planted the White Houses' first organic garden and installed beehives on the South Lawn of the White House, which will supply organic produce and honey to the First Family and for state dinners and other official gatherings.

Like the English and Irish country houses it was modeled on, the White House was, from the start, open to the public until the early part of the 20th century. President Thomas Jefferson held an open house for his second inaugural in 1805, and many of the people at his swearing-in ceremony at the Capitol followed him home, where he greeted them in the Blue Room. Those open houses sometimes became rowdy: in 1829, President Andrew Jackson had to leave for a hotel when roughly 20,000 citizens celebrated his inauguration inside the White House. His aides ultimately had to lure the mob outside with washtubs filled with a potent cocktail of orange juice and whiskey. Even so, the practice continued until 1885, when newly elected Grover Cleveland arranged for a presidential review of the troops from a grandstand in front of the White House instead of the traditional open house. Jefferson also permitted public tours of his house, which have continued ever since, except during wartime, and began the tradition of annual receptions on New Year's Day and on the Fourth of July. Those receptions ended in the early 1930s, although President Bill Clinton would briefly revive the New Year's Day open house in his first term.

The White House remained accessible in other ways; President Abraham Lincoln complained that he was constantly beleaguered by job seekers waiting to ask him for political appointments or other favors, or eccentric dispensers of advice like “General” Daniel Pratt, as he began the business day. Lincoln put up with the annoyance rather than risk alienating some associate or friend of a powerful politician or opinion maker. In recent years, however, the White House has been closed to visitors because of terrorism concerns.

In 1974, a stolen Army helicopter landed without authorization on the White House grounds. Twenty years later, in 1994, a light plane crashed on the White House grounds, and the pilot died instantly. As a result of increased security regarding air traffic in the capital, the White House was evacuated in 2005 before an unauthorized aircraft could approach the grounds.

On May 20, 1995, primarily as a response to the Oklahoma City bombing of April 19, 1995, the United States Secret Service closed off Pennsylvania Avenue to vehicular traffic in front of the White House from the eastern edge of Lafayette Park to 17th Street. Later, the closure was extended an additional block to the east to 15th Street, and East Executive Avenue, a small street between the White House and the Treasury Building. The Pennsylvania Avenue closing, in particular, has been opposed by organized civic groups in Washington, D.C. They argue that the closing impedes traffic flow unnecessarily and is inconsistent with the well-conceived historic plan for the city. As for security considerations, they note that the White House is set much further back from the street than numerous other sensitive federal buildings are.

Prior to its inclusion within the fenced compound that now includes the Old Executive Office Building to the west and the Treasury Building to the east, this sidewalk served as a queuing area for the daily public tours of the White House. These tours were suspended in the wake of the September 11 attacks. In September 2003, they resumed on a limited basis for groups making prior arrangements through their Congressional representatives and submitting to background checks, but the White House remains closed to the public. The White House Complex is protected by the United States Secret Service and the United States Park Police.

NASAMS (Norwegian Advanced Surface to Air Missile System) were used to guard air space over Washington, D.C. during the 2005 presidential inauguration. The same NASAMS units has since been used to protect the president and all air space around the White House.