Arlen was born Hyman Arluck, in Buffalo, New York, United States, the child of a Jewish cantor. His twin brother died the next day. He learned thepiano as a youth and formed a band as a young man. He achieved some local success as a pianist and singer and moved to New York City in his early 20s. He worked as an accompanist in vaudeville.[3] At this point, he changed his name to Harold Arlen. Between 1926 and about 1934, Arlen appeared occasionally as a band vocalist on records by The Buffalodians, Red Nichols, Joe Venuti, Leo Reisman and Eddie Duchin, usually singing his own compositions.

In 1929, Arlen composed his first well-known song: "Get Happy" (with lyrics by Ted Koehler). Throughout the early and mid-1930s, Arlen and Koehler wrote shows for the Cotton Club, a popular Harlem night club, as well as for Broadway musicals and Hollywood films. Arlen and Koehler's partnership resulted in a number of hit songs, including the familiar standards "Let's Fall in Love" and "Stormy Weather." Arlen continued to perform as a pianist and vocalist with some success, most notably on records with Leo Reisman's society dance orchestra.

Arlen's compositions have always been popular with jazz musicians because of his facility at incorporating a blues feeling into the idiom of the conventional American popular song.

In the mid-1930s, Arlen married, and spent increasing time in California, writing for movie musicals. It was at this time that he began working with lyricist E.Y. "Yip" Harburg. In 1938, the team was hired by Metro-Goldwyn-Mayer to compose songs for The Wizard of Oz. The most famous of these is the song "Over the Rainbow" for which they won the Academy Award for Best Music, Original Song. They also wrote "Down with Love" (featured in the 1937 Broadway show, Hooray for What!), a song later featured in the 2003 movie Down with Love.

Arlen was a longtime friend and former roommate of actor Ray Bolger who would star in The Wizard of Oz, the film for which "Over the Rainbow" was written.

In the 1940s, he teamed up with lyricist Johnny Mercer, and continued to write hit songs like "Blues in the Night", "Out of this World", "That Old Black Magic," "Ac-Cent-Tchu-Ate the Positive," "Any Place I Hang My Hat Is Home", "Come Rain or Come Shine"" and "One for My Baby (and One More for the Road)" .

Arlen composed two defining tunes which bookend Judy Garland's musical persona: as a yearning, innocent girl in "Over the Rainbow" and a world-weary, "chic chanteuse" with "The Man that Got Away", the latter written for the 1954 version of the movie 'A Star Is Born'.