A. Bartlett Giamatti, the Renaissance scholar and former president of Yale University who gave up a brilliant academic career in 1986 to join the rough-and-tumble of big league baseball, died after apparently suffering a heart attack at his summer home on Martha's Vineyard yesterday.

The 51-year-old Dr. Giamatti, who became commissioner of baseball last spring after more than two years as president of the National League, collapsed at midafternoon at his home in Edgartown, Mass., and was pronounced dead at Martha's Vineyard Hospital in Oak Bluffs.

''He was found unconscious and in full cardiac arrest,'' said Steven Mac-Kinnon, the deputy police chief in Edgartown. He said officers who rushed to his home administered cardiopulmonary resuscitation and took him to the hospital, where doctors tried for more than an hour to revive him. Banned Rose

He was pronounced dead at 4:32 P.M., said a hospital spokeswoman, Linda Brantley, who added that his wife and a son, Paul, were with him. While a heart attack was believed to be the cause of death, Dr. Robert W. Nevin, the Medical Examiner for the Dukes County district that includes Edgartown, scheduled an autopsy for today in Falmouth, Mass., at the family's behest.

Dr. Giamatti, who resolved months of controversy last week by banning Pete Rose from baseball amid allegations that he bet on games, left New York City yesterday morning to spend the weekend with his wife, Toni, and other members of his family at their retreat on the island off Cape Cod. His home was in Hamden, Conn.

The announcement of his death prompted expressions of sorrow around the nation from leading figures in education, government and sports, as well as from colleagues and former associates and students, many of whom recalled his zest for life and an astonishing range of learning.

President Bush, a Yale alumnus, said Dr. Giamatti in a brief tenure as baseball commissioner ''made a real contribution to the game, standing for the highest possible ethical standards.'' Waves in Baseball

''Bart Giamatti's premature and shocking death is a tragedy for his family, for Yale and for the nation,'' Benno C. Schmidt Jr., who succeeded Dr. Giamatti as president of Yale, said last night. ''He gave of himself magnificently as teacher, scholar and leader.''

Though well known in scholarly circles for decades, Dr. Giamatti's name did not become a household word until 1986, when he entered major league baseball and began making waves by tinkering with the strike zone and revamping the balk rule.

Sports fans may remember him best for taking on Pete Rose, the pugnacious Cincinnati Reds manager and former playing star whose fall from grace was said to have been precipitated by a secret passion for gambling, including bets on his own team. Mr. Rose consented to the ban in a written agreement but has continued to deny that he bet on games.

Mr. Rose may apply for reinstatement after a year and it will be up to Dr. Giamatti's successor to rule on it. Baseball's executive council, composed of the two league presidents and eight club owners, will administer baseball until a new commissioner is chosen.

Mr. Rose's lawyer, Robert A. Pitcairn, said his client was ''deeply saddened'' by the news of Mr. Giamatti's death. ''In spite of their dispute, Pete had great personal respect for the Commissioner,'' Mr. Pitcairn said in a statement in Cincinnati. ''He extends his deepest sympathy to Commissioner Giamatti's family.'' Scholarly Depths And Campus Pinnacles

But while an avalanche of publicity marked Dr. Giamatti's tenure in baseball, it was his academic tenure in New Haven that may best be remembered.

In a career that plumbed the depths of scholarship and rose to pinnacles of American university life, Dr. Giamatti became one of the nation's leading scholars in English and comparative literature; wrote books on Dante, Spenser and other subjects, and, from 1978 to 1986, served as the 19th president of Yale, his alma mater.

The youngest person in 200 years to hold the coveted presidency of Yale, Dr. Giamatti, who was only 40 when he succeeded Kingman Brewster, was anything but the stuffy blue blood that was the stereotype of Ivy League academic leadership.

''A human being as president of a university - my God, what will that be like?'' quipped a colleague when Dr. Giamatti was selected by Yale's 16-member governing corporation. In 11 years as a professor at Yale, he had captured the respect of administrators, the faculty and students, not only for his academic excellence but also for his style.

Known to friends as Bart, Dr. Giamatti was a man of wit and passion, who in his public and private utterances seemed to exude what made Yale distinctive: a striving for excellence, a devotion to undergraduate education, a love of the humanities and the virtues of learning for its own sake.

His handsome, urbane predecessor had presided over a period of turbulence and expansion with a patrician manner and a relish for political and social issues, and some alumni said he went too far in trying to understand and make compromises with students angered over the Vietnam War and racial injustice.

Dr. Giamatti, a short, stocky man with a beard gone white, was first and foremost a teacher. He resolved to put the university back on a solid footing, academically and financially. He inherited an institution that, by all accounts, was in a malaise. Among the problems were budget deficits, a foundering capital-fund drive, crumbling buildings, an uncertain academic mission and a disgruntled body of alumni.

His first task, upon taking office in July 1978, was to get control over the finances of a school that had run up more than $10 million in cumulative deficits in the previous three years. After two years, Yale reached what he called ''financial equilibrium.'' Focusing on new income, he soon turned the corner.

During Dr. Giamatti's tenure, the size of the endowment rose to $1.35 billion from $544 million and alumni contributions rose to $26.4 million from $9.6 million. When he left office in 1986, it was generally agreed that Yale's finances were in order, the physical plant was well on its way to being restored and, despite inconsistent performances by the football team, Yale's 100,000 alumni were once again enthusiastic. Ivy Vision And Turn to Diamond

The changes he wrought were big and small. He downgraded water polo from a sport to a club activity, phased out 40 senior faculty positions and adopted a new billing system under which students paid tuition earlier.

Dr. Giamatti's most trying moments involved labor relations, most conspicuously a 10-week strike in the fall of 1984 by Local 35, the newly formed union of clerical and technical workers. The walkout produced antagonism between striking and nonstriking employees that persisted long after the settlement.

In his academic policies, Dr. Giamatti pushed undergraduate writing, providing funds for a new set of intensive composition courses that eventually reached nearly all of the students, and he successfully urged the faculty to reinstate a foreign language requirement.

He also encouraged more interdisciplinary majors and programs of study, reinforcing his goal of a leaner, more versatile institution. ''My goal has been to encourage jointness, to push people to think of affiliations rather than to operate as solo entrepreneurs,'' he said.

While his focus at Yale was on university life, he occasionally spoke out on issues of national concern. In a 1981 address to incoming freshmen, he called the Moral Majority and similar conservative groups ''peddlers of coercion'' engaged in ''a radical assault'' on civil rights and pluralism in America.

Later, he criticized the Reagan Administration for what he called defense spending at the expense of education, and for the ''chaos'' of its education policies.

Dr. Giamatti, a heavy smoker who once said cigarettes were his ''primary vice,'' announced late in 1985 that he would give up Yale's presidency the following year. He had been unable to write the major work on the Renaissance that he had long planned, he said. But within a year, he had put down new roots in a world of baseball that he had long romanticized.

The rules of the game, collective bargaining and a variety of other controversies marked his new life in the game. But the most widely publicized events of his tenures as president of the National League and as the commissioner of baseball had to do with Pete Rose.

In 1987, he suspended the Cincinnati manager for 30 days for twice shoving an umpire during an argument on the field. And nine days ago, on Aug. 24, he announced the agreement with Mr. Rose that banned him from baseball for life. Life of Joy In Academic World

Angelo Bartlett Giamatti was born in Boston on April 4, 1938, and reared in classic Ivy League fashion. His parents were Valentine Giamatti, a Yale graduate and professor emeritus of Italian at Mount Holyoke College, and Mary Claybaugh Walton, a graduate of Smith College.

He grew up in South Hadley, Mass., near Springfield. He was never an athlete, but he developed a lifelong interest in sports, especially baseball, and served as manager of the South Hadley high school baseball team. Encouraged by his parents, he also developed an aptitude for languages and literature. The dinner-table conversations often centered on such topics as Dante's ''Divine Comedy.''

While his father was on sabbatical in Italy, the boy attended the International School of Rome. He subsequently graduted from Phillips Academy in Andover, Mass., and then entered Yale, where he majored in English. He earned a bachelor's degree, magna cum laude, in 1960, and went on to Yale's graduate school as a Woodrow Wilson Fellow, obtaining a doctorate in comparative literature in 1964.

That fall, he began his teaching career as an instructor in Italian and comparative literature at Princeton University. In 1965, he was promoted to assistant professor at Princeton and the following year, after teaching a summer course at New York University, he joined the Yale faculty as an assistant professor of English.

He advanced rapidly through the academic ranks at Yale, becoming associate professor of English in 1968, an associate professor of English and comparative literature in 1970 and a full professor in 1971.

Outside classrooms, he served as master of Ezra Stiles College, one of Yale's 12 undergraduate residential colleges, from 1970 to 1972, and as director of the university's visiting faculty program from 1974 to 1976. In 1977, he assumed the newly founded John Hay Whitney Professorship of English and Comparative Literature.

''He's the best English teacher I've ever had,'' one student said in a typical comment. ''He makes you excited about the subject, even if you're not.''

During those years, he became known in academic circles as both a teacher and scholar. His first book, ''The Early Paradise and the Renaissance Epoch,'' was published in 1966, and was followed by ''Play of Double Senses: Spenser's Faerie Queene.''

His latest book, which appeared last year, was ''A Free and Ordered Space: The real World of the University.'' Its wide-ranging ideas examined the nature and purpose of a university, the goals and principles of education and the relationships between the university, the government, private industry and other institutions.

His writings over the years displayed an astonishing range, and included articles for Harper's magazine on Muhammad Ali and other pieces on Tom Seaver, the former Mets pitcher. His lifelong love of baseball (and of the Boston Red Sox) he once summarized in response to a question about his aspirations.

''All I ever wanted to be,'' he said, ''was president of the American League.'' Sports Business Looking for a Leader

He did not exactly get his wish. Instead, after stepping down from the presidency of Yale, he was elected president of the National League in June of 1986 and assumed that post in December of that year. After two and a half years at that helm, he became the commissioner of baseball last April 1, succeeding Peter Ueberroth.

The club owners, who selected him, knew that baseball faced important new ethical questions and that the commissioner would have to be a person of unassailable integrity.

''We knew when we brought him in that we needed someone to handle the sort of problems that for some reason we hadn't had before,'' said Avron Fogelman, a co-owner of the Kansas City Royals. ''We wanted the kind of man who could be totally objective. Whatever decision he makes on this, nobody wins except the integrity of the game.''

A month after he became commissioner, Dr. Giamatti said in an interview: ''I'm the world's expert on sterotypes held by academics about athletes and held by athletes about academics. To me, both of them are caricatures. This is not the first time in my life where you know going into a job that you're going to hear in stereo what was wrong with what you did.''

Survivors include his wife, the former Toni Smith, whom he met at Yale and married in 1960; three children, Marcus Bartlett, Paul Edward Valentine and Elena Walton Giamatti, and a brother, Dino, of Scarboro, Me.

A. Bartlett Giamatti, Renaissance scholar, former president of Yale University and commissioner of baseball, left his mark on the game he loved. He was 51 years old. (DOT/Brad Trent, 1988) (pg. 1); A. Bartlett Giamatti in 1986 at the head of his last commencement procession as president of Yale University (Associated Press); Mr. Giamatti in a production of the ''Nutcracker'' at the American Shakespeare Theater in 1977, shortly after he was chosen to head Yale (The New York Times) (pg. 6)