The Christian Social Union took 37.2 percent of the vote, down from 47.7 percent five years ago. It was the party's worst performance since 1950 in a state vote in Bavaria, which it has traditionally dominated.
Constant squabbling in Merkel's national government and a power struggle at home have weighed on the CSU. It is traditionally a touch more right-wing than the chancellor's party and has taken a hard-line on migration, clashing with Merkel on the issue.
There were gains for parties to its left and right. The Greens won 17.5 percent to secure second place, double their support in 2013. The far-right Alternative for Germany, or AfD, entered the state legislature with 10.2 percent of the vote.
Meanwhile, the center-left Social Democrats, Merkel's other national coalition partner in Berlin, finished in fifth place with a disastrous 9.7 percent, less than half what they received in 2013 and their worst in the state since World War II.
The CSU has governed Bavaria, the prosperous southeastern state that is home to some 13 million of Germany's 82 million people, for more than six decades. Needing coalition partners to govern is itself a major setback for the party, which exists only in Bavaria and held an absolute majority in the state parliament for all but five of the past 56 years.
"Of course this isn't an easy day for the CSU," the state's governor, Markus Soeder, told supporters in Munich, adding that the party accepted the "painful" result "with humility." Pointing to goings-on in Berlin, Soeder said, "It's not so easy to uncouple yourself from the national trend completely."
Still, he stressed that the CSU emerged as the state's strongest party with a mandate to form the next Bavarian government. He said his preference was for a center-right coalition. That would see the CSU partner with the Free Voters, a local conservative rival that made modest gains to win 11.6 percent.
The Greens, traditionally bitter opponents of the CSU with a more liberal approach to migration and an emphasis on environmental issues, are another possible partner. A pro-business party, the Free Democrats, scraped into the state legislature with 5.1 percent support but won't be needed to form a coalition.
The CSU has long leveraged its strength at the state level to punch above its weight in national politics. In Berlin, the party is one of three in Merkel's federal coalition government along with its conservative sister, Merkel's Christian Democratic Union, and the Social Democrats.
That government has been notable largely for internal squabbling since it took office in March. The CSU leader, Interior Minister Horst Seehofer, has often played a starring role. Back in Bavaria, a long-running CSU power struggle saw the 69-year-old Seehofer give up his job as state governor earlier this year to Soeder, a younger and sometimes bitter rival.
Seehofer has sparred with Merkel about migration on and off since 2015, when he assailed her decision to leave Germany's borders open as refugees and others crossed the Balkans. They argued in June over whether to turn back small numbers of asylum-seekers at the German-Austrian border, briefly threatening to bring down the national government.
The interior minister also featured prominently in a coalition crisis last month over Germany's domestic intelligence chief, who was accused of playing down recent far-right violence against migrants.
Seehofer has faced widespread speculation lately that a poor Bavarian result would cost him his job. He told ZDF television his party's election performance had causes in both Berlin and Munich. "Of course, I as party leader bear a share of responsibility for this result," Seehofer said, adding that he was prepared to discuss consequences for Sunday's outcome, but not immediately.
It remains to be seen whether and how the Bavarian result will affect the national government's stability or Merkel's long-term future. Any aftershocks may be delayed because another state election is coming Oct. 28 in neighboring Hesse, where conservative Volker Bouffier is defending the 19-year hold of Merkel's CDU on the governor's office. Bouffier has criticized the CSU for diminishing people's trust in Germany's conservatives.
The CDU's general secretary, Annegret Kramp-Karrenbauer, said the party must show discipline and focus on Hesse. She acknowledged that the national government's woes have been unhelpful. "It is totally undisputed that the way we have treated each other in the coalition, and also the way we argued with each other in the summer, was anything but inspiring for the state election in Bavaria," she said.