Parliament voted by 195 votes to 154 to reject the Speaker's plan for a coalition of the center-right Moderates and the small Christian Democrats, with Moderates leader Ulf Kristersson as prime minister.
Those opposing the coalition said it would give influence to the anti-immigrant Sweden Democrats — the third-largest party but considered a pariah by many — because the government would be dependent on that party's support in parliament.
Wednesday's vote was the first of a possible four before Speaker Andreas Norlen must call new elections. It was the first time that a candidate for prime minister has ever been rejected by the 349-seat Riksdagen.
Norlen said he would continue talks with party leaders on Thursday, without giving details. The September election produced a hung parliament with the left-leaning side and the center-right bloc securing about 40 percent of the vote each, leaving neither with a majority and paving the way for months of uncertainty and complex coalition talks.
Both blocs have refused to cooperate with the Sweden Democrats, who made great strides in the election. Annie Loof, the leader of the Center Party that voted against Kristersson, said he had broken up the 14-year-old center-right alliance by suggesting a two-party government, leaving out her party and the Liberals.
Kristersson said that that although the alliance has suffered "its worst crisis," it is not dead. "We have a common platform, but we cannot agree when it comes to taking power." Caretaker Prime Minister Stefan Lofven, who heads Sweden's largest party — the Social Democrats — said he was willing to discuss with Loof should she be tapped to try to form a coalition.
Loof earlier said that she doesn't want to be prime minister herself but is willing to try form a government. Pundits say either Loof or Lofven would be next in line to try to form a government. The emergence of Sweden Democrats — with roots in a neo-Nazi movement — reflects how deeply that Sweden is being transformed by migration.
In 2015, Sweden took in a record 163,000 migrants — the highest per capita of any European country. Since then, the country has sharply restricted immigration, but many Swedes complain that society cannot cope with integrating so many newcomers, many of them Muslims from Africa and the Middle East.
Ahead of Wednesday's vote, Jimmie Akesson, the leader of the Sweden Democrats, said he sees "a new political landscape mushroom in Sweden, exactly like in the rest of the western world."
Jan M. Olsen in Copenhagen, Denmark, contributed to this report.