BERLIN--The euroskeptic Alternative for Germany party clinched its first seats in a German state parliament Sunday, taking a larger-than-expected 10% of the vote in Saxony's legislature.

German Chancellor Angela Merkel's center-right Christian Democrats took the largest share of the vote with 39.2%, according to initial results, roughly in line with the party's result of 2009. But the party's traditional coalition partner on the right, the pro-business Free Democrats, missed the 5% hurdle needed to gain representation, garnering just 3.8%. With that result, the Free Democrats lost all their seats in Saxony, the last regional government position it held in the country.

While polls had predicted a comfortable victory for Ms. Merkel's conservatives, the insurgent anti-euro party known by its German initials AfD exceeded its expected 7% of the vote to secure 10%.

The results showed the protest party had managed to carry most of the support it had received at this year's European elections over to the regional level, where the protest vote is traditionally less strong.

The AfD's strong showing in the ballot--an expected 14 of 130 seats in the state legislature--raised questions of whether the Christian Democrats would form a coalition with the euroskeptic party in the eastern German state, although the party leadership in Berlin has ruled out such a step.

Incumbent CDU state premier Stanislaw Tillich, who is also the Christian Democrat's lead candidate for Saxony, said his party would pursue coalition talks with the center-left Social Democrats, who won an expected 12.3%, and the Greens, who took 5.8%. Nonetheless, in an interview with broadcaster ZDF he declined to explicitly rule out a coalition with the AfD in Saxony.

Frauke Petry, the leading candidate for the AfD in Saxony, said her party is keen to play a role in the state government, but the Christian Democrats have to decide who to approach. At a national level, the Christian Democrats govern in a coalition with the Social Democrats, or SPD.

"Any coalition would stabilize the AfD; the [Christian Democrat] union can't afford that," said Gero Neugebauer, a political scientist at Berlin's Free University, adding the election in Saxony shows Ms. Merkel's party is struggling to integrate the right-wing of its party.

As well as its anti-euro platform, the AfD campaigned in the state on law-and-order issues, such as securing Germany's border against crime, which is a high priority for the state of four million people which borders both Poland and the Czech Republic.

Peter Tauber, secretary general of the Christian Democrats, called his party's performance "a good signal for the elections in Thuringia and Brandenburg." The two states--which, like Saxony, are in the ex-Communist region of East Germany--will also hold state elections in two weeks.

Write to Harriet Torry at harriet.torry@wsj.com

Subscribe to WSJ: http://online.wsj.com?mod=djnwires