By Charles Forelle, Nektaria Stamouli and Alkman Granitsas 

ATHENS--Greek voters were set to hand power to a radical leftist party in national elections on Sunday, a popular rebellion against the bitter economic medicine Greece has swallowed for five years and a rebuke of the fellow European countries that prescribed it.

With nearly all votes counted, opposition party Syriza was on track to win about half the seats in Parliament. In the wee hours of the morning, it clinched a coalition deal with a small right-wing party also opposed to Europe's economic policy to give the two a clear majority.

"Today the Greek people have written history," Syriza's young leader and likely new prime minister, Alexis Tsipras, said in his victory speech late Sunday. "The Greek people have given a clear, indisputable mandate for Greece to leave behind austerity."

A Syriza victory marks an astonishing upset of Europe's political order, which decades ago settled into an orthodox centrism while many in Syriza describe themselves as Marxists. It emboldens the challenges of other radical parties, from the right-wing National Front in France to the newly formed left-wing Podemos party in Spain, and it sets Greece on a collision course with Germany and its other eurozone rescuers.

Within minutes of the close of the polls, Germany's powerful central-bank chief, Jens Weidmann, pushed back.

"It is clear that Greece will remain dependent on support and it's also clear that this aid will be provided only when it is in an aid program," he said in an interview with television broadcaster ARD.

A message on U.K. Prime Minister David Cameron's usual Twitter account, meanwhile, warned that the Greek result will "increase economic uncertainty across Europe."

Mr. Tsipras staked his campaign on resistance to the policies of fiscal austerity--budget cuts and tax increases--demanded by Germany in return for a EUR240 billion ($269 billion) bailout, and many Greek voters embraced him.

"Europe is self-destructing," said Polyxeni Konstantinou, a 56-year-old public-sector worker voting in central Athens. "I voted for Syriza because I hope that it will help change the tragic circumstances that now govern Europe. Will Syriza be able to achieve everything it says? Probably not. But whatever it does achieve, then that will be good for Europe."

It may be less positive for financial markets, which had reckoned on a narrower victory that would have tempered Syriza's ambitions by forcing it to seek partners in a coalition. The euro slid slightly in the very earliest whispers of trading in New Zealand overnight.

"Greece is two years ahead of the curve" when it comes to politics, said Bill Blain, chief strategist at Mint Partners, a brokerage. "Other countries' [radical parties] are all going to have their day fairly soon and it's really about markets trying to anticipate how other countries will follow Greece."

Europe's financial markets are far more resilient today than they were three years ago, when troubles in Greece sent investors fleeing from government bonds of other troubled countries. Nonetheless, the specter of strained conflict in Greece could cause a selloff of riskier assets.

Mr. Tsipras has promised to roll back austerity, first with a spending package aimed at Greece's struggling poor, and to win forgiveness of some of Greece's towering debt.

He will have a mammoth task at home and abroad.

For one thing, Syriza is a broad coalition of the left that includes factions that believe Greece should leave the eurozone. Those factions would pressure Mr. Tsipras if he moves to compromise with Europe.

The pressure to compromise will be intense. Under the bailout program's rigorous schedule, Greece is required to complete a review of its progress with the so-called troika of bailout inspectors by the end of February. Mr. Tsipras has said he doesn't recognize the troika's authority.

If negotiations to revive the bailout falter, Greece would be without an umbrella of protection. That program ensures that Greece's government has access to a stream of cheap financing, and ensures Greek banks have access to cheap funding from the European Central Bank.

The next hurdle will be EUR7 billion in bonds held by the ECB that mature in July and August. Greece doesn't have the cash to repay them, and failure to do so could ultimately lead to Greece's exit from the eurozone.

The country is already deeply strained. In the past three months, Greece's main stock index has shed roughly a fifth of its value, while nervous depositors have withdrawn several billion euros from the country's banking system.

At the same time, the rest of Europe is financially healthier--even as it is mired in economic stagnation. The ECB's newly announced program of government-bond purchases, while primarily designed to boost sagging inflation, will likely "muffle any contagion effects" by damping financial-market volatility, said Lucy O'Carroll, chief economist at Aberdeen Asset Management.

After midnight local time, with more than 80% of the ballots counted, Syriza had 36.3% of the vote, trouncing the incumbent New Democracy party, with just 27.9%. The polls also showed voters backed a handful of smaller parties--including the extreme-right Golden Dawn party and the centrist To Potami party.

But in the early hours of Monday, even before the final votes were counted, Syriza officials said a deal had been struck with the Independent Greeks--a party that shares little common ground with Syriza except for its rejection of the austerity measures. The party, which is poised to win 13 seats in Parliament, would mean the coalition would have 162 or 163 seats--a comfortable governing majority in Greece's 300-seat legislature.

A meeting between Mr. Tsipras and Independent Greeks party leader Panos Kammenos is scheduled to take place Monday morning, Syriza officials said.

Since first seeking a bailout in 2010, Greece has undertaken broad economic overhauls and cutbacks that have helped mend its public finances and nudged the economy back to growth following six years of deep recession. Those cutbacks have come at a cost: Some 25% of Greeks remain jobless, while a quarter of households live close to the poverty line.

Syriza has promised to change all of that, pledging immediate relief to the poor, rolling back unpopular taxes and negotiating a debt write-down with the country's creditors to free up spending on social programs.

Stelios Bouras, Juliet Samuel and Friedrich Geiger contributed to this article.

Write to Charles Forelle at charles.forelle@wsj.com, Nektaria Stamouli at nektaria.stamouli@wsj.com and Alkman Granitsas at alkman.granitsas@wsj.com