Rescue teams that included elite Navy SEAL divers were racing against time to find up to 292 people missing in the freezing waters.
Officials confirmed three people — a woman, a man and a boy — had died as the vessel sank 20 kilometres off the southern island of Byungpoong.
Government officials retracted an earlier statement that 368 people had been rescued. So far, only 164 have been confirmed rescued.
The remaining 292 are unaccounted for, said Lee Gyeong-Og, the vice minister of security and public administration, at a press briefing in Seoul.
Time was running out for those who still remain unaccounted for, as the temperature of the water is 12.6 degrees Centigrade. Humans can survive for about two hours in such temperature.
Dramatic television aerial footage showed terrified passengers wearing life jackets clambering into inflatable boats as water lapped over the rails of the vessel as it sank.
Some could be seen sliding down the steeply inclined side of the ferry and into the water, as rescuers, including the crew of what appeared to be a small fishing boat, struggled to pull them to safety.
Different reports put the total number of people on board between 459 and 477 – over 320 of them high school students bound for holiday island of Jeju, known as “South Korea’s Hawaii” and one of the country’s most popular tourist destinations.
Photos broadcast on television immediately after the disaster showed the ferry tilted over 45 degrees on the port side with helicopters flying overhead, and then fully capsized with only its stern visible.
The ferry, bound for the southern resort island of Jeju, send out a distress signal at 9:00 am (0000 GMT) with passenger testimony suggesting it may have run aground.
“We heard a big thumping sound and the boat stopped,” one passenger told the YTN news channel by telephone.
The 6,825-tonne ferry, which had sailed out of the western port of Incheon on Tuesday evening.
Lee Gyeong-Og said a total of 34 naval, coastguard and civilian vessels were involved in the rescue operation, along with 18 helicopters.
“There is so much mud in the sea water and the visibility is very low,” he added.
The US 7th Fleet said an amphibious assault ship, the USS Bonhomme Richard which was on routine patrol west of the Korean peninsula, was being sent to help.
The cause of the accident was not immediately clear, although rescued passengers reported the ferry coming to a sudden, shuddering halt — indicating it may have run aground.
The weather was described as “fine” with moderate winds and sea swell.
Og said President Park Geun-Hye in a personal message “ordered us to make efforts not to leave a single casualty”.
The ferry manifest included 150 cars.