The head of state passed on Friday evening, Reuters reported, citing state media.
Muscat subsequently declared a three-day period of national mourning.
Recently, reports had suggested that the sultanate was preparing for an elaborate succession process as Qaboos’ health was not taking any turn for the better.
Qaboos had traveled to Belgium last December to seek medical treatment.
He absented himself from an annual Persian Gulf Cooperation Council summit in Saudi Arabia in the same month, giving rise to speculations about his failing constitution.
Sultan Qaboos rose to power in the former British protectorate in a 1970 coup that was assisted by London.
He had no children and had not publicly appointed a successor.
The sultanate’s court now has to pick a replacement in three days. The sultan has already named his desired successor but has kept his identity secret to avoid political fallout.
Should the court fail to name the next sultan, the authorities will have to open secret letters, in which Qaboos has penned the name of the heir to the throne.
The task of opening the envelopes has to be performed jointly by the nation’s defense council, the head of the Supreme Court, and the heads of the two chambers of the consultative council.
Oman brokered conflict resolution negotiations among many regional states under Qaboos.
The sultanate, however, notoriously hosted Israel’s Prime Minister Benjamin Netanyahu in 2018 amid vocal cautions against attempts at bringing the regional Arab states and the regime close.
Oman swears in successor to Sultan Qaboos
This photo shows Oman's Minister of Heritage and Culture Haitham bin Tariq al-Said with a picture of Sultan Qaboos bin Said al-Said in his background.
Oman's Minister of Heritage and Culture Haitham bin Tariq al-Said has been sworn in as new ruler of the Arab country after the death of his cousin Sultan Qaboos bin Said al-Said.
"Haitham bin Tariq was sworn in as the new sultan of the country... after a meeting of the family which decided to appoint the one who was chosen by the sultan," the Omani government said in a tweet on Saturday.
Meanwhile, state television broadcast images of funeral procession in the capital Muscat amid tight security.
The casket was carried into Sultan Qaboos Grand Mosque, where people joined prayers inside.
In his first address to the nation, Haitham vowed to uphold the Persian Gulf country's foreign policy, which he said was built on "peaceful coexistence" and friendly ties with all nations.
"We will continue to follow in the same course the late Sultan adopted … embracing foreign policies based on peaceful coexistence among peoples and countries without any interference in the domestic affairs of other states," he said in a televised speech after the funeral service.
"We will continue as always … contributing and calling for peaceful and amicable solutions for all disputes," he added.
The new sultan further promised to follow in the footsteps of his predecessor and "remain guided by his wisdom into the future, and preserve … the achievements he made."
Sultan Qaboos, the Arab world's longest-serving ruler, died Friday aged 79.
“With great sorrow and deep sadness... the royal court mourns His Majesty Sultan Qaboos bin Said, who passed away on Friday,” said a statement from the royal court.
Muscat announced three days of national mourning after the Sultan’s death.
Sultan Qaboos was unmarried, had no children and had not appointed a successor.
The Omani Constitution requires the royal family to choose a new sultan within three days of the throne falling vacant. Should that fail, a letter written by Sultan Qaboos will be opened before military and government officials revealing his choice for a successor.
Sultan Qaboos had been ill for some time and was believed to have been suffering from colon cancer.
Last month, he returned home after undergoing medical checks and treatment in Belgium.
The sultan rose to power in the former British protectorate in a 1970 coup that was assisted by London.