Yonggungsa Temple is located on flat ground at the foot of Baegunsan Mountain on Yeongjongdo Island. It is said that Great Master Wonhyo founded the temple in the 10th year of King Munmu of Silla (670) and named it Baegunsa Temple, which is also known as Gudamsa Temple.

It is said that in the fifth year of King Cheoljong's reign (1854) of the Joseon Dynasty, Regent Heungseon Daewongun rebuilt and renamed it Yonggungsa Temple and lived there until Gojong became king.

Legend has it that a fisherman named Yoon Gong, who lived in Woljon of Joongsan on Yeongjongdo Island, had a dream, after which he erected a small statue of the Buddha made of jade from the sea with his fishing net and enshrined it in the temple. The statue is now gone, and there is a replica in its place.

Gwaneumjeon hall has a gable roof and single layer eaves, and is where a Gwaneum statue made of jade was once enshrined. However, the statue was stolen during the Japanese colonial era, and there is now an image of the Gwaneum in the main platform painting hung on the wall. In the front of the building are verse couplets carved on four planks, which were put on pillars written by Haegang Kim Gyujin.