As per the Legends, In Dwaparayuga, when Pandavas exiled from the Kingdom, they prayed to Lord Krishna for support during their Vanavasam. Krishna gave the Idols of Sita, Rama, Lakshmana and had asked Pandavas to worship these deities. As instructed by the Lord, Pandavas used to worship these idols. Pandavas while moving to a different place, they had asked Vaikhasana Preist Vedagarbha to worship these idols. Vedagarbha and clan used to perform Abhishekam, Archana to these deities regularly. During the times of Buddhism, to protect idols from demolition, the Vaikhasana priests dug the pit and covered with the mud.
In 16th Century AD, King Poosapati Sita Ramachandra of Gajapati dynasty made Kumbilampuram (Present Kumili Village) as the capital and started ruling this region. Villagers used to gather wood from the forest nearby for their livelihood. One day, while villagers are gathering wood, it started raining heavily with thunders and villagers took shelters under trees. One aged dumb lady a devotee of Lord Rama took shelter under a banyan tree. Lord Rama appeared before her and blessed her with the speech by writing Beejaksharam (Alphabets) on her tongue. Lord Rama asked her to recover Dwaparayuga Idols from the nearby pond and disappeared. As instructed by the Lord, Aged lady somehow conveyed the message to the King. King with this staff recovered the idols from the pond. King with priests installed the Idols on Bhishma Ekadasi day and built the temple on Swetachalam hill. As the Idols recovered from the pond, this shrine was called by Ramatheertham.