The house was apparently built in 1690 but unfortunately there are no surviving records from that period; the County of Barnstable records office burnt down on October 22, 1827. There are two conflicting accounts of the earliest history. According to Sydney A. Clark, the travel writer who grew up in this house, the house was built by a man named Morey:

"Tradition asserts categorically that it was built in 1690 but the builders name is not certainly known, for all local deeds and records were destroyed in a fire which consumed the Barnstable County records some years ago. Apparently a man named Morey secured a land grant of thousands of acres from the British crown (recently donned by William III) and erected this pioneer home to shelter his family. The architect who designed it, very possibly Morey himself, did not set to work to build something in self conscious Colonial style. He merely built in the only way he knew, and the result breathes the spirit of that unhurried age when a house was definitely the product of craftsmen. The machine age, however clever at camouflage, cannot duplicate it." (Excerpt from "As the Pioneers Built" by Sydney A. Clark.)

One record suggests that the house was built by or for a man named Gibbs and another says it was built by John Ellis. They all agree it was the first house built in the Sagamore Beach area. In either case the house was purchased by Paul Crowell at the beginning of the 19th Century (probably 1818). It is thought that the Crowell family may have brought the house with them from Dennis when they moved to what was then Sandwich. Presumably the land was farmed up until the death of Paul Crowell (1778-1866). Paul Crowell was married to Sally Sears (1780-1861) and most likely had all sixteen children in this house (she also lived to see her children's children to the fifth generation). For reasons unknown, the house seems to have been abandoned near the end of the 19th Century, after Paul Crowell's death.

The house seems to have been included in the original purchase of land that established Sagamore Beach as a summer colony for the Christian Endeavor Society in 1904. The house appears to have been neglected until then when it was acquired by Dr Frances E. Clark (see left)  who "built back" the house. It is known that Dr Francis E. Clark moved into the house after living on the Bluff.

The picture at right is the earliest photo of the house, probably from the time it was acquired by Dr Clark. It is from a post-card (undated, author unknown) that reads,

Old Crowell Farm now home of Mother Clark who with Dr Clark founded the Christian Endeavor Society which is World Wide.

Note that in the picture, there are no large trees and the windmill is absent; these were all added by the Clarks.

Dr Francis Clark brought the house back and more, adding a second story to the lean-to (left of main house in post-card) and a replica Cape Cod octagonal windmill as a summer house. The Clark family retained the house until 1970.