Give Poncirus trifoliata a try as an ornamental. I think it is pretty, and would give the same look. Anything else, even a kumquat, will take major protection.
Years ago I tried protecting a Meyer's lemon. Couldn't do it. 😳

Along with the straight Poncirus (Citrus) trifoliata species, there are a couple other species and hybrids which are much hardier to cold than you'd expect. The question of edibility though is another matter, and is more variable.

There's Citrus ichangensis, which is a base for a lot of hybrids, and is hardy to close to - 18 C (0 F). There's the Yuzu, Citrus junos, which is hardy to around -15 C (5 F), and the juice and rind are used in various products. Probably the hardiest edible citrus is the Thomasville Citrangequat (Poncirus x orange x kumquat), which is hardy to - 15 C (5 F). Some other Poncirus crosses hardy to around - 18 C (0 F) include the various citranges (Rusk, Morton, Troyer, etc), as well as the Citrumelo (Dunstan, etc).

The most commonly available hardy citrus would likely be the Changsha mandarin, Citrus reticulata "Changsha", which is hardy to around - 12 C (10 F). You are going to have a much easier time tracking down hardy citrus plants in the US than growers in Canada.

I have a Poncirus trifoliata which is fully hardy here in Massachusetts zone 6b. The fruit is not edible but looks nice.

I also have been messing with a Citrumelo. It did fine last winter when we hit -2.9F with no supplemental heat since our power was out that night due to a snow storm. Here's a thread on it......it's 7 feet tall now.

http://palmsnorth.com/forum/viewtopic.php?t=4264&postdays=0&postorder=asc&start=0