Combine the egg yolks with the sugar in the top of a double boiler. Whisk together and place the pan over medium heat.
Add the orange zest and the orange juice. Continue cooking until the mixture has thickened slightly, stirring frequently for about 20 minutes. Set aside to cool completely.
When the orange custard has cooled completely, stir in 1 teaspoon of the vanilla, the lemon zest, and lemon juice.
Place the butter in a mixing bowl and beat on medium speed until light and fluffy, about 2 to 3 minutes.
Pour in the cooled citrus custard, blending into the butter on medium speed. Scrape down the sides and the bottom to make certain the butter is completely blended into the custard.
In a separate bowl beat the egg whites until stiff peaks form, about 2 minutes.
Fold the egg whites into the custard.
Line a trifle dish or bowl with a flat bottom with a layer of ladyfingers. (I use an 8” trifle dish.) Cover with a portion of the citrus custard. Repeat the layers ending with the custard.
Cover and refrigerate overnight.
Before serving, whip the cream along on medium speed until it starts to thicken slightly, about 1 to 2 minutes. Add the remaining ½ teaspoon vanilla, powdered sugar, and the orange oil or extract. Continue beating until stiff peaks form.
Spread over the top of the dessert, garnish with a few thin slices of orange.