Why is it that when having an active call and a call on hold, pressing the flash hook sometimes forms a 3-way call but some other times it toggles the two calls?

Whether it is toggling/switching calls or forming a 3-way call when pressing the flash hook of the FXS port analog phone depends on the 2nd call from the BT110’s (the mixing party’s) perspective. If the 2nd call is an inbound call, then, pressing the flash hook of the FXS analog phone will toggle the call, i.e., it switches the on-hold call into the active call, and the active call into the on-hold call.  On the other hand, if the 2nd call is an outbound call, then, pressing the FXS analog phone flash hook forms a 3-way call. 

Note that you can choose among switching the calls, transfer the call, or merging the calls to form a 3-way call if you use the BuddyTalk Tablet Controller or the Mobile App by tapping the Switch icon or the Merge icon.