Is your off-duty firearm of equal caliber to your duty weapon? I know that little Derringer is easy to conceal, but what kind of stopping power does it have? How many rounds does it hold? Do you carry extra magazines while off duty? I would bet that most officers don't.
Third party protection issues
When my dad came outside, I was momentarily distracted and vulnerable to attack. This issue would have been compounded if my dad had argued with me or if I had been responsible for protecting a child or children.
Make sure you have a plan for the unexpected. Your significant other needs to know there may come a time when you are required to take immediate police action. You can develop a code phrase with your significant other and children that will alert them that you're switching gears from citizen to police officer. Part of your plan should include having them leave the area as quickly and safely as possible and calling 911.
If it comes down to going after the bad guy or keeping my family safe, the bad guy is going to get away. Remember, you're primary function during an off duty incident is one of VIP protection for your own family!
The adage of "Walk softly and carry a big stick" seems particularly relevant to the topic of off-duty survival. Make sure you're armed whenever possible, because chances are danger will find you when you least expect it. Don't take action unless it's a matter of life and death and even then, do not do so at the cost of jeopardizing the lives of your loved ones. Practice drawing from your off-duty holster and for God's sake, make sure you carry your badge with your gun! Don't take unnecessary risks. Be a good witness!