One temptation for many officers is to scrimp on caliber for the sake of having a small, easy to carry off-duty gun. Please don't. We have a saying: "Friends don't let friends carry mouse guns." While there may be some validity to the idea that any gun is better than no gun, very few people really need to have such a small gun that the only choices for caliber are something south of 9mm. In the case of law enforcement officers, you have responsibilities that you may have to fulfill, and being under-powered will not be a good thing. While some ordinary citizens may be deliberately trying to keep a very low profile when they carry, a police officer usually does not. You may be required to act. Private citizens have no such requirement and can legitimately fade away if a problem arises. I wouldn't want to see an officer trying to stop a crazed gunman, say at a shopping mall (I'm thinking Trolley Square here), with a gun that is best suited to a second back-up role. Stick with the first line calibers.
I addressed the best solution to this earlier. Even if you are using a compact sized pistol, try to have a higher capacity magazine as a reload. After all, if you’re that far into a gunfight, concealability of the gun became irrelevant a long time before. If not, and you have a lower capacity gun then plan on carrying more spare ammo. I carry one spare maximum capacity magazine if I am carrying an auto loader that has at least a 10 round capacity. The spare is the compatible full capacity magazine. If I carry a lower capacity gun, such as a revolver (yes, I do), then I always have at least two reloads - and sometimes more.
There are some excellent small guns available today for concealed carry. They generally suffer from two faults. They do not carry very many rounds, usually 6 or 7, and they tend to be more expensive. You pay for the convenience of small size. That said, they are a good choice if you really need to scale things down. The SIG P239 I mentioned earlier is a great little gun. I own several, but it has a single stack magazine. Likewise some of the small Kahr pistols are excellent. The PM9 is about as compact as a 9mm can usefully be, but again, capacity is the trade-off. Such slim guns can be handy. One problem with the Glocks is that they are all the same basic width within the same caliber. The exception is the Glock 36, which gives you the tried and true .45ACP caliber in a very slim pistol. I still would not call it a tiny gun, but it is slimmer than the other Glocks. If you want all the advantages of a Glock in a slim frame, check it out. Also, if you have not already checked out the compact versions of the Springfield XD and the Smith and Wesson M&P pistols, you should. There are some good things happening at both companies and they are continuing to expand their compact offerings. I shot an XD 9mm sub-compact this past weekend and it was superbly accurate and an absolute pussycat to shoot. I am also looking forward to the Smith & Wesson M&P .45ACP Compact pistol. They had a prototype of it at the SHOT Show, but it was not available for live-fire testing. It looked and felt great, and I am really looking forward to test driving one. In other words, there are a lot of good choices out there.
Finally, how you carry can have a big effect on what gun to choose. Let me say up front, you may have to change your wardrobe to accommodate carrying a decent concealed carry gun. This is usually more of a problem in areas of the country that have a lot of hot weather and minimal clothing is necessary for survival. In my case, here in sunny Florida, I usually wear a "safari" vest in the cooler weather. It easily hides any size carry gun and, besides, I have been spoiled with having all the pockets. It has become my portable office. When it gets hot, I go to shorts and an un-tucked polo shirt. I buy the shirts one size larger than normal, which gives plenty of drape at the waist. I carry a compact gun in an inside the waistband holster. No problem. Sometimes I wear a T-shirt (tucked in) with a light-weight shirt unbuttoned and un-tucked as my concealment garment. Very tropical. If necessary, I hum a few bars of "Margueritaville" and act like a tourist. Where there's a will, there's a way. If you are in a hot and humid area, or you sweat a lot, as I certainly do, consider the durability of your gun's finish. In hot weather, I usually have a Glock in a kydex holster. Leather and/or blue steel are not good choices here.