Because of physics… But there’s a lot of P&S with fast lenses, from your S110 to LX7… Or larger sensor compacts like the RX100, the new G1X MkII (which has one of the more impressive P&S lens in a while, at least on paper), etc. f0.95 on a 1″+ sensor wouldn’t be compact anymore (or have much of a zoom factor).
Advanced P&S still have lenses way more advanced than any phone. It’s the budget/bottom of the barrel P&S that get worse all the time, but then, cameras on non-flagship phones aren’t much to write about either. I do think three introductory price of cameras like the S120, LF1, RX100, etc is too high tho…
Street prices always end up falling by a hundred bucks or two within months, if such advanced compacts were just a little more affordable and didn’t share the limelight with so much crap, the P&S market would probably be doing better (also if they had started introducing connected models much earlier).
Still, pictures from our One X, One, or Nexus 5 still look like smartphone pics more often than not… /shrug

Started out doing photography at the age of 6 using an uncle's old 1940 kodak brownie box camera. At 15 years of age, I decided to buy my very own 1975 Praktica SLR camera. I now shoot with a Nikon D850. I do unpaid TFP and commercial paid work.