Cayman scored her third goal of the season and Howard netted her first before Deyna Castellanos and Laura Coombs levelled things up in Manchester. Janina Leitzig's heroics in the penalty shootout then confirmed the midweek victory for the Foxes.
Willie Kirk made four changes to the side that drew 1-1 with Tottenham Hotspur last weekend, with Lize Kop getting her first start for the Club, while Remy Siemsen made her first appearance of the current campaign; Aimee Palmer and Aileen Whelan also came into the 11.
The first chance of the game came when the visitors won the ball back on the edge of Man City's box, allowing Siemsen a chance at goal with Khiara Keating scrambling back towards her net, but the Australian couldn't make the most of such an early opportunity.
Throughout the first half-an-hour, City remained in the game, continuing to press from the front and cause problems for the home side, but they were a threat themselves on the break, especially with the electric pace of Chloe Kelly, creating a good battle with CJ Bott.
In what was a tightly-fought first half, Leicester took a 2-0 lead into the interval with two goals in four minutes, and you'd have to say it was deserved. Cayman scored her third goal in three games when a clever pass from Missy Goodwin carved the opening for the Belgian to place the ball under Keating.
Just moments later, City had a corner and rising highest to challenge for the cross was Howard, and the effort was too strong for the Citizens' keeper to keep out, ensuring a two-goal advantage when the whistle blew.
In the second half, the hosts responded with a triple change that saw Lauren Hemp and Yui Hasegawa introduced in an attempt to turn things around. The hosts did quickly half the deficit when Castellanos fired an effort into the top corner from outside the box.
Over the next 20 minutes, Leicester looked to steady the ship but it was the home side who found the next goal when they pounced on a defensive error, allowing the chance for Coombs to sweep home an equaliser.
With the scores level at the end of play, the game went to a penalty shootout to determine who would take the extra point. Just before the final whistle, Leitzig replaced Kop, and she certainly looked to be the key during the spot-kicks.
Saving three of Man City's penalties, Leicester won 4-3 in the shootout to claim the win and two points from Joie Stadium, keeping their unbeaten run in Group B alive, with five points from two games. Jutta Rantala, Whelan, Hannah Cain and Howard found the net from 12 yards for the visiting outfit.