Would you be able to use the LEDs as light sensors and eliminate the switch component entirely? Maybe a tubular shield over each LED so it doesn't interact with others around it then covering it with your finger could be read as toggling the state?
I'd assume you could detect a difference between a completely/nearly completely covered LED shield and one with a large gap between your hand at the top of the shield.
Would you be able to use the LEDs as light sensors and eliminate the switch component entirely? Maybe a tubular shield over each LED so it doesn't interact with others around it then covering it with your finger could be read as toggling the state?