I'm getting a strange outline around certain small images that I have programmatically focused using jquery. How can I fix this?

I need a border on it because I have a :focus class that defines what color I want the outline to be. However, when I remove the border and add 1px solid #2b5367; to outline, or remove the outline and set the border to 1px solid #2b5367; I get this:

Which is also not what I want. I want the outter border in the first image to appear, but the inner one to not be there.
Here is the fiddle: http://jsfiddle.net/u1enqp1e/