Open matte is a filming technique that involves matting out the top and bottom of the film frame in the movie projector (known as a soft matte) for the widescreen theatrical release and then scanning the film without a matte (at Academy ratio) for a full screen home video release.
In other words, removing the mattes was a cheap way to release the film on VHS later on so there wouldn't be black bars on people's TV sets. But, sometimes when you remove the mattes, you see things you were never supposed to see.
A great example of this is Pee-Wee's Big Adventure. Just watch this scene where Pee-Wee removes the chain from its container.
That hole in the bottom of the container where the chain is fed through was never supposed to be seen. That's open matte for ya.