I love Pet Sounds....it is one of the albums that I own multiple copies of (stereo, mono, OOPs, REs, CD, box sets, outtakes). In general, I love post-surfer BB's (after The Beach Boys Today!) and have a soft spot for the surfer-era music, as my father used to play it when I was a child (although I don't buy it). I also wholeheartedly disagree that it hasn't stood the test of time. The instrumentation doesn't sound dated - usually music sounds dated when it utilizes very era-specific instrument sounds that are no longer in vogue (a lot of synth driven music may fall victim to that - one day heavily auto-tuned music will feel that way, I believe). It is all great playing and orchestration that is time-independent, IMO. As well, the lyrics are so universal in dealing with the anxiety of growing up and feeling out of place that they are not tied to a certain zeitgeist, social/political climate, etc.
However....I HATE the phrase - or ones of similar ilk - "you haven't listened to it" ("really" or otherwise). Or, you just don't understand it. This creates a false dilemma that assumes there is only a binary proposition (i.e. an exclusive either/or). Thus, it would look like this...Either you understand/have REALLY listened to X and love and appreciate it OR you don't understand it and that's why you don't like it. That a person could not possibly both understand what you love, just the same as you understand it, and still not love it themselves. Then again, people do the same with their POS children.
So, in closing, if you do not like Pet Sounds don't feel bad. I mean, you're stupid...but probably too stupid to feel bad about it. As you were.