It means bending your character to unnatural extremes to maximize the efficiency of your character's strengths: squeezing out every stat-point from every weakness to re-allocate it to accentuate character strengths, etc.
For example, in Dungeons and Dragons, there are six stats -- three "physical" (Strength, Dexterity, Agility), and three "intellectual/social" (Intelligence, Wisdom, Charisma). Normally, a well-rounded character would have fairly moderate stats across the board, but reasonable deficiencies in some areas and strengths in others.
But a min/max player will meta-game the system, and create a character that is abhorrent for roleplay purposes, but extremely capable for statistical/combat purposes. If they have a fighter-type character, they might make intelligence 4 or 6, just to pump up strength to 18 or 20, etc