Simply put, dexterity and agility are lowered if the equipment you're wearing is not flexible enough for your current score. The penalty varies by material, equipment type, equipment slot and your current stats. Troll hide for instance is very stiff, which limits your movement especially when used for things like cloaks. The two other ways the gray number appears is either by using equipment that boosts your stats, or by sustaining debilitating injuries (greyed out) to a limb or your head.