A musty odor is the "off-gassing" of mold. It typically indicates the presence of mold that is growing. Dormant mold (think winter time) will not off gas and the musty odor will not be present.

Does inactive mold need to be removed?

Strictly speaking, it's technically impossible to “remove” all mold. Dormant microscopic mold spores are inherent in nature and pervasive in both outdoor and indoor environments. Unless (or until) they are triggered into active growing mode, dormant spores pose no issues.

How long can mold remain inactive?

Mold starts with spores that can lie dormant indefinitely. Dormant spores stay inactive if they don't sense the right combination of moisture, temperature, air, and food. When something changes, the dormant spores start actively growing mold.