While you can certainly do it, the fact remains that you still won't be wearing a suit. Wearing a navy blazer with navy pants for example won't look nearly as good, or dressy, as wearing that same blazer with some medium-dark gray pants; especially if that blazer has bright colored buttons.