You're arguing over grammatical semantics, and you're not doing a very good job at doing so. The phase, "a matter of fact," usually refers to an issue that there are generally accepted means of settling, whether or not it has actually been settled.