The words "no, it does not matter" form a sort of principle to adopt in one's life. Of course there are things which obviously do matter. It matters a great deal if you do not follow your ideal; if you make a slip of the tongue, or any other shortcoming. But it does not matter if somebody else is not following the same ideal. You cannot alter his plan of life or his opinions, so they cannot matter to you!