I don't think Linseed Oil is like Tru Oil, Gun Oil, Danish Oil or Teak Oil.
All of those are essentially thin varnishes. Apart from that, I can't see any reason why you can't use Linseed Oil. I'd do a test in the control cavity to see if the grain filler lifts after oiling to be certain. Similarly with top coats, do a test.
PS: Linseed Oil is usually used to preserve wood from within and you'd want it to penetrate the pores. Grain filler will negate that. What was your reason for choosing Linseed Oil? Is it handy or something?