I'd guess that the cam lobe is toast too.. that will be a bitch job to get done.
Try running oil with more zddp in it.. that will help prevent metal on metal wear for flat tappet cam followers / lifters
were you able to see the the cam lobe when you took the follower out? it should be visible in there, but I would assume it's damaged. It's harder in general, but if it's had jagged metal grinding over it like that, It very well could be damaged