Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.
Isn't that a bit of a holy grail though? If your software can fact check the output of LLMs and prevent hallucinations then why not use that as the AI to get the answers in the first place?
Because you - hopefully - have a check against something that is on average of higher quality than the combined input of an LLM.
I'm not sure if this can work or not but it would be nice to see a trial, you could probably do this by hand if you wanted to by breaking up the answer from an LLM into factoids and then to check each of those individually, and to assign a score to them based on the amount of supporting evidence for the factoid. I'd love that as a plug-in to a browser too.
https://arxiv.org/pdf/2308.04430.pdf is interesting from that point of view. They've tackled it from the perspective of avoiding copyright content in training, but including it in inference but I think it ought to mean less hallucination because they also (claim to) solve the attribution problem.
My hypothesis is that including information in the LLM’s prompt to support its answer changes the task roughly from text generation, very hallucination prone, to text summarization, or reformulation with some reasoning, and this is less likely to hallucinate.
That was my personal experience in general with ChatGPT as well as LLaMa1/2.
