# What AI Still Cannot Do - Savonia AMK

> AI has become increasingly popular in many industries. But the real question is: how trustworthy are the answers?

## Overview
AI responses tend to be most reliable for well-established, widely documented knowledge: explaining concepts, standard methods, common code patterns, and summarizing text you provide. They are least reliable for specific facts (precise figures, citations, dates), recent events, and anything that requires judgment on a real-world situation the model cannot verify.

## 1. Hallucinations and Factuality
When a large language model produces confident text that is wrong, this is called hallucination: plausible but nonsensical or unfaithful generated content (Ji et al., 2023). 

*   **Factuality hallucinations:** Output conflicts with real-world facts or generates unverifiable details.
*   **Faithfulness hallucinations:** Output doesn’t match the source material (instructions or earlier statements) (Huang, L. et al., 2025).

**Workarounds:**
*   Attach your own material to the prompt. Paste the source document instead of asking from memory.
*   Only trust AI-supplied references after checking them against the original source.
*   Ask the model to say “I don’t know” when uncertain, and treat a confident tone as no signal of correctness.
*   Verify every number and fact that will carry weight in a decision or publication.

## 2. Fragile Reasoning
AI usually produces output by matching patterns from training data rather than by systematic reasoning. Performance decays with complexity, and models cannot reliably check their own work.

**Workarounds:**
*   Break complex problems into small steps and verify each step yourself.
*   Check AI calculations against hand calculations or known special cases.
*   Use external verification wherever it exists: run the code, test the formula, compare against published results.
*   Do not ask the model to grade its own output. Use a separate review pass by a human or a different model.

## 3. Sensitivity to Input
AI performance is fragile; rewording a question, adding irrelevant details, or changing the position of information can significantly alter the accuracy of the output.

**Workarounds:**
*   Place the most important information at the start or end of long prompts.
*   Test the same question with different phrasings and compare answers.
*   Keep prompts and inputs as short and focused as the task allows.
*   Be extra careful when working in languages other than English.

## 4. Knowledge Gaps
Models have temporal gaps (knowledge frozen at the training cutoff) and density gaps (rarely documented topics are learned poorly).

**Workarounds:**
*   For anything time-sensitive, use a model with web access or supply current sources yourself.
*   Check the publication date of anything the model tells you about standards, laws, or software.
*   For niche domains, treat the model as a starting point, not an authority. Confirm with primary sources.

## 5. Security and Code Vulnerabilities
AI-generated code frequently contains security flaws, and models are susceptible to prompt injection attacks.

**Workarounds:**
*   Review and security-test all AI-generated code before deployment. Treat it like code from an unknown contributor.
*   Run static analysis and dependency checks as a standard step.
*   Never paste secrets, credentials, or confidential data into prompts unless your organization’s policy explicitly allows it.
*   Be cautious when models process untrusted external content.

## 6. Bias and Accountability
Models can amplify social biases and cannot fulfill authorship standards because they cannot assume accountability for their work.

**Workarounds:**
*   Keep a human accountable for every AI-assisted output. The tool assists; the person answers for the result.
*   Disclose AI use where norms or policies require it, especially in academic and professional publishing.
*   Review outputs for bias, particularly in decisions affecting people.
*   Keep conceptual decisions, judgment calls, and interpretation with the human. Use AI for drafting, coding support, and ideation, with verification built into the workflow.

## Further Reading
If you would like to explore the topic in more detail, you can read our publication *AI-Assisted Simulation for Mass Customization: A Conceptual Framework for SMEs*: [https://urn.fi/URN:NBN:fi-fe20260818118151](https://urn.fi/URN:NBN:fi-fe20260818118151)

## References
*   COPE (2023). Authorship and AI tools: COPE position statement. Committee on Publication Ethics.
*   Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1), 64–93.
*   Dziri, N., et al. (2023). Faith and Fate: Limits of Transformers on Compositionality. Advances in Neural Information Processing Systems 36 (NeurIPS 2023).
*   Fritz, M. (2024) Fundamental Risks in the Current Deployment of General-Purpose AI Models: What Have We (Not) Learnt From Cybersecurity? arXiv. doi:10.48550/arXiv.2501.01435.
*   Fu, Y., Liang, P., et al. (2025). Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study. ACM Transactions on Software Engineering and Methodology.
*   Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024). Bias and Fairness in Large Language Models: A Survey. Computational Linguistics, 50(3), 1097–1179.
*   Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. International Conference on Learning Representations (ICLR 2024).
*   Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T. (2025). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 43(2), 1–55.
*   ICMJE (2025). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals: Artificial Intelligence. International Committee of Medical Journal Editors.
*   Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 1–38.
*   Kalai, A., Nachum, O., Vempala, S., & Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664.
*   Kandpal, N., Deng, H., Roberts, A., Wallace, E., & Raffel, C. (2023). Large Language Models Struggle to Learn Long-Tail Knowledge. Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, 15696–15707.
*   Liu, N. F., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
*   Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22, 216–242.
*   Mirzadeh, I., et al. (2024). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229. Published at ICLR 2025.
*   OpenAI (2025). OpenAI o3 and o4-mini System Card. 16 April 2025.
*   Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2023). Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. Communications of the ACM, 68(2).
*   Vectara (2025). Hughes Hallucination Evaluation Model (HHEM) Leaderboard.
*   Veracode (2025). 2025 GenAI Code Security Report.
*   Veracode (2026). Spring 2026 GenAI Code Security Update.
*   Xuan, W., et al. (2025). MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation. arXiv:2503.10497. Published at EMNLP 2025.