A scientific evaluation of the use of limited versions of AI tools as support in identifying and defining simple non-English lithological terms

Authors

  • Urszula Stępień Polish Geological Institute-National Research Institute
  • Aleksandra Fronczak University of Warsaw, Faculty of Geology
  • Wiktor Witkowski University of Warsaw, Faculty of Geology
  • Daniel Zaszewski University of Warsaw, Faculty of Geology https://orcid.org/0000-0003-0830-8547

DOI:

https://doi.org/10.7306/gq.1830

Keywords:

large language models, lithology, geological terminology, chatbots

Abstract

This study was prompted by the need to examine how well AI tools and large language models (LLMs) handle geological issues, particularly lithological issues, in languages other than English. The study aimed to evaluate the quality of responses in Polish generated by free versions of AI tools accessible to non-geologists with limited technological expertise. The survey, which was conducted between February and May 2025, involved people with a background in geology and students of geosciences, whose task was to evaluate each of the responses received. The lithology questions were the same for all respondents. The study involved using ChatGPT, Claude, DeepSeek AI, Google Gemini, Microsoft Copilot, Perplexity AI, and Qwen2.5. Respondents were most likely to use ChatGPT, Microsoft Copilot and Perplexity. The assessment covered the factual accuracy of the responses, the reliability of the sources referenced, and the comprehensibility of the responses received. The study revealed that not all AI tools can process the Polish language effectively, and a lack of relevant publications in Polish hinders the improvement of response quality. It was shown that more complete and complex queries that delve deeper into substantive knowledge enable higher quality and more satisfactory results. These results indicate the need to adapt algorithms to regional scientific terminology specifics, which could enhance the quality, reliability and usefulness of the content.

Downloads

Published

2026-01-12

Issue

Vol. 69 No. 4 (2025)

Section

Articles

License

Authors who publish with this journal agree to the following terms:
  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as this can lead to productive exchanges and earlier and more frequent citation of the published work (See The Effect of Open Access).