Annotated Bibliography

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

The paper discussing the transformer architecture. The principle focus of the epistemic evaluation.

Blagec, K., Dorffner, G., Moradi, M., Ott, S., & Samwald, M. (2022). A global analysis of metrics used for measuring performance in natural language processing. arXiv preprint arXiv:2204.11574.

A paper describing the importance of translation tasks in evaluation of NLP models.

Hu, K. (2023). ChatGPT sets record for fastest-growing user base - analyst note. Reuters Technology.

Article providing statistic for the ChatGPT rate of user growth.

Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2023). A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.

Paper showing how multimodal large models exhibit emergent properties in modalities outside of where they were originated.

Caelen, O. (2023). Unleashing the Power of GPT-3: Fine-Tuning for Superhero Descriptions. Towards Data Science.

An article describing the process of fine-tuning.

Goldman, A. I. (1986). Epistemology and cognition. harvard university Press.

The set epistemic values used for the overall review.

OpenAI (2023). GPT-4 Technical report. arXiv preprint arXiv:2303.08774.

Technical report published by OpenAI on GPT-4 providing some (but nowhere near enough) information on the model.

Glorioso, C. (2023). Fake News? ChatGPT Has a Knack for Making Up Phony Anonymous Sources. NBC New York I-Team.

Article describing an instance of ChatGPT hallucinating a quote.

Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017, July). Convolutional sequence to sequence learning. In International conference on machine learning (pp. 1243-1252). PMLR.

Paper describing a NLP model which was the state-of-the-art before the transformer. Notably gives training times.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Paper describing BERT, a pre-trained (and tested with fine-tuning) transformer. Provides metrics and training times.

Krotov, V., & Silva, L. (2018). Legality and ethics of web scraping.

Paper discussing the legality and ethics of webscraping. Mainly to show that there are potential risks associated with it.

Knight, W. (2023). OpenAI’s CEO Says the Age of Giant AI Models Is Already Over. Wired.

Article which quotes Sam Altman about the cost of training GPT-4.

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.

Paper describing the LSTM architecture. One of the leading NLP architectures before the advent of the transformer.

Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2), 157-166.

Paper discussing the vanishing gradient problem. Which the solving (one way) of created the LSTM and GRU and therefore indirectly the transformer.

Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.

Paper describing backpropagation the update rule used for non feed-forward neural networks like RNNs (LSTMs/GRUs).

Zinkula, J. (2023). ChatGPT is already stealing work from freelancers. Business Insider.

Article describing how freelances on Fiverrr and similar sites are experiencing a work-drought after ChatGPT came out.

Coyle, J., & The Associated Press (2023). ChatGPT is the ‘terrifying’ subtext of the writers’ strike that is reshaping Hollywood. Fortune.

Article describing how ChatGPT was a key issue related to the WGA strike.

Abramson, A. (2023). How to use ChatGPT as a learning tool. American Psychological Association, 54(4).

Article describing some benefits to ChatGPT.

The Associated Press (2023). OpenAI’s Unusual Nonprofit Structure Led to Dramatic Ouster of Sought-After CEO. U.S. News.

Article the restructuring of OpenAI from not-for-profit to for-profit and the consequences of that.