Grab a ☕ & get comfy! Nathan Benaich released a whopper 163-page The State of AI 2023. Great review of a crazy year in AI. What resonated with me:

🌟 GPT-4 the uncontested, most generally capable model. Solved tasks GPT-3.5 unable to, like the Uniform Bar Exam (90% score vs 10%). p.12

🌟 Economic stakes high! OpenAI & Google technical reports on latest models don’t disclose information useful for researchers. p.16

🌟 LLaMa-2 70B by Meta competitive with ChatGPT on most tasks. Can use commercially, downloaded 32M times! p.19

🌟 Although context windows (the text input length) of models growing in size, there’s a “Lost in the Middle” performance problem. p.24

🌟 Innovations including FlashAttention & 4-bit quantization tackle challenges like reducing memory footprint & accelerating inference. p.25

🌟 Microsoft researchers find small language models (SLMs) trained with very specialized, curated datasets rival models 50x larger on specific tasks, e.g., phi-1.5. p.26

🌟 Research to embed output from models with digital signatures to support identifying real from fake. p.30

🌟 LLMs are great prompt engineers, out-performing human designed prompts. p.35

🌟 Continuously monitor performance of GPT models, as they are continuously updated, with varying performance. p.36

🌟 Google’s Med_PaLM2 sets new SOTA results for medical-related benchmarks. p.63

🌟 More than 70% of most cited AI papers in last 3 years have authors from US organizations. p.68

🌟 NVIDIA joins $1T market cap club. p.70

🌟 Not just top hyperscalers buying GPUs. Startup infra provider Lambda spending 9-figure $ sums, with over 45,000 GPUs installed. (p.71). Other companies buying vast quantities include Cohere, Inflection, & Imbue. (p.72)

🌟 Compute is the new oil, even in Saudi Arabia! One university buys 3000 cards for LLM research. (p.73)

🌟 Microsoft leads cloud service provider AI spending as % of total capex. p.79

🌟 For mid-level professional writing, workers using ChatGPT took 40% less time, quality 18% better. p.91

🌟 Compared to YouTube, Instagram, & TikTok, GenAI apps like ChatGPT suffer from lower retention rates & daily active users. p.94

🌟 Legal complications on text and image copyright infringements surfacing. p.98

🌟 Governments building GenAI compute capacity, but lagging behind private sector. p.131

🌟 Ukraine a lab for AI-powered warfare. p.133

🌟 Policymakers adopting wait-and-see to potential job losses. Support for Universal Basic Income. p.137

🌟 Open vs Closed source debate continues; open source levels playing field but higher risk of misuse, versus closed source for greater security but less transparency. p.146

🌟 Top predictions include craze on huge models seeing >$1B spent on training a single model, investigation of Microsoft/OpenAI deal on competition grounds, & a large AI company acquiring an AI chip company. p. 157