This article in VentureBeat identifies a range of opportunities and challenges associated with serving the Natural Language Processing market, which is expected to triple in size by 2025. Data models can output biases that were built into the training data or those which might repeat obscenities when interacting with users. It also identifies the large costs associated with implementing these solutions, especially if operating close to real time. We all reap the benefits of these novel voice-based solutions, but as with internet search engines, the costs are invisible and so there is little awareness of consequences:
“Large language models capable of writing poems, summaries, and computer code are driving the demand for “natural language processing (NLP) as a service.” As these models become more capable — and accessible, relatively speaking — appetite in the enterprise for them is growing. According to a 2021 survey from John Snow Labs and Gradient Flow, 60% of tech leaders indicated that their NLP budgets grew by at least 10% compared to 2020, while a third — 33% — said that their spending climbed by more than 30%.
Take, for example, Megatron 530B, which was jointly created and released by Microsoft and Nvidia. The model was originally trained across 560 Nvidia DGX A100 servers, each hosting 8 Nvidia A100 80GB GPUs. Microsoft and Nvidia say that they observed between 113 and 126 teraflops per second per GPU while training Megatron 530B, which would put the training cost in the millions of dollars. (A teraflop rating measures the performance of hardware, including GPUs.)
Inference — actually running the trained model — is another challenge. Getting inferencing (e.g., sentence autocompletion) time with Megatron 530B down to a half a second requires the equivalent of two $199,000 Nvidia DGX A100 systems. While cloud alternatives might be cheaper, they’re not dramatically so — one estimate pegs the cost of running GPT-3 on a single Amazon Web Services instance at a minimum of $87,000 per year.”
Overview by Tim Sloane, VP, Payments Innovation at Mercator Advisory Group