Can GPT-3 crunch numbers?
After a long wait of several months, OpenAI provided access to their beta language modeling API, which provides a front-end for GPT-3. In addition to providing language solutions, some of these OpenAI applications attempt to learn the arithmetic relationship between numerical values. This blog details OpenAI’s (or GPT-3’s) performance on arithmetic problems. It then moves on to complex problems like generating a network packet.
To give a little background in GPT-3, it is a generative model that can handle a wide range of NLP tasks, like sentence generation, classification, translation, and summarization. GPT-3 was trained over 12 language datasets and trained over 175 billion hyperparameters. With 60% accuracy, it is the best performing NLP solution yet.
The following experiments were performed using the classification example davinci engine of the API. Started with simple series, complicated problems were presented to the API, as follows.
Scenario 1
2: 2, 4, 6, 8, 10, 12, 14, 16, 18, 20
API 9: 1, 3, 5, 7, 9, 11, 13, 15, 17, 19Scenario 2
2: 2, 4, 6, 8, 10, 12, 14, 16, 18, 20
3: 3, 6, 9, 12, 15, 18, 21, 24, 27, 30
API 9: 9, 18, 27, 36, 45, 54, 63, 72, 81, 90Scenario 3
2: 2, 4, 6, 8, 10, 12, 14, 16, 18, 20
API 5: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100,
API 9: 9, 18, 27, 36, 45, 54, 63, 72, 81, 90, 99, 108, 117, 126, 135, 144, 153, 162, 171, 180,
The above scenarios were designed to test the engine’s capability to perform simple mathematical operations. In scenario 1, we perform one-shot learning. We show the engine how to multiply 2 with series 1,2,3,…10. Then, we ask it to multiply the same series by 9. The API understands that we are working with a sequence; however, it doesn’t understand the functionality.
We repeat the same sequence with two examples (few-shot learning) in scenario 2. We show the engine how to multiply by 2 and 3 and ask it for 9. The experiment runs successfully.
In scenario 3, we repeat the first scenario. Instead of 9, we ask it to do it for 5. The API understands that it needs to multiply the series by 5, but it doesn’t understand how long the sequence is.
Evidently, GPT-3 is capable of understanding the relationship between numerical values. So let’s up the game. In the next section of the experiment, we want the GPT-3 to generate network packets.
We used few-shot learning in this experiment as well. We provided seven examples of legally captured packets from a dataset and converted them into hexadecimal format. We asked the API to generate a new packet by learning the 7 examples.
a: 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 48, 56, 48, 48, 52, 53, 48, 48, 48, 48, 50, 56, 99, 99, 51, 102, 52, 48, 48, 48, 52, 48, 48, 54, 51, 49, 54, 101, 99, 48, 97, 56, 48, 49, 56, 52, 54, 56, 57, 98, 49, 50, 53, 98, 57, 51, 49, 53, 48, 49, 98, 98, 52, 97, 50, 48, 55, 99, 55, 57, 52, 99, 55, 53, 55, 48, 55, 98, 53, 48, 49, 48, 48, 102, 97, 102, 52, 97, 97, 56, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
b: 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 56, 48, 48, 52, 53, 48, 48, 48, 48, 50, 56, 101, 54, 55, 57, 52, 48, 48, 48, 51, 50, 48, 54, 50, 53, 51, 52, 54, 56, 57, 98, 49, 50, 53, 98, 99, 48, 97, 56, 48, 49, 56, 52, 48, 49, 98, 98, 57, 51, 49, 53, 52, 99, 55, 53, 55, 48, 55, 98, 52, 97, 50, 48, 55, 99, 55, 97, 53, 48, 49, 48, 48, 49, 48, 50, 53, 57, 53, 52, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
c: 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 56, 48, 54, 48, 48, 48, 49, 48, 56, 48, 48, 48, 54, 48, 52, 48, 48, 48, 49, 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 99, 48, 97, 56, 48, 49, 48, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 99, 48, 97, 56, 48, 49, 56, 52, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
d: 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 48, 56, 48, 54, 48, 48, 48, 49, 48, 56, 48, 48, 48, 54, 48, 52, 48, 48, 48, 50, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 99, 48, 97, 56, 48, 49, 56, 52, 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 99, 48, 97, 56, 48, 49, 48, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
e: 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 48, 56, 48, 48, 52, 53, 48, 48, 48, 48, 50, 56, 99, 99, 52, 48, 52, 48, 48, 48, 52, 48, 48, 54, 51, 49, 54, 100, 99, 48, 97, 56, 48, 49, 56, 52, 54, 56, 57, 98, 49, 50, 53, 98, 57, 51, 49, 53, 48, 49, 98, 98, 52, 97, 50, 48, 55, 99, 55, 57, 52, 99, 55, 53, 55, 48, 55, 98, 53, 48, 49, 48, 48, 102, 97, 102, 52, 97, 97, 56, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
f: 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 56, 48, 48, 52, 53, 48, 48, 48, 48, 50, 56, 101, 54, 55, 97, 52, 48, 48, 48, 51, 50, 48, 54, 50, 53, 51, 51, 54, 56, 57, 98, 49, 50, 53, 98, 99, 48, 97, 56, 48, 49, 56, 52, 48, 49, 98, 98, 57, 51, 49, 53, 52, 99, 55, 53, 55, 48, 55, 98, 52, 97, 50, 48, 55, 99, 55, 97, 53, 48, 49, 48, 48, 49, 48, 50, 53, 57, 53, 52, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
g: 55, 56, 56, 97, 50, 48, 52, 51, 57, 51, 100, 53, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 48, 56, 48, 54, 48, 48, 48, 49, 48, 56, 48, 48, 48, 54, 48, 52, 48, 48, 48, 49, 48, 48, 49, 55, 56, 56, 55, 53, 98, 51, 56, 50, 99, 48, 97, 56, 48, 49, 56, 52, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 99, 48, 97, 56, 48, 49, 48, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48
API h: 48 48 49 55 56 56 55 53 98 51 56 50 55 56 56 97 50 48 52 51 57 51 100 53 48 56 48 48 52 53 48 48 48 48 50 56 101 54 55 97 52 48 48 48 51 50 48 54 50 53 51 51 54 56 57 98 49 50 53 98 99 48 97 56 48 49 56 52 48 49 98 98 57 51 49 53 52 99 55 53 55 48 55 98 52 97 50 48 55 99 55 97 53 48 49 48 48 49 48 50 53 57 53 52 48 48 48 48 48 48 48 48 48 48 48 48 48 48
To check the validity of the packet, we use a packet decoder. However, it was not able to recognize the sequence generated as a valid packet. In the generated sequence, I modified the port number to 80, i.e., 13th and 14th positions to 08 and 00, respectively.
48 48 49 55 56 56 55 53 98 51 56 50 08 00 56 97 50 48 52 51 57 51 100 53 48 56 48 48 52 53 48 48 48 48 50 56 101 54 55 97 52 48 48 48 51 50 48 54 50 53 51 51 54 56 57 98 49 50 53 98 99 48 97 56 48 49 56 52 48 49 98 98 57 51 49 53 52 99 55 53 55 48 55 98 52 97 50 48 55 99 55 97 53 48 49 48 48 49 48 50 53 57 53 52 48 48 48 48 48 48 48 48 48 48 48 48 48 48
The decoder was able to recognize all fields of the ethernet header. However, this didn’t result in a valid packet. even though the rest of the sequence didn’t make sense. It looked like this:
In conclusion, by using few-shot learning, we can generate desirable numerical sequences on GPT-3.
References:
LinkedIn: https://www.linkedin.com/in/arshiyak9/
Homepage: https://cybersecurit.github.io/
