Github Copilot extensive review. A threat to developers?
A month ago I received a notification from github that I can use github copilot for free for one month. After about a year in the waiting list the time has arrived to test it and investigate its possibilities.
I tested it on three cases using javascript, since it’s one of the languages that copilot plays well.
The first case is algorithmic questions from leetcode, one easy and one difficult.
In the second case I wanted to see solutions for more project oriented things in plain javascript(without using any framework).
The third case is again project oriented questions, using a framework this time.The framework that I used was React.
So I will add the generated code by Copilot with explanations/suggestions from my side.
Also, at the end of each case I will evaluate it with fail or pass, depending on how satisfied I was with the solutions.(This step do not take It too seriously)
First case — Algorithmic exercises
Easy: Test whether a number is a palindrome, that is, whether a number remains the same when its digits are reversed.
Github Copilot solution:
As we can see a lot of things are happening in this code.
Firstly, two if’s are being used, so two branchings are created, instead of one. Something that affects the CPU and perform slower.
An additional step is taken to convert the number to a string and store that value in memory and later the same is happening with the size of the string that is also stored in memory.
We can understand from the while loop that the algorithm is O(n) time complexity.
Human solution:
The time complexity to check if a number is a palindrome or not is O(log10(n)).
Because we are dividing the number by 10 in every iteration.
So the time complexity can be said is equal to the number of digits in a number.
Also, two ifs are not used and the unnecessary conversion of the number into a string is not done.
Difficult: Given the head of a linked list, reverse the nodes of the list k at a time, and return the modified list.
Github Copilot solution:
Imagine that you have a linked list with a head with 9 nodes:
var head; // head of list
/* Linked list Node */
class Node {
constructor(val) {
this.data = val;
this.next = null;
}
}function push(new_data) {
/*
1 & 2: Allocate the Node & Put in the data
*/
new_node = new Node(new_data);/* 3. Make next of new Node as head */
new_node.next = head;/* 4. Move the head to point to new Node */
head = new_node;
}
And we push to the list in that order to have the following constructed list:
1->2->3->4->5->6-> 7->8->9->null
push(9);
push(8);
push(7);
push(6);
push(5);
push(4);
push(3);
push(2);
push(1);
In copilot’s solution we see that there are mistakes, but let’s take the logic in order.
Sets the variable next equal to the next node of the current node
next = current.next;
Later it sets current.next to null
current.next = null;
This is wrong, the correct is current.next is equal to prev, so:
current.next = prev
And below if we see the code has many logical errors, which do not make much sense.
It is very important to say that we are talking about the function that reverses the first k nodes and we are not discussing the whole implementation.
Such a thing is out of scope and maybe the solution will be analyzed in the next article.
So here is the correct solution for the reverse function:
Human solution:
So the result of the first case would be:
Second case — Project Oriented Exercises with plain JS
In this case I thought of making four examples that a javascript developer might need in his daily life.
I wanted to see how copilot is behaving in project related things.
So I chose some of the most popular fields that exist at the moment, such as machine learning, game development, data transformation and unit testing.
Let’s take them in order:
Machine Learning Project-Bitcoin Price Prediction
For the bitcoin prediction Copilot generates the above code, that is not fully functional yet because It misses some important parts, that we have to think about.
First of all we should create the csv with the data with the necessary keys and values.
This is easy enough again using copilot:
The second problem is that we don’t know what library it uses to train the model, specifically in the line:
var model = ml.regression.linear(features, labels);
It is correct on the one hand because it shows that linear regression should be used as in this type of problem linear regression analysis is used to predict the value of a variable based on the value of another variable, but we do not know which library should be used.
So we will need to do manual work to either create the function (https://blog.oliverjumpertz.dev/simple-linear-regression-theory-math-and-implementation-in-javascript) or to Integrate the appropriate library like TensorFlow (https://towardsdatascience.com/linear-regression-with-tensorflow-js-e7b39713572).
So in conclusion we would say that copilot gives you a good skeleton and makes a good enough logical design to have a good starting point, but it will definitely take a lot of research from the developer to implement a relatively good solution and take into account the other parameters.
Simple Game Development — Create a pong game
We can see that a better job has been done here in terms of describing and constructing the functions and objects.
Nevertheless, here too some work should be done for the user part and the cases of what happens when he presses arrows.
Example:
if (e.key == ‘Enter’) { gameState = gameState == ‘start’ ? ‘play’ : ‘start’;}
Simple data Manipulation
In this case I told copilot that I want a new json to be created with only the first and last name of each person.
Here we could say that he did a pretty good job.
Simple Unit Test
Here we could say that it did what I asked for and for such a simple function I couldn’t ask for much more.
But I think there could be more variety in terms of tests.
So the result of the second case, taking into account that in the first two examples the code worked mainly as a skeleton, that the third example did what it was supposed to and the fourth solved our problem in a very beginner’s way, is:
Third case — REACT
We will do three experiments at this point and the degree of difficulty will be scalar:
Here as we can see, Copilot did exactly what we told It to do.
That is, to create a functional component and a react class component, as well as later to export one of the two.
Below we will test the creation of a specific component, starting from the simplest case to something more complex and later to evaluate it.
Creating a very simple react component:
Creating a more dynamic react component:
Creating a much more dynamic react component using state:
We noticed that, even if it is necessary in some cases to add some functions to the onChange listeners etc, the result is quite good.
So overall for react we could say that copilot:
Conclusion
Github Copilot in most cases is a very advanced autocomplete.
It can create very advanced skeletons, but you have to be experienced enough to be able to judge the quality of the solutions which gives you.
Thank you for reading!
Cheers!
