CHATGPT FOR POINT CLOUD 3D OBJECT PROCESSING

: Large-scale pretrained language models have been a revolution in human-machine communication. Recently, such language models also generate code for required tasks. The objective of this work is to evaluate the functionality of the codes generated by ChatGPT (version 15-Dec-2022) for point cloud processing. The programming language selected for the test was MATLAB due to the extensive use in prototyping and toolboxes for Computer Vision and LiDAR. Using the Question-Answer system, the ChatGPT was asked for codes to calculate surface normals, curvature, eigenvalues, and eigenfeatures, with specific parameters and outputs. The provided codes were compiled and executed. The results show that ChatGPT can generate functional code for very specific and short applications, however, it is not capable of generating large code involving the correct use of loops, indexes, or equations.


INTRODUCTION
Point cloud processing is a crucial technology for 3D object detection and mapping.A point cloud is a set of points in 3D space that represent the surface of an object or scene.These points are usually acquired using 3D sensors such as lidar or stereo cameras, and they provide a detailed representation of the geometry and structure of the object or scene.
Point cloud processing refers to the algorithms and techniques used to analyse, manipulate, and extract information from point clouds.This includes tasks such as segmentation, registration, feature extraction, and classification.These techniques are essential for 3D object detection and mapping, as they allow us to identify and locate objects in a scene, and to create accurate and detailed 3D maps of the environment.
Point cloud processing has many applications in fields such as robotics, autonomous vehicles, augmented and virtual reality, and computer vision.It allows us to accurately perceive and understand the 3D world around us, and to interact with it in a more natural and intuitive way.It also has important implications for fields such as security, surveillance, and environmental monitoring, where the ability to detect and track objects in 3D is crucial.
Overall, point cloud processing is a key technology for 3D object detection and mapping, and it has numerous applications and implications in a variety of fields.
The above introduction was ChatGPT's response to the question Can you write an introduction on the relevance of point cloud processing for 3D object detection and mapping?Although a bit repetitive emphasizing the importance of point clouds, the introduction was automatically structured on a correct definition of point clouds, the main tasks that compose processing (Che et al., 2019;Wang and Kim, 2019;Xia et al., 2020), and their applications (Li et al., 2021;Liu et al., 2019;Xie et al., 2020).

* Corresponding author
ChatGPT is, in its own words, a large artificial neural network that has been trained on a vast amount of text data to generate human-like responses to questions and prompts.Its primary function is to assist with a variety of tasks by providing information and answering questions to the best of its ability (answer to the question In technical words, who are you?).
Unlike other chatbots, ChatGPT is based on the GPT-3 AI language model, developed by OpenAI, and is not only able to provide more human-like responses to a conversation, but can also generate text and code in the desired programming language.
The objective of this work is to evaluate the use of GPT chat to obtain the codes for point cloud processing in MATLAB, such as normal estimation or geometric feature extraction.This way it is evaluated if the proposed codes are functional and to what extent they can be useful to people with different degrees of knowledge in point cloud processing.
The rest of this paper is organized as follows.In Section 2, the state of the art on Artificial Intelligence chats is collected.In Section 3, ChatGPT and the methodology of questions-answers are explained.Section 4 presents, analyses and discuss the results of scripts prosed by the chat.Section 5 concludes the work.

STATE OF THE ART LARGE-SCALE PRETRAINED LANGUAGE MODELS
The understanding and the treatment of the ubiquitous textual data is a major research challenge when the tremendous amount of such data produced by our society has exploded over the recent years.Natural Language Processing (NLP) aims to provide a set of techniques able to explain a wide variety of natural language tasks in the meaning extraction process to be successful.Recurrent Neural Networks (RNN) were massively used to solve NLP problems, especially with the success of Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures in preventing the vanishing gradient issue in training.However, those two models are also not perfect because of the recurrent structure made them hard to parallelize and the treatment of very long clauses is also problematic (Gillioz et al., 2020).
To counter those limiting constraints, the Transformer architecture was introduced with the massively use of unsupervised pre-training principle (Gillioz et al., 2020).The pre-training principle works in two steps: 1) the pre-training phase, which computes a general representation from raw data in an unsupervised fashion; 2) the use of finetuning techniques to adapt to a concrete task.The original Transformer used an encoder-decoder architecture like earlier sequence-to-sequence models with adoption of the self-attention mechanism, differentially weighting the significance of each part of the input data.The recent significant increase in the performance of NLP models is due to the use of word embeddings and the attention mechanism aims to catch the long-term dependencies of sentences.Mathematically, Transformer takes a sequence X=(x1, …, xN) and produce a latent representation Z=(z1, …, zN), then the output sequence YM=(y1, …, yM) is produced one element at a time.This property is called the autoregressive property of the model, where a model takes the previous outputs to produce the next outcome.
After that, various models coupled with the Transformer re using this property to produce accurate Language Model languages like ELMo (by Allen Institute for AI), BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pretrained Transformer), and GPT-2 (by OpenAI).Due to the high performance of BERT and the need of larger training datasets, there are a lot of post-BERT models (like RoBERTa, DistilBERT, AIBERT) aiming to tweaked it to specific needs such as simpler architecture, smaller models, reducing number of layers without a loss of performance.
In the context with GPT, it is suitable to mention that OpenAI was founded as a non-profit research organisation in 2015 with the mission is to ensure that artificial general intelligence (AGI) benefits all of humanity.In 2019, OpenAI had restructured as a profit company and in 2020, despite various own previous concerns about malicious applications, it announced GPT-3, a new language model more than 100 times larger than GPT-2, with 175B parameters and 96 layers trained on a corpus of 499B tokens of web content (Dale, 2021).The model itself was not made available, but the access was provided via an API giving the model's creators more control over its use (e.g., waiting list, high price).The typical use of the API is a prompt with an initial text to get the model going, along with optional parameter setting (Floridi and Chiriatti, 2020).
Several produced outputs are truly breath-taking as candidates for being human-authored text.The technology has also been lauded for its results in a wide range of other areas such as capability in generating poetry, playing chess, doing arithmetic, and writing programming code based on requirements expressed in natural language.The technology made incredible, amazing, and challenging impression.It is also threatening, which can be expressed shortly as "even with 175B parameters and 450 gigabytes of input data, it's not a reliable interpreter of the world" (Gary Marcus and Ernest Davis, MIT Technology Review).After a hype and the conclusion (not reliable narrator), this is not to say that GPT-3 is devoid of practical application; meaning some use cases are appropriate and some are not (Zhang and Li, 2021) and unreliable doesn't mean useless.As the GPT-3 model presents security and uncontrollability problems, including false content and biased information during content generation, its application value is primarily reflected in intelligent auxiliary tasks, and it cannot directly interface with the end-users.

ChatGPT
The GPT-3 model offers many powerful functions that can cope with many practical application scenarios, such as question answering, reading comprehension, summary generation, automatic chat, search matching, code generation, and article generation (Zhang and Li, 2021).Recently, ChatGPT is finetuned from a model in the GPT-3.5 series (Yao Fu et al., 2022), which finished training in early 2022 on an Azure AI supercomputing infrastructure.It is a sibling model to (OpenAI) InstructGPT model, which is trained to follow an instruction in a prompt (Figure 1) and provide a detailed response under the Reinforcement Learning from Human Feedback (RLHF) approach (OpenAI, 2022).
Despite its powerful modelling and description capabilities, there are significant issues and limitations.Such GPT-a-like model does not understand writing well and sometimes generates uncontrollable content.Secondly, training such models requires a large amount of computing power, data, and capital investments.The most important thing is although ChatGPT is trained with human-in-loop, it is just more truthful and less toxic.

Question and Answer process
The way to obtain information from ChatGPT is through a process of queries and responses (Figure 2).The first question is the generation of a code for the execution of the desired operation.Then, the answer is evaluated to see if it matches the desired method.If it is observed that the code has no relation with the desired one, or if no code is generated, the Chat is instructed to try again, either by making variations in the question to focus the result or not.If the code looks like the desired one, it is tested in MATLAB with a point cloud.If the code generates an error, it is indicated to the chat in question mode.When a code that generates the desired output is obtained, modifications in parameters or outputs are applied.
In order to evaluate code generation, some basic point cloud processing methods to extract feature information are selected to be generated automatically by ChatGPT.These features are the basis of point cloud processing for object segmentation and identification.Given a 3D point cloud X = {p1, p2, • • • , pN } ∈ R N×3 , with for each local sample Pi = {pi,j |pi,j ∈ KNN(pi)}, constructed by the k nearest neighbour (kNN) search (Zhou et al., 2021).The covariance matrix of the local neighbourhood can be used to estimate local surface properties (Pauly et al., 2002).The covariance matrix for a point p is given in equation ( 1) where ̅ is the centroid of the neighbourhood.
Consider the eigenvector problem in equation ( 2) and since the covariance matrix C is symmetric and positive semi-definite, all eigenvalues  are real-valued and the eigenvectors v form an orthogonal frame, corresponding to the principal components of the point set (Jolliffe, 2002).The  measure the variation of the pi along the direction of the corresponding eigenvectors.
The curvature describes the surface variation along surface normal in equation ( 3).Alternatively, it can also be calculated as the variance of the normal, as shown in equation ( 4).

RESULTS AND ANALYSIS
The communication was performed on ChatGPT Dec 15 Version.The code generated were tested with the Stanford bunny point cloud (50 thousand points) in ply format in MATLAB 2022b version.

I hope this helps! Let me know if you have any questions or need further assistance.
Based on the authors' prior knowledge of the programming language, this response was perfectly suited to the desired objective.In addition, ChatGPT provided an explanation of how the code works almost line by line, as well as references to the necessary library, an example output of a point and an output image (Figure 3).
In view of the success obtained, the next question was towards the variation of the number of nearest neighbours used for the calculation of normals: in MATLAB, generate a code to calculate the normals of a point cloud based on 25 neighbours.The answer given was a brief explanation of the code (which will not be reproduced below to save space), the Code 1, and an alternative Code 2. This result can be considered a success.ChatGPT made references to the previous answer so as not to repeat the explanation, it added the explanation of the new function, and it generated a code that exported the point cloud on a txt file with a correct separation between columns.Bunny point cloud with estimated normal is visible in Figure 4).

Curvature
For the curvature calculation, the following question was asked: in MATLAB, generate a code to calculate the point curvature of a point cloud based on 25 neighbours.Unfortunately, the chat response (Code 4) referred to adding a curvature output to the calculation of normal, which was not correct, since this function does not support multiple outputs.In the proposed response (Code 6), it can be seen how the indications were followed, however, the result was not as expected.
Code 6: Curvature (Attempt 3) ptCloud = pcread('pointCloud.ply');normals = pcnormals(ptCloud, 25); variance = var(normals); Finally, the authors decided to ask the ChatGPT what information it had on the point cloud curvature: which is the point cloud curvature?The following response was given: In the context of point clouds, curvature refers to the degree of curvature or bend in a surface at a particular point.In other words, it is a measure of how much a surface deviates from being flat at a given point.In a point cloud, curvature can be calculated by analysing the shape and orientation of the points in the cloud and how they vary in relation to one another.There are several different methods for calculating curvature in a point cloud, including principal curvature, mean curvature, and Gaussian curvature.The specific method used may depend on the intended application and the type of data being analysed.
Based on the information provided (which was correct) and as there is no reference to the variance of surface normal to calculate the curvature, it was decided to ask for a code based on one of the methods mentioned: in MATLAB, generate a code to calculate point cloud Gaussian curvature.However, the answer (Code 7) again referred to a function that does not exist in MATLAB and the authors were decided to end this test without obtaining a correct code to calculate the curvature in point clouds.

Eigenvalues
For the eigenvalues calculation, the following question was asked: in MATLAB, generate a code to calculate the eigenvalues of a point cloud.The answer referred to the Statistics and Machine Learning Toolbox with Code 8.The first thing to note was the change of input to a mat file.This was no problem, assuming that the point cloud is stored in a mat file.Next, Principal Component Analysis (PCA) was calculated, and the eigenvalues were indicated as the latent output.The code

Figure 2 .
Figure 2. Workflow of questions (Q.) and answers (A.) to generate functional code.
calculate the normals of a point cloud in MATLAB, you can use the function pcnormals, which is part of the Computer Vision System Toolbox.Here's an example of how you can use this function: Load the point cloud data into MATLAB.For example: ptCloud = pcread('pointCloud.ply');Use the pcnormals function to calculate the normals of the point cloud.You can specify the number of nearest neighbors to use when calculating the normals, as well as the search radius.For example: normals = pcnormals(ptCloud, 10); The output of the pcnormals function is a matrix with the same number of rows as the input point cloud, and three columns for the x, y, and z components of the normal vector at each point.You can access the normals for a particular point by indexing into the matrix.For example: normal = normals(100,:); % get the normal at point 100 If you want to visualize the normals, you can use the pcshow function to display the point cloud, and then use the quiver3 function to plot the normals as arrows.For example:

Figure 3 .Figure 4 .
Figure 3. Image generated from Matlab code provided by ChatGPT of normal estimation and zoom in.Code 1: Normal estimation k neighbours % Load the point cloud pc = pcread('pointCloud.ply');% Extract the xyz coordinates of the point cloud xyz = pc.Location; % Set the number of neighbours to use for normal estimation k = 25; % Calculate the normals of the point cloud normals = pcnormals(pc, k); Code 2: Normal estimation (alternative) % Load the point cloud pc = pcread('pointCloud.ply');% Extract the xyz coordinates of the point cloud xyz = pc.Location; % Set the number of neighbours to use for plane fitting k = 25; % Fit a plane to the points in the neighbourhood of each point [model, inlierIndices, outlierIndices] = pcfitplane(pc, k); % Extract the normal vector of the fitted plane normals = model.Normal; point cloud pc = pcread('pointCloud.ply');% Extract the xyz coordinates of the point cloud xyz = pc.Location; % Set the number of neighbours to use for normal estimation k = 25; % Calculate the normals and curvature of the point cloud [normals, curvature] = pcnormals(pc, k); When it was indicated that 'pcnormals' function did not support two outputs (Error using pcnormals.Too many output arguments), ChatGPT proposed an alternative fitting the point cloud to a geometric model (Code 5): I apologize for the error in the previous answer.It looks like the pcnormals function does not return the curvature of the point cloud when called with two output arguments.Code 5: Curvature (Attempt 2) % Load the point cloud pc = pcread('pointCloud.ply');% Extract the xyz coordinates of the point cloud xyz = pc.Location; % Set the number of neighbors to use for plane fitting k = 25; % Fit a plane to the points in the neighbourhood of each point [model, inlierIndices, outlierIndices] = pcfitplane(pc, k); % Extract the curvature of the fitted plane curvature = model.Curvature;

Code 7 :
Curvature (Attempt 4) % Load the point cloud data ptCloud = pcread('pointCloud.ply');% Fit a surface to the point cloud using fitgmdist gmModel = fitgmdist(ptCloud.Location, 2); % Calculate the Gaussian curvature of the surface at each point K = gausscurvature(gmModel); The surface normals can be calculated using the pcnormals function from the Computer Vision System Toolbox, and the curvature can be estimated using the pcurvature function.And by indicating that the 'pccurvature' function does not exist, ChatGPT re-proposed Code 5.Afterwards, an attempt was made to guide the GPT Chat by providing more information on how to calculate the curvature: in MATLAB, generate a code to calculate the normals of a point cloud based on 25 neighbours.Then calculate the curvature based on variance of normal.
However, that model does not support curvature field.By a new attempt, ChatGPT referenced a function that does not exist in MATLAB: